You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2019/05/27 14:40:33 UTC
[JENKINS] Lucene-Solr-NightlyTests-7.7 - Build # 21 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.7/21/
2 tests failed.
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test
Error Message:
shard2 is not consistent. Got 91 from https://127.0.0.1:33145/collection1_shard2_replica_n5 (previous client) and got 92 from https://127.0.0.1:40355/collection1_shard2_replica_n1
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 91 from https://127.0.0.1:33145/collection1_shard2_replica_n5 (previous client) and got 92 from https://127.0.0.1:40355/collection1_shard2_replica_n1
at __randomizedtesting.SeedInfo.seed([6E64149722DDF104:E6302B4D8C219CFC]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1423)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1402)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:165)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1075)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1047)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.hdfs.HdfsUnloadDistributedZkTest.test
Error Message:
Error from server at http://127.0.0.1:41798: ADDREPLICA failed to create replica
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:41798: ADDREPLICA failed to create replica
at __randomizedtesting.SeedInfo.seed([6E64149722DDF104:E6302B4D8C219CFC]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.cloud.UnloadDistributedZkTest.testUnloadShardAndCollection(UnloadDistributedZkTest.java:127)
at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1075)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1047)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 14375 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> 2648209 INFO (SUITE-ChaosMonkeySafeLeaderTest-seed#[6E64149722DDF104]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/init-core-data-001
[junit4] 2> 2648210 WARN (SUITE-ChaosMonkeySafeLeaderTest-seed#[6E64149722DDF104]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=88 numCloses=88
[junit4] 2> 2648225 INFO (SUITE-ChaosMonkeySafeLeaderTest-seed#[6E64149722DDF104]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 2648227 INFO (SUITE-ChaosMonkeySafeLeaderTest-seed#[6E64149722DDF104]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 2648227 INFO (SUITE-ChaosMonkeySafeLeaderTest-seed#[6E64149722DDF104]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
[junit4] 2> 2648335 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 2648354 INFO (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2648355 INFO (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 2648464 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkTestServer start zk server on port:34467
[junit4] 2> 2648464 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:34467
[junit4] 2> 2648464 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 34467
[junit4] 2> 2648836 INFO (zkConnectionManagerCallback-5428-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2648962 INFO (zkConnectionManagerCallback-5430-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2648975 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 2648976 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 2648976 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 2648977 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 2648978 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 2648995 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 2648995 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 2648996 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 2648997 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 2648998 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 2648998 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 2649092 INFO (zkConnectionManagerCallback-5434-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2649105 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
[junit4] 2> 2650980 WARN (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
[junit4] 2> 2650981 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
[junit4] 2> 2650981 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
[junit4] 2> 2650981 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
[junit4] 2> 2651049 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2651049 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2651049 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 2651053 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4fd9768a{/,null,AVAILABLE}
[junit4] 2> 2651054 WARN (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@3cccf11[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 2651055 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@17a7b529{SSL,[ssl, http/1.1]}{127.0.0.1:35182}
[junit4] 2> 2651055 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.e.j.s.Server Started @2651140ms
[junit4] 2> 2651055 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/tempDir-001/control/data, hostContext=/, hostPort=35182, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/control-001/cores}
[junit4] 2> 2651055 ERROR (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2651055 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 2651055 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.7.2
[junit4] 2> 2651055 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2651055 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2651055 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2019-05-27T13:04:37.703Z
[junit4] 2> 2651089 INFO (zkConnectionManagerCallback-5436-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2651122 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 2651122 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/control-001/solr.xml
[junit4] 2> 2651125 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 2651125 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 2651139 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 2652531 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
[junit4] 2> 2652615 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34467/solr
[junit4] 2> 2652666 INFO (zkConnectionManagerCallback-5440-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2652760 INFO (zkConnectionManagerCallback-5442-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2653272 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35182_
[junit4] 2> 2653272 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_ ] o.a.s.c.Overseer Overseer (id=74764209601642501-127.0.0.1:35182_-n_0000000000) starting
[junit4] 2> 2653362 INFO (zkConnectionManagerCallback-5449-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2653433 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34467/solr ready
[junit4] 2> 2653469 INFO (OverseerStateUpdate-74764209601642501-127.0.0.1:35182_-n_0000000000) [n:127.0.0.1:35182_ ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:35182_
[junit4] 2> 2653469 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35182_
[junit4] 2> 2653505 INFO (zkCallback-5448-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2653505 INFO (OverseerStateUpdate-74764209601642501-127.0.0.1:35182_-n_0000000000) [n:127.0.0.1:35182_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2653671 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 2653800 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2653945 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2653945 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2654036 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [n:127.0.0.1:35182_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/control-001/cores
[junit4] 2> 2654306 INFO (zkConnectionManagerCallback-5455-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2654391 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2654416 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34467/solr ready
[junit4] 2> 2654623 INFO (qtp1672465640-39037) [n:127.0.0.1:35182_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:35182_&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2654699 INFO (OverseerThreadFactory-4550-thread-1-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
[junit4] 2> 2654896 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 2654896 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2656035 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.2
[junit4] 2> 2656124 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
[junit4] 2> 2656817 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 2656886 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
[junit4] 2> 2656887 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2656887 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/control-001/cores/control_collection_shard1_replica_n1/data/]
[junit4] 2> 2656907 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=7, maxMergedSegmentMB=58.0263671875, floorSegmentMB=1.8046875, forceMergeDeletesPctAllowed=25.489696409831605, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=33.81642606487709
[junit4] 2> 2656931 WARN (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 2657125 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2657125 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2657147 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
[junit4] 2> 2657147 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2657148 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@34b16866
[junit4] 2> 2657148 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@c57fd3[control_collection_shard1_replica_n1] main]
[junit4] 2> 2657178 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2657178 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2657179 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 2657179 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1634690435726180352
[junit4] 2> 2657183 INFO (searcherExecutor-4555-thread-1-processing-n:127.0.0.1:35182_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@c57fd3[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2657200 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
[junit4] 2> 2657200 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/control_collection/leaders/shard1
[junit4] 2> 2657202 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 2657202 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 2657202 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:35182/control_collection_shard1_replica_n1/
[junit4] 2> 2657202 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 2657214 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:35182/control_collection_shard1_replica_n1/ has no replicas
[junit4] 2> 2657214 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 2657215 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/74764209601642501-core_node2-n_0000000000
[junit4] 2> 2657217 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:35182/control_collection_shard1_replica_n1/ shard1
[junit4] 2> 2657218 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 2657252 INFO (qtp1672465640-39035) [n:127.0.0.1:35182_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2356
[junit4] 2> 2657254 INFO (qtp1672465640-39037) [n:127.0.0.1:35182_ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
[junit4] 2> 2657361 INFO (zkCallback-5441-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 2657361 INFO (qtp1672465640-39037) [n:127.0.0.1:35182_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:35182_&wt=javabin&version=2} status=0 QTime=2738
[junit4] 2> 2657380 INFO (zkCallback-5441-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 2657473 INFO (zkConnectionManagerCallback-5460-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2657490 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2657491 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34467/solr ready
[junit4] 2> 2657491 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 2657526 INFO (qtp1672465640-39037) [n:127.0.0.1:35182_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2657563 INFO (OverseerThreadFactory-4550-thread-2-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 2657563 INFO (OverseerCollectionConfigSetProcessor-74764209601642501-127.0.0.1:35182_-n_0000000000) [n:127.0.0.1:35182_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 2657781 WARN (OverseerThreadFactory-4550-thread-2-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
[junit4] 2> 2657783 INFO (qtp1672465640-39037) [n:127.0.0.1:35182_ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
[junit4] 2> 2657784 INFO (qtp1672465640-39037) [n:127.0.0.1:35182_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=257
[junit4] 2> 2657802 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances pullReplicaCount=0 numOtherReplicas=3
[junit4] 2> 2658839 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-1-001 of type NRT
[junit4] 2> 2658876 WARN (closeThreadPool-5461-thread-1) [ ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
[junit4] 2> 2658877 INFO (closeThreadPool-5461-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
[junit4] 2> 2658877 INFO (closeThreadPool-5461-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
[junit4] 2> 2658877 INFO (closeThreadPool-5461-thread-1) [ ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
[junit4] 2> 2659130 INFO (closeThreadPool-5461-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2659130 INFO (closeThreadPool-5461-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2659130 INFO (closeThreadPool-5461-thread-1) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 2659147 INFO (closeThreadPool-5461-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1e2eafd4{/,null,AVAILABLE}
[junit4] 2> 2659147 WARN (closeThreadPool-5461-thread-1) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@38632f7b[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 2659163 INFO (closeThreadPool-5461-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@48a5d748{SSL,[ssl, http/1.1]}{127.0.0.1:33145}
[junit4] 2> 2659163 INFO (closeThreadPool-5461-thread-1) [ ] o.e.j.s.Server Started @2659247ms
[junit4] 2> 2659163 INFO (closeThreadPool-5461-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=33145, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-1-001/cores}
[junit4] 2> 2659163 ERROR (closeThreadPool-5461-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2659163 INFO (closeThreadPool-5461-thread-1) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 2659163 INFO (closeThreadPool-5461-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.7.2
[junit4] 2> 2659163 INFO (closeThreadPool-5461-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2659163 INFO (closeThreadPool-5461-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2659163 INFO (closeThreadPool-5461-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2019-05-27T13:04:45.811Z
[junit4] 2> 2659223 INFO (zkConnectionManagerCallback-5463-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2659312 INFO (closeThreadPool-5461-thread-1) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 2659312 INFO (closeThreadPool-5461-thread-1) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-1-001/solr.xml
[junit4] 2> 2659316 INFO (closeThreadPool-5461-thread-1) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 2659316 INFO (closeThreadPool-5461-thread-1) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 2659370 INFO (closeThreadPool-5461-thread-1) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 2659568 INFO (OverseerCollectionConfigSetProcessor-74764209601642501-127.0.0.1:35182_-n_0000000000) [n:127.0.0.1:35182_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 2660351 INFO (closeThreadPool-5461-thread-1) [ ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
[junit4] 2> 2660462 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-2-001 of type NRT
[junit4] 2> 2660443 INFO (closeThreadPool-5461-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34467/solr
[junit4] 2> 2660476 WARN (closeThreadPool-5461-thread-2) [ ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
[junit4] 2> 2660476 INFO (closeThreadPool-5461-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
[junit4] 2> 2660476 INFO (closeThreadPool-5461-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
[junit4] 2> 2660476 INFO (closeThreadPool-5461-thread-2) [ ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
[junit4] 2> 2660532 INFO (closeThreadPool-5461-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2660532 INFO (closeThreadPool-5461-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2660532 INFO (closeThreadPool-5461-thread-2) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 2660532 INFO (closeThreadPool-5461-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5702f438{/,null,AVAILABLE}
[junit4] 2> 2660533 WARN (closeThreadPool-5461-thread-2) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@1dd1ed4c[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 2660556 INFO (closeThreadPool-5461-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@28ca1c9d{SSL,[ssl, http/1.1]}{127.0.0.1:37001}
[junit4] 2> 2660556 INFO (closeThreadPool-5461-thread-2) [ ] o.e.j.s.Server Started @2660641ms
[junit4] 2> 2660556 INFO (closeThreadPool-5461-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=37001, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-2-001/cores}
[junit4] 2> 2660557 ERROR (closeThreadPool-5461-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2660557 INFO (closeThreadPool-5461-thread-2) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 2660557 INFO (closeThreadPool-5461-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.7.2
[junit4] 2> 2660557 INFO (closeThreadPool-5461-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2660557 INFO (closeThreadPool-5461-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2660557 INFO (closeThreadPool-5461-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2019-05-27T13:04:47.205Z
[junit4] 2> 2660557 INFO (zkConnectionManagerCallback-5467-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2660585 INFO (zkConnectionManagerCallback-5469-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2660585 INFO (closeThreadPool-5461-thread-2) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 2660586 INFO (closeThreadPool-5461-thread-2) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-2-001/solr.xml
[junit4] 2> 2660621 INFO (closeThreadPool-5461-thread-2) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 2660621 INFO (closeThreadPool-5461-thread-2) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 2660623 INFO (closeThreadPool-5461-thread-2) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 2660722 INFO (zkConnectionManagerCallback-5471-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2660725 INFO (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2660739 INFO (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_ ] o.a.s.c.ZkController Publish node=127.0.0.1:33145_ as DOWN
[junit4] 2> 2660740 INFO (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2660740 INFO (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33145_
[junit4] 2> 2660741 INFO (zkCallback-5448-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2660741 INFO (zkCallback-5459-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2660741 INFO (zkCallback-5441-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2660840 INFO (zkCallback-5470-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2668154 INFO (zkConnectionManagerCallback-5478-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2668410 INFO (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2668411 INFO (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34467/solr ready
[junit4] 2> 2668503 INFO (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 2668595 INFO (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2668760 INFO (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2668761 INFO (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2668774 INFO (closeThreadPool-5461-thread-1) [n:127.0.0.1:33145_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-1-001/cores
[junit4] 2> 2668958 INFO (closeThreadPool-5461-thread-2) [ ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
[junit4] 2> 2669139 INFO (closeThreadPool-5461-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34467/solr
[junit4] 2> 2669175 INFO (zkConnectionManagerCallback-5483-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2669231 INFO (zkConnectionManagerCallback-5485-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2669234 INFO (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2669249 INFO (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_ ] o.a.s.c.ZkController Publish node=127.0.0.1:37001_ as DOWN
[junit4] 2> 2669250 INFO (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2669250 INFO (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37001_
[junit4] 2> 2669251 INFO (zkCallback-5470-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 2669251 INFO (zkCallback-5441-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 2669252 INFO (zkCallback-5459-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 2669252 INFO (zkCallback-5448-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 2669305 INFO (zkCallback-5477-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 2669305 INFO (zkCallback-5484-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 2669361 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[6E64149722DDF104]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-3-001 of type NRT
[junit4] 2> 2669415 WARN (closeThreadPool-5461-thread-3) [ ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
[junit4] 2> 2669415 INFO (closeThreadPool-5461-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
[junit4] 2> 2669415 INFO (closeThreadPool-5461-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
[junit4] 2> 2669416 INFO (closeThreadPool-5461-thread-3) [ ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_191-b12
[junit4] 2> 2669507 INFO (closeThreadPool-5461-thread-3) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2669507 INFO (closeThreadPool-5461-thread-3) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2669507 INFO (closeThreadPool-5461-thread-3) [ ] o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 2669508 INFO (closeThreadPool-5461-thread-3) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@575803bb{/,null,AVAILABLE}
[junit4] 2> 2669508 WARN (closeThreadPool-5461-thread-3) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@4e1f4151[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 2669559 INFO (zkConnectionManagerCallback-5492-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2669560 INFO (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 2669561 INFO (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34467/solr ready
[junit4] 2> 2669575 INFO (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 2669580 INFO (closeThreadPool-5461-thread-3) [ ] o.e.j.s.AbstractConnector Started ServerConnector@5626e959{SSL,[ssl, http/1.1]}{127.0.0.1:40355}
[junit4] 2> 2669580 INFO (closeThreadPool-5461-thread-3) [ ] o.e.j.s.Server Started @2669664ms
[junit4] 2> 2669580 INFO (closeThreadPool-5461-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/tempDir-001/jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=40355, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-3-001/cores}
[junit4] 2> 2669580 ERROR (closeThreadPool-5461-thread-3) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2669580 INFO (closeThreadPool-5461-thread-3) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 2669580 INFO (closeThreadPool-5461-thread-3) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.7.2
[junit4] 2> 2669580 INFO (closeThreadPool-5461-thread-3) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2669580 INFO (closeThreadPool-5461-thread-3) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2669580 INFO (closeThreadPool-5461-thread-3) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2019-05-27T13:04:56.228Z
[junit4] 2> 2669620 INFO (zkConnectionManagerCallback-5495-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2669653 INFO (closeThreadPool-5461-thread-3) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 2669653 INFO (closeThreadPool-5461-thread-3) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-3-001/solr.xml
[junit4] 2> 2669656 INFO (closeThreadPool-5461-thread-3) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 2669656 INFO (closeThreadPool-5461-thread-3) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 2669670 INFO (closeThreadPool-5461-thread-3) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 2669714 INFO (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2669824 INFO (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2669824 INFO (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2669825 INFO (closeThreadPool-5461-thread-2) [n:127.0.0.1:37001_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-2-001/cores
[junit4] 2> 2670746 INFO (closeThreadPool-5461-thread-3) [ ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
[junit4] 2> 2670780 INFO (closeThreadPool-5461-thread-3) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34467/solr
[junit4] 2> 2670855 INFO (zkConnectionManagerCallback-5499-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2670928 INFO (zkConnectionManagerCallback-5501-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2670931 INFO (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 2670951 INFO (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_ ] o.a.s.c.ZkController Publish node=127.0.0.1:40355_ as DOWN
[junit4] 2> 2670952 INFO (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 2670952 INFO (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40355_
[junit4] 2> 2670966 INFO (zkCallback-5441-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2670982 INFO (zkCallback-5470-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2670985 INFO (zkCallback-5459-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2670985 INFO (zkCallback-5448-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2671002 INFO (zkCallback-5477-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2671002 INFO (zkCallback-5484-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2671019 INFO (zkCallback-5491-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2671034 INFO (zkCallback-5500-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2671277 INFO (zkConnectionManagerCallback-5508-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2671294 INFO (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 2671295 INFO (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34467/solr ready
[junit4] 2> 2671346 INFO (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 2671422 INFO (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2671494 INFO (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2671494 INFO (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2671512 INFO (closeThreadPool-5461-thread-3) [n:127.0.0.1:40355_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-3-001/cores
[junit4] 2> 2672463 INFO (qtp712521339-39093) [n:127.0.0.1:33145_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:40355_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2672549 INFO (qtp712521339-39097) [n:127.0.0.1:33145_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:33145_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2672696 INFO (qtp712521339-39094) [n:127.0.0.1:33145_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:37001_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2672697 INFO (OverseerThreadFactory-4550-thread-3-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ c:collection1 s:shard2 ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:40355_ for creating new replica of shard shard2 for collection collection1
[junit4] 2> 2672698 INFO (OverseerThreadFactory-4550-thread-3-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ c:collection1 s:shard2 ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
[junit4] 2> 2672732 INFO (OverseerThreadFactory-4550-thread-4-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ c:collection1 s:shard1 ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:37001_ for creating new replica of shard shard1 for collection collection1
[junit4] 2> 2672733 INFO (OverseerThreadFactory-4550-thread-4-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ c:collection1 s:shard1 ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
[junit4] 2> 2673079 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ x:collection1_shard2_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n1&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 2673114 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ x:collection1_shard1_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n2&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 2674271 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.2
[junit4] 2> 2674306 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.2
[junit4] 2> 2674574 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema [collection1_shard2_replica_n1] Schema name=test
[junit4] 2> 2674577 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.s.IndexSchema [collection1_shard1_replica_n2] Schema name=test
[junit4] 2> 2675251 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 2675399 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 2675456 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n2' using configuration from collection collection1, trusted=true
[junit4] 2> 2675457 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n2' (registry 'solr.core.collection1.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2675457 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.SolrCore [[collection1_shard1_replica_n2] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-2-001/cores/collection1_shard1_replica_n2], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-2-001/cores/collection1_shard1_replica_n2/data/]
[junit4] 2> 2675473 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=7, maxMergedSegmentMB=58.0263671875, floorSegmentMB=1.8046875, forceMergeDeletesPctAllowed=25.489696409831605, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=33.81642606487709
[junit4] 2> 2675506 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n1' using configuration from collection collection1, trusted=true
[junit4] 2> 2675507 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n1' (registry 'solr.core.collection1.shard2.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2675519 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [[collection1_shard2_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-3-001/cores/collection1_shard2_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-3-001/cores/collection1_shard2_replica_n1/data/]
[junit4] 2> 2675522 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=7, maxMergedSegmentMB=58.0263671875, floorSegmentMB=1.8046875, forceMergeDeletesPctAllowed=25.489696409831605, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=33.81642606487709
[junit4] 2> 2675642 WARN (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 2675715 WARN (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 2676021 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2676021 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2676055 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
[junit4] 2> 2676055 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2676056 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@7e390287
[junit4] 2> 2676078 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2676078 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2676080 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms;
[junit4] 2> 2676080 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2676081 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@cc1eb8e
[junit4] 2> 2676095 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1d5d8aaf[collection1_shard2_replica_n1] main]
[junit4] 2> 2676095 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@cbb0764[collection1_shard1_replica_n2] main]
[junit4] 2> 2676096 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2676096 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 2676096 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2676096 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 2676097 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 2676097 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 2676097 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1634690455563141120
[junit4] 2> 2676097 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1634690455563141120
[junit4] 2> 2676112 INFO (searcherExecutor-4588-thread-1-processing-n:127.0.0.1:40355_ x:collection1_shard2_replica_n1 c:collection1 s:shard2) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.SolrCore [collection1_shard2_replica_n1] Registered new searcher Searcher@1d5d8aaf[collection1_shard2_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2676114 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node4=0}, version=0}
[junit4] 2> 2676114 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard2
[junit4] 2> 2676133 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 2676133 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 2676133 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:40355/collection1_shard2_replica_n1/
[junit4] 2> 2676133 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 2676133 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:40355/collection1_shard2_replica_n1/ has no replicas
[junit4] 2> 2676133 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 2676134 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard2/leader after winning as /collections/collection1/leader_elect/shard2/election/74764209601642515-core_node4-n_0000000000
[junit4] 2> 2676134 INFO (searcherExecutor-4587-thread-1-processing-n:127.0.0.1:37001_ x:collection1_shard1_replica_n2 c:collection1 s:shard1) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.SolrCore [collection1_shard1_replica_n2] Registered new searcher Searcher@cbb0764[collection1_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2676148 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:40355/collection1_shard2_replica_n1/ shard2
[junit4] 2> 2676150 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node3=0}, version=0}
[junit4] 2> 2676150 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/collection1/leaders/shard1
[junit4] 2> 2676169 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 2676169 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 2676169 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:37001/collection1_shard1_replica_n2/
[junit4] 2> 2676169 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 2676169 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.SyncStrategy https://127.0.0.1:37001/collection1_shard1_replica_n2/ has no replicas
[junit4] 2> 2676169 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 2676169 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/collection1/leaders/shard1/leader after winning as /collections/collection1/leader_elect/shard1/election/74764209601642511-core_node3-n_0000000000
[junit4] 2> 2676170 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:37001/collection1_shard1_replica_n2/ shard1
[junit4] 2> 2676177 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 2676305 INFO (qtp420959561-39154) [n:127.0.0.1:40355_ c:collection1 s:shard2 x:collection1_shard2_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n1&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3226
[junit4] 2> 2676313 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 2676476 INFO (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1 ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:40355_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=4012
[junit4] 2> 2676477 INFO (qtp1558407297-39115) [n:127.0.0.1:37001_ c:collection1 s:shard1 x:collection1_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n2&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=3363
[junit4] 2> 2676565 WARN (OverseerCollectionConfigSetProcessor-74764209601642501-127.0.0.1:35182_-n_0000000000) [n:127.0.0.1:35182_ ] o.a.s.c.LockTree lock_is_leaked at[collection1, shard1]
[junit4] 2> 2676576 INFO (OverseerThreadFactory-4550-thread-4-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ c:collection1 s:shard1 ] o.a.s.c.LockTree Unlocked multiple times : collection1/shard1
[junit4] 2> 2676585 INFO (qtp712521339-39094) [n:127.0.0.1:33145_ c:collection1 ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:37001_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=3889
[junit4] 2> 2676601 INFO (OverseerThreadFactory-4550-thread-5-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ c:collection1 s:shard2 ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:33145_ for creating new replica of shard shard2 for collection collection1
[junit4] 2> 2676617 INFO (OverseerCollectionConfigSetProcessor-74764209601642501-127.0.0.1:35182_-n_0000000000) [n:127.0.0.1:35182_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000008 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 2676653 INFO (OverseerCollectionConfigSetProcessor-74764209601642501-127.0.0.1:35182_-n_0000000000) [n:127.0.0.1:35182_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 2676690 INFO (OverseerThreadFactory-4550-thread-5-processing-n:127.0.0.1:35182_) [n:127.0.0.1:35182_ c:collection1 s:shard2 ] o.a.s.c.a.c.AddReplicaCmd Returning CreateReplica command.
[junit4] 2> 2676734 INFO (qtp712521339-39093) [n:127.0.0.1:33145_ x:collection1_shard2_replica_n5] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n5&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 2677852 INFO (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1 s:shard2 x:collection1_shard2_replica_n5] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.7.2
[junit4] 2> 2677980 INFO (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1 s:shard2 x:collection1_shard2_replica_n5] o.a.s.s.IndexSchema [collection1_shard2_replica_n5] Schema name=test
[junit4] 2> 2678326 INFO (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1 s:shard2 x:collection1_shard2_replica_n5] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 2678382 INFO (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1 s:shard2 x:collection1_shard2_replica_n5] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n5' using configuration from collection collection1, trusted=true
[junit4] 2> 2678382 INFO (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1 s:shard2 x:collection1_shard2_replica_n5] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n5' (registry 'solr.core.collection1.shard2.replica_n5') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42d0c249
[junit4] 2> 2678383 INFO (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1 s:shard2 x:collection1_shard2_replica_n5] o.a.s.c.SolrCore [[collection1_shard2_replica_n5] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-1-001/cores/collection1_shard2_replica_n5], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/../../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/J2/temp/solr.cloud.ChaosMonkeySafeLeaderTest_6E64149722DDF104-001/shard-1-001/cores/collection1_shard2_replica_n5/data/]
[junit4] 2> 2678398 INFO (qtp712521339-39093) [n:127.0.0.1:33145_ c:collection1 s:shard2 x:collection1_shard2_replica_n5] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=7, maxMergedSegmentMB=58.0263671875, floorSegmentMB=1.8046875, forceMergeDeletesPctAllowed=25.489696409831605, segmentsPerTier=4.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0, deletesPctAllowed=33.81642606487709
[junit4] 2> 2678501 WARN (qtp712521339-39093) [n:12
[...truncated too long message...]
cene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-authn-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-authz-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-changelog-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/api-all-1.0.0-M20.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/bcprov-jdk15on-1.60.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/byte-buddy-1.9.3.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/commons-collections-3.2.2.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/ehcache-core-2.4.4.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/hadoop-common-2.7.4-tests.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/hadoop-hdfs-2.7.4-tests.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/hadoop-minikdc-2.7.4.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/htrace-core-3.2.0-incubating.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/jersey-core-1.9.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/jersey-server-1.9.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/jetty-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/jetty-sslengine-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/jetty-util-6.1.26.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/mina-core-2.0.0-M5.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/mockito-core-2.23.4.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/netty-all-4.0.36.Final.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/core/test-lib/objenesis-2.6.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/analysis/icu/lucene-analyzers-icu-7.7.2-SNAPSHOT.jar:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/contrib/solr-analysis-extras/classes/java:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/queryparser/classes/test:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/build/backward-codecs/classes/test:/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/contrib/analysis-extras/lib/icu4j-62.1.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-launcher.jar:/home/jenkins/.ant/lib/ivy-2.4.0.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-oro.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-javamail.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jsch.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-resolver.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bsf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-logging.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-xalan2.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jdepend.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-netrexx.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-commons-net.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-regexp.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jai.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-junit4.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-testutil.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-jmf.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-antlr.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-swing.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-bcel.jar:/home/jenkins/tools/ant/apache-ant-1.8.4/lib/ant-apache-log4j.jar:/usr/local/asfpackages/java/jdk1.8.0_191/lib/tools.jar:/home/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.7.2.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/temp/junit4-J0-20190527_122026_3735801384452750691395.events @/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build/solr-core/test/temp/junit4-J0-20190527_122026_3736170435525204442908.suites -stdin
[junit4] ERROR: JVM J0 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1563)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:123)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:997)
[junit4] at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:994)
[junit4] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] at java.lang.Thread.run(Thread.java:748)
BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/build.xml:651: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/build.xml:577: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/build.xml:59: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/build.xml:231: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/solr/common-build.xml:558: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/common-build.xml:1571: The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.7/checkout/lucene/common-build.xml:1098: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
Total time: 450 minutes 37 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[Fast Archiver] No artifacts from Lucene-Solr-NightlyTests-7.7 #19 to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any