You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2017/09/26 03:46:00 UTC
[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk1.8.0_144) - Build # 490
- Still Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/490/
Java: 64bit/jdk1.8.0_144 -XX:-UseCompressedOops -XX:+UseParallelGC
3 tests failed.
FAILED: org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testExecutorStream
Error Message:
Error from server at https://127.0.0.1:35431/solr/mainCorpus_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 </title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /solr/mainCorpus_shard2_replica_n3/update. Reason: <pre> Can not find: /solr/mainCorpus_shard2_replica_n3/update</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/> </body> </html>
Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:35431/solr/mainCorpus_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/mainCorpus_shard2_replica_n3/update. Reason:
<pre> Can not find: /solr/mainCorpus_shard2_replica_n3/update</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>
at __randomizedtesting.SeedInfo.seed([67874D4D39A483A3:4547CCB61ACEA9B3]:0)
at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:539)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:993)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testExecutorStream(StreamExpressionTest.java:7265)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:35431/solr/mainCorpus_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/mainCorpus_shard2_replica_n3/update. Reason:
<pre> Can not find: /solr/mainCorpus_shard2_replica_n3/update</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:591)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:516)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
FAILED: org.apache.solr.cloud.ShardSplitTest.testSplitWithChaosMonkey
Error Message:
There are still nodes recoverying - waited for 330 seconds
Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
at __randomizedtesting.SeedInfo.seed([CF40D441C78FABDC:4467079086890058]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:185)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:140)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:135)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:908)
at org.apache.solr.cloud.ShardSplitTest.testSplitWithChaosMonkey(ShardSplitTest.java:436)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.TestTlogReplica.testRecovery
Error Message:
Can not find doc 3 in http://127.0.0.1:45071/solr
Stack Trace:
java.lang.AssertionError: Can not find doc 3 in http://127.0.0.1:45071/solr
at __randomizedtesting.SeedInfo.seed([CF40D441C78FABDC:EB0ADEDEADF617B]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:868)
at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:559)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 11909 lines...]
[junit4] Suite: org.apache.solr.cloud.TestTlogReplica
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_CF40D441C78FABDC-001/init-core-data-001
[junit4] 2> 280678 WARN (SUITE-TestTlogReplica-seed#[CF40D441C78FABDC]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
[junit4] 2> 280678 INFO (SUITE-TestTlogReplica-seed#[CF40D441C78FABDC]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 280679 INFO (SUITE-TestTlogReplica-seed#[CF40D441C78FABDC]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 280679 INFO (SUITE-TestTlogReplica-seed#[CF40D441C78FABDC]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 280679 INFO (SUITE-TestTlogReplica-seed#[CF40D441C78FABDC]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_CF40D441C78FABDC-001/tempDir-001
[junit4] 2> 280679 INFO (SUITE-TestTlogReplica-seed#[CF40D441C78FABDC]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 280680 INFO (Thread-1063) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 280680 INFO (Thread-1063) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 280681 ERROR (Thread-1063) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 280780 INFO (SUITE-TestTlogReplica-seed#[CF40D441C78FABDC]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:34221
[junit4] 2> 280784 INFO (jetty-launcher-445-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 280785 INFO (jetty-launcher-445-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 280789 INFO (jetty-launcher-445-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@d8eebcf{/solr,null,AVAILABLE}
[junit4] 2> 280789 INFO (jetty-launcher-445-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@72d0cade{/solr,null,AVAILABLE}
[junit4] 2> 280790 INFO (jetty-launcher-445-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3e65fcbb{HTTP/1.1,[http/1.1]}{127.0.0.1:45071}
[junit4] 2> 280790 INFO (jetty-launcher-445-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@8cb347e{HTTP/1.1,[http/1.1]}{127.0.0.1:44195}
[junit4] 2> 280791 INFO (jetty-launcher-445-thread-1) [ ] o.e.j.s.Server Started @282082ms
[junit4] 2> 280791 INFO (jetty-launcher-445-thread-2) [ ] o.e.j.s.Server Started @282082ms
[junit4] 2> 280791 INFO (jetty-launcher-445-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45071}
[junit4] 2> 280791 INFO (jetty-launcher-445-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44195}
[junit4] 2> 280791 ERROR (jetty-launcher-445-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 280791 ERROR (jetty-launcher-445-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 280791 INFO (jetty-launcher-445-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 280791 INFO (jetty-launcher-445-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 280791 INFO (jetty-launcher-445-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 280791 INFO (jetty-launcher-445-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 280791 INFO (jetty-launcher-445-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 280791 INFO (jetty-launcher-445-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 280791 INFO (jetty-launcher-445-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T02:58:13.316Z
[junit4] 2> 280791 INFO (jetty-launcher-445-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T02:58:13.316Z
[junit4] 2> 280794 INFO (jetty-launcher-445-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 280794 INFO (jetty-launcher-445-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 280797 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ebc1ede550002, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 280802 INFO (jetty-launcher-445-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34221/solr
[junit4] 2> 280802 INFO (jetty-launcher-445-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34221/solr
[junit4] 2> 280839 INFO (jetty-launcher-445-thread-2) [n:127.0.0.1:44195_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 280839 INFO (jetty-launcher-445-thread-1) [n:127.0.0.1:45071_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 280840 INFO (jetty-launcher-445-thread-2) [n:127.0.0.1:44195_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44195_solr
[junit4] 2> 280841 INFO (jetty-launcher-445-thread-1) [n:127.0.0.1:45071_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45071_solr
[junit4] 2> 280841 INFO (jetty-launcher-445-thread-2) [n:127.0.0.1:44195_solr ] o.a.s.c.Overseer Overseer (id=98723082613882885-127.0.0.1:44195_solr-n_0000000000) starting
[junit4] 2> 280850 INFO (zkCallback-457-thread-1-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 280850 INFO (zkCallback-456-thread-1-processing-n:127.0.0.1:44195_solr) [n:127.0.0.1:44195_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 280854 INFO (jetty-launcher-445-thread-2) [n:127.0.0.1:44195_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44195_solr
[junit4] 2> 280855 INFO (zkCallback-456-thread-1-processing-n:127.0.0.1:44195_solr) [n:127.0.0.1:44195_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 280855 INFO (zkCallback-457-thread-1-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 280910 INFO (jetty-launcher-445-thread-1) [n:127.0.0.1:45071_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45071.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@218cd85f
[junit4] 2> 280915 INFO (jetty-launcher-445-thread-1) [n:127.0.0.1:45071_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45071.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@218cd85f
[junit4] 2> 280915 INFO (jetty-launcher-445-thread-1) [n:127.0.0.1:45071_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45071.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@218cd85f
[junit4] 2> 280915 INFO (jetty-launcher-445-thread-1) [n:127.0.0.1:45071_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_CF40D441C78FABDC-001/tempDir-001/node1/.
[junit4] 2> 280928 INFO (jetty-launcher-445-thread-2) [n:127.0.0.1:44195_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44195.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@218cd85f
[junit4] 2> 280932 INFO (jetty-launcher-445-thread-2) [n:127.0.0.1:44195_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44195.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@218cd85f
[junit4] 2> 280932 INFO (jetty-launcher-445-thread-2) [n:127.0.0.1:44195_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44195.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@218cd85f
[junit4] 2> 280933 INFO (jetty-launcher-445-thread-2) [n:127.0.0.1:44195_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_CF40D441C78FABDC-001/tempDir-001/node2/.
[junit4] 2> 280948 INFO (SUITE-TestTlogReplica-seed#[CF40D441C78FABDC]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 280948 INFO (SUITE-TestTlogReplica-seed#[CF40D441C78FABDC]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34221/solr ready
[junit4] 2> 280949 INFO (SUITE-TestTlogReplica-seed#[CF40D441C78FABDC]-worker) [ ] o.a.s.c.TestTlogReplica Using legacyCloud?: false
[junit4] 2> 280951 INFO (qtp1234247926-3349) [n:127.0.0.1:44195_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterprop with params val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 280953 INFO (qtp1234247926-3349) [n:127.0.0.1:44195_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 280963 INFO (TEST-TestTlogReplica.testKillLeader-seed#[CF40D441C78FABDC]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testKillLeader
[junit4] 2> 280964 INFO (qtp1234247926-3351) [n:127.0.0.1:44195_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_kill_leader&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 280965 INFO (OverseerThreadFactory-1326-thread-1-processing-n:127.0.0.1:44195_solr) [n:127.0.0.1:44195_solr ] o.a.s.c.CreateCollectionCmd Create collection tlog_replica_test_kill_leader
[junit4] 2> 281070 INFO (OverseerStateUpdate-98723082613882885-127.0.0.1:44195_solr-n_0000000000) [n:127.0.0.1:44195_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_kill_leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"tlog_replica_test_kill_leader_shard1_replica_t1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:44195/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 281072 INFO (OverseerStateUpdate-98723082613882885-127.0.0.1:44195_solr-n_0000000000) [n:127.0.0.1:44195_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_kill_leader",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"tlog_replica_test_kill_leader_shard1_replica_t2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:45071/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 281275 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_kill_leader_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_kill_leader&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 281275 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=tlog_replica_test_kill_leader_shard1_replica_t2&action=CREATE&numShards=1&collection=tlog_replica_test_kill_leader&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 281276 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 281276 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 281378 INFO (zkCallback-457-thread-1-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 281378 INFO (zkCallback-456-thread-1-processing-n:127.0.0.1:44195_solr) [n:127.0.0.1:44195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 281379 INFO (zkCallback-457-thread-2-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 281379 INFO (zkCallback-456-thread-2-processing-n:127.0.0.1:44195_solr) [n:127.0.0.1:44195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 282283 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 282297 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 282300 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.s.IndexSchema [tlog_replica_test_kill_leader_shard1_replica_t2] Schema name=minimal
[junit4] 2> 282302 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 282303 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_kill_leader_shard1_replica_t2' using configuration from collection tlog_replica_test_kill_leader, trusted=true
[junit4] 2> 282303 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.s.IndexSchema [tlog_replica_test_kill_leader_shard1_replica_t1] Schema name=minimal
[junit4] 2> 282304 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45071.solr.core.tlog_replica_test_kill_leader.shard1.replica_t2' (registry 'solr.core.tlog_replica_test_kill_leader.shard1.replica_t2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@218cd85f
[junit4] 2> 282304 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 282304 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.SolrCore [[tlog_replica_test_kill_leader_shard1_replica_t2] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_CF40D441C78FABDC-001/tempDir-001/node1/tlog_replica_test_kill_leader_shard1_replica_t2], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_CF40D441C78FABDC-001/tempDir-001/node1/./tlog_replica_test_kill_leader_shard1_replica_t2/data/]
[junit4] 2> 282305 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 282305 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_kill_leader_shard1_replica_t1' using configuration from collection tlog_replica_test_kill_leader, trusted=true
[junit4] 2> 282306 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44195.solr.core.tlog_replica_test_kill_leader.shard1.replica_t1' (registry 'solr.core.tlog_replica_test_kill_leader.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@218cd85f
[junit4] 2> 282306 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 282306 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.SolrCore [[tlog_replica_test_kill_leader_shard1_replica_t1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_CF40D441C78FABDC-001/tempDir-001/node2/tlog_replica_test_kill_leader_shard1_replica_t1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_CF40D441C78FABDC-001/tempDir-001/node2/./tlog_replica_test_kill_leader_shard1_replica_t1/data/]
[junit4] 2> 282333 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 282333 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 282333 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 282333 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 282334 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7433e298[tlog_replica_test_kill_leader_shard1_replica_t1] main]
[junit4] 2> 282336 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 282336 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 282337 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 282337 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 282337 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 282337 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579569323559616512
[junit4] 2> 282338 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 282338 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 282338 INFO (searcherExecutor-1332-thread-1-processing-n:127.0.0.1:44195_solr x:tlog_replica_test_kill_leader_shard1_replica_t1 s:shard1 c:tlog_replica_test_kill_leader r:core_node3) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.SolrCore [tlog_replica_test_kill_leader_shard1_replica_t1] Registered new searcher Searcher@7433e298[tlog_replica_test_kill_leader_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 282340 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@404d5c55[tlog_replica_test_kill_leader_shard1_replica_t2] main]
[junit4] 2> 282340 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 282340 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 282341 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 282341 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
[junit4] 2> 282342 INFO (searcherExecutor-1331-thread-1-processing-n:127.0.0.1:45071_solr x:tlog_replica_test_kill_leader_shard1_replica_t2 s:shard1 c:tlog_replica_test_kill_leader r:core_node4) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.SolrCore [tlog_replica_test_kill_leader_shard1_replica_t2] Registered new searcher Searcher@404d5c55[tlog_replica_test_kill_leader_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 282342 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579569323564859392
[junit4] 2> 282443 INFO (zkCallback-456-thread-1-processing-n:127.0.0.1:44195_solr) [n:127.0.0.1:44195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 282443 INFO (zkCallback-457-thread-2-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 282443 INFO (zkCallback-457-thread-1-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 282443 INFO (zkCallback-456-thread-2-processing-n:127.0.0.1:44195_solr) [n:127.0.0.1:44195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 282842 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 282842 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 282842 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:44195/solr/tlog_replica_test_kill_leader_shard1_replica_t1/
[junit4] 2> 282842 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_kill_leader_shard1_replica_t1 url=http://127.0.0.1:44195/solr START replicas=[http://127.0.0.1:45071/solr/tlog_replica_test_kill_leader_shard1_replica_t2/] nUpdates=100
[junit4] 2> 282844 INFO (qtp1047709242-3357) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 283142 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_kill_leader_shard1_replica_t1 url=http://127.0.0.1:44195/solr DONE. We have no versions. sync failed.
[junit4] 2> 283143 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 283143 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 283143 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 283143 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.ZkController tlog_replica_test_kill_leader_shard1_replica_t1 stopping background replication from leader
[junit4] 2> 283144 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:44195/solr/tlog_replica_test_kill_leader_shard1_replica_t1/ shard1
[junit4] 2> 283246 INFO (zkCallback-456-thread-1-processing-n:127.0.0.1:44195_solr) [n:127.0.0.1:44195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 283246 INFO (zkCallback-457-thread-2-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 283246 INFO (zkCallback-456-thread-2-processing-n:127.0.0.1:44195_solr) [n:127.0.0.1:44195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 283246 INFO (zkCallback-457-thread-1-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 283295 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 283296 INFO (qtp1234247926-3342) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_kill_leader_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_kill_leader&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2021
[junit4] 2> 283346 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.ZkController tlog_replica_test_kill_leader_shard1_replica_t2 starting background replication from leader
[junit4] 2> 283346 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
[junit4] 2> 283349 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
[junit4] 2> 283349 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 283350 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=tlog_replica_test_kill_leader_shard1_replica_t2&action=CREATE&numShards=1&collection=tlog_replica_test_kill_leader&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2074
[junit4] 2> 283352 INFO (qtp1234247926-3351) [n:127.0.0.1:44195_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 283452 INFO (zkCallback-456-thread-1-processing-n:127.0.0.1:44195_solr) [n:127.0.0.1:44195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 283452 INFO (zkCallback-456-thread-2-processing-n:127.0.0.1:44195_solr) [n:127.0.0.1:44195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 283452 INFO (zkCallback-457-thread-2-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 283452 INFO (zkCallback-457-thread-1-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 284352 INFO (qtp1234247926-3351) [n:127.0.0.1:44195_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_kill_leader&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=2&wt=javabin&version=2} status=0 QTime=3387
[junit4] 2> 284359 INFO (qtp1047709242-3346) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=http://127.0.0.1:44195/solr/tlog_replica_test_kill_leader_shard1_replica_t1/&wt=javabin&version=2}{add=[1 (1579569325676691456)]} 0 0
[junit4] 2> 284361 INFO (qtp1234247926-3356) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[1 (1579569325676691456)]} 0 5
[junit4] 2> 284364 INFO (qtp1047709242-3346) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:44195/solr/tlog_replica_test_kill_leader_shard1_replica_t1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 284364 INFO (qtp1234247926-3344) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579569325685080064,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 284364 INFO (qtp1234247926-3344) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@695f1d30 commitCommandVersion:1579569325685080064
[junit4] 2> 284372 INFO (qtp1234247926-3344) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@40fe41fb[tlog_replica_test_kill_leader_shard1_replica_t1] main]
[junit4] 2> 284372 INFO (qtp1234247926-3344) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 284374 INFO (searcherExecutor-1332-thread-1-processing-n:127.0.0.1:44195_solr x:tlog_replica_test_kill_leader_shard1_replica_t1 s:shard1 c:tlog_replica_test_kill_leader r:core_node3) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.SolrCore [tlog_replica_test_kill_leader_shard1_replica_t1] Registered new searcher Searcher@40fe41fb[tlog_replica_test_kill_leader_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):C1)))}
[junit4] 2> 284374 INFO (qtp1234247926-3344) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:44195/solr/tlog_replica_test_kill_leader_shard1_replica_t1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 10
[junit4] 2> 284375 INFO (qtp1234247926-3343) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/update params={_stateVer_=tlog_replica_test_kill_leader:5&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 12
[junit4] 2> 284379 INFO (qtp1234247926-3356) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0
[junit4] 2> 284382 INFO (qtp1234247926-3345) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0
[junit4] 2> 284386 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 284487 INFO (qtp1047709242-3348) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 284587 INFO (qtp1047709242-3348) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 284688 INFO (qtp1047709242-3348) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 284789 INFO (qtp1047709242-3348) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 284891 INFO (qtp1047709242-3346) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 284967 INFO (OverseerCollectionConfigSetProcessor-98723082613882885-127.0.0.1:44195_solr-n_0000000000) [n:127.0.0.1:44195_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 284991 INFO (qtp1047709242-3350) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 285092 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 285193 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 285293 INFO (qtp1047709242-3346) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 285394 INFO (qtp1047709242-3350) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 285495 INFO (qtp1047709242-3348) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 285596 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 285696 INFO (qtp1047709242-3346) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 285797 INFO (qtp1047709242-3350) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 285897 INFO (qtp1047709242-3348) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 285998 INFO (qtp1047709242-3354) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 286099 INFO (qtp1047709242-3346) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 286199 INFO (qtp1047709242-3350) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 286251 INFO (indexFetcher-1342-thread-1) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.h.IndexFetcher Updated masterUrl to http://127.0.0.1:44195/solr/tlog_replica_test_kill_leader_shard1_replica_t1/
[junit4] 2> 286253 INFO (qtp1234247926-3345) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 286253 INFO (indexFetcher-1342-thread-1) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.h.IndexFetcher Master's generation: 2
[junit4] 2> 286253 INFO (indexFetcher-1342-thread-1) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.h.IndexFetcher Master's version: 1506394696889
[junit4] 2> 286253 INFO (indexFetcher-1342-thread-1) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 286253 INFO (indexFetcher-1342-thread-1) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 286253 INFO (indexFetcher-1342-thread-1) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.h.IndexFetcher Starting replication process
[junit4] 2> 286253 INFO (qtp1234247926-3356) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=0
[junit4] 2> 286254 INFO (indexFetcher-1342-thread-1) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.h.IndexFetcher Number of files in latest index in master: 19
[junit4] 2> 286256 INFO (indexFetcher-1342-thread-1) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MockDirectoryWrapper(RAMDirectory@1f6e08b7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@324c6cd5)
[junit4] 2> 286256 INFO (qtp1234247926-3349) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286257 INFO (qtp1234247926-3343) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_LuceneVarGapDocFreqInterval_0.pos&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286259 INFO (qtp1234247926-3345) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_LuceneFixedGap_0.tib&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286262 INFO (qtp1234247926-3356) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.nvd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286263 INFO (qtp1234247926-3349) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_LuceneVarGapDocFreqInterval_0.tib&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286276 INFO (qtp1234247926-3343) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fdx&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286277 INFO (qtp1234247926-3345) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_LuceneFixedGap_0.doc&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286279 INFO (qtp1234247926-3356) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fdt&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286280 INFO (qtp1234247926-3349) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_LuceneFixedGap_0.pos&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286281 INFO (qtp1234247926-3343) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_LuceneFixedGap_0.tii&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286291 INFO (qtp1234247926-3345) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Asserting_0.dvd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286292 INFO (qtp1234247926-3345) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_LuceneVarGapDocFreqInterval_0.tiv&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286294 INFO (qtp1234247926-3349) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Memory_0.mdvd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286295 INFO (qtp1234247926-3343) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Memory_0.mdvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286296 INFO (qtp1234247926-3343) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Asserting_0.dvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286297 INFO (qtp1234247926-3345) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.nvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286298 INFO (qtp1234247926-3349) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_LuceneVarGapDocFreqInterval_0.doc&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286298 INFO (qtp1234247926-3356) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fnm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286299 INFO (qtp1234247926-3356) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t1] webapp=/solr path=/replication params={generation=2&qt=/replication&file=segments_2&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
[junit4] 2> 286300 INFO (indexFetcher-1342-thread-1) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=false,bytesDownloaded=3206) : 0 secs (null bytes/sec) to MockDirectoryWrapper(RAMDirectory@1f6e08b7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@324c6cd5)
[junit4] 2> 286300 INFO (qtp1047709242-3348) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 286302 INFO (indexFetcher-1342-thread-1) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
[junit4] 2> 286304 INFO (indexFetcher-1342-thread-1) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@5f613631[tlog_replica_test_kill_leader_shard1_replica_t2] main]
[junit4] 2> 286305 INFO (searcherExecutor-1331-thread-1-processing-n:127.0.0.1:45071_solr x:tlog_replica_test_kill_leader_shard1_replica_t2 s:shard1 c:tlog_replica_test_kill_leader r:core_node4) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.SolrCore [tlog_replica_test_kill_leader_shard1_replica_t2] Registered new searcher Searcher@5f613631[tlog_replica_test_kill_leader_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):C1)))}
[junit4] 2> 286400 INFO (qtp1047709242-3348) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.S.Request [tlog_replica_test_kill_leader_shard1_replica_t2] webapp=/solr path=/select params={q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0
[junit4] 2> 286401 INFO (TEST-TestTlogReplica.testKillLeader-seed#[CF40D441C78FABDC]) [ ] o.a.s.c.ChaosMonkey monkey: kill jetty! 44195
[junit4] 2> 286402 INFO (TEST-TestTlogReplica.testKillLeader-seed#[CF40D441C78FABDC]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@8cb347e{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 286402 INFO (TEST-TestTlogReplica.testKillLeader-seed#[CF40D441C78FABDC]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1710448509
[junit4] 2> 286402 INFO (TEST-TestTlogReplica.testKillLeader-seed#[CF40D441C78FABDC]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 286402 INFO (TEST-TestTlogReplica.testKillLeader-seed#[CF40D441C78FABDC]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1a92d3ea: rootName = solr_44195, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@15d36a37
[junit4] 2> 286407 INFO (TEST-TestTlogReplica.testKillLeader-seed#[CF40D441C78FABDC]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 286407 INFO (TEST-TestTlogReplica.testKillLeader-seed#[CF40D441C78FABDC]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@276dd49f: rootName = solr_44195, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@62b1fddd
[junit4] 2> 286410 INFO (TEST-TestTlogReplica.testKillLeader-seed#[CF40D441C78FABDC]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 286410 INFO (TEST-TestTlogReplica.testKillLeader-seed#[CF40D441C78FABDC]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7e01e9dd: rootName = solr_44195, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@54bb8faa
[junit4] 2> 286410 INFO (TEST-TestTlogReplica.testKillLeader-seed#[CF40D441C78FABDC]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 286411 INFO (coreCloseExecutor-1344-thread-1) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.c.SolrCore [tlog_replica_test_kill_leader_shard1_replica_t1] CLOSING SolrCore org.apache.solr.core.SolrCore@262b547f
[junit4] 2> 286411 INFO (coreCloseExecutor-1344-thread-1) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_kill_leader.shard1.replica_t1, tag=640373887
[junit4] 2> 286411 INFO (coreCloseExecutor-1344-thread-1) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@21315445: rootName = solr_44195, domain = solr.core.tlog_replica_test_kill_leader.shard1.replica_t1, service url = null, agent id = null] for registry solr.core.tlog_replica_test_kill_leader.shard1.replica_t1 / com.codahale.metrics.MetricRegistry@39c8f6b0
[junit4] 2> 286412 INFO (zkCallback-456-thread-2-processing-n:127.0.0.1:44195_solr) [n:127.0.0.1:44195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 286412 INFO (zkCallback-457-thread-1-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 286412 INFO (zkCallback-456-thread-1-processing-n:127.0.0.1:44195_solr) [n:127.0.0.1:44195_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 286412 INFO (zkCallback-463-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 286412 INFO (zkCallback-457-thread-2-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [2])
[junit4] 2> 286419 INFO (coreCloseExecutor-1344-thread-1) [n:127.0.0.1:44195_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node3 x:tlog_replica_test_kill_leader_shard1_replica_t1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.tlog_replica_test_kill_leader.shard1.leader, tag=640373887
[junit4] 2> 286423 INFO (TEST-TestTlogReplica.testKillLeader-seed#[CF40D441C78FABDC]) [ ] o.a.s.c.Overseer Overseer (id=98723082613882885-127.0.0.1:44195_solr-n_0000000000) closing
[junit4] 2> 286423 INFO (OverseerStateUpdate-98723082613882885-127.0.0.1:44195_solr-n_0000000000) [n:127.0.0.1:44195_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:44195_solr
[junit4] 2> 286424 INFO (zkCallback-463-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 286424 INFO (zkCallback-457-thread-1-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:45071_solr
[junit4] 2> 286424 INFO (zkCallback-457-thread-3-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 286425 INFO (zkCallback-457-thread-1-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.Overseer Overseer (id=98723082613882886-127.0.0.1:45071_solr-n_0000000001) starting
[junit4] 2> 286425 INFO (zkCallback-457-thread-2-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 286435 INFO (zkCallback-457-thread-3-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [1])
[junit4] 2> 286435 INFO (zkCallback-457-thread-1-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [1])
[junit4] 2> 286435 INFO (zkCallback-463-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [1])
[junit4] 2> 287923 WARN (zkCallback-456-thread-2-processing-n:127.0.0.1:44195_solr) [n:127.0.0.1:44195_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 287924 INFO (TEST-TestTlogReplica.testKillLeader-seed#[CF40D441C78FABDC]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@72d0cade{/solr,null,UNAVAILABLE}
[junit4] 2> 287924 INFO (TEST-TestTlogReplica.testKillLeader-seed#[CF40D441C78FABDC]) [ ] o.a.s.c.ChaosMonkey monkey: stop jetty! 44195
[junit4] 2> 288925 INFO (zkCallback-457-thread-2-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:45071/solr/tlog_replica_test_kill_leader_shard1_replica_t2/
[junit4] 2> 288925 INFO (zkCallback-457-thread-2-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 288925 INFO (zkCallback-457-thread-2-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.SyncStrategy http://127.0.0.1:45071/solr/tlog_replica_test_kill_leader_shard1_replica_t2/ has no replicas
[junit4] 2> 288925 INFO (zkCallback-457-thread-2-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.ZkController tlog_replica_test_kill_leader_shard1_replica_t2 stopping background replication from leader
[junit4] 2> 288926 INFO (zkCallback-457-thread-2-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext New leader does not have old tlog to replay
[junit4] 2> 288928 INFO (zkCallback-457-thread-2-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr c:tlog_replica_test_kill_leader s:shard1 r:core_node4 x:tlog_replica_test_kill_leader_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:45071/solr/tlog_replica_test_kill_leader_shard1_replica_t2/ shard1
[junit4] 2> 288928 INFO (zkCallback-457-thread-3-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [1])
[junit4] 2> 288928 INFO (zkCallback-457-thread-2-processing-n:127.0.0.1:45071_solr) [n:127.0.0.1:45071_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_leader] has occurred - updating... (live nodes size: [1])
[junit4] 2> 288928 INFO (zkCallback-463-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_leader/state.json] for collection [tlog_replica_test_kill_lea
[...truncated too long message...]
ination.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@2dab5e74
[junit4] 2> 91869 INFO (jetty-closer-534-thread-3) [ ] o.a.s.c.Overseer Overseer (id=98723226606305289-127.0.0.1:34403_solr-n_0000000000) closing
[junit4] 2> 91869 INFO (coreCloseExecutor-198-thread-2) [n:127.0.0.1:40095_solr c:destination s:shard2 r:core_node4 x:destination_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.destination.shard2.replica_n2, tag=1403652865
[junit4] 2> 91869 INFO (coreCloseExecutor-198-thread-2) [n:127.0.0.1:40095_solr c:destination s:shard2 r:core_node4 x:destination_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@54d84e2a: rootName = solr_40095, domain = solr.core.destination.shard2.replica_n2, service url = null, agent id = null] for registry solr.core.destination.shard2.replica_n2 / com.codahale.metrics.MetricRegistry@6565e643
[junit4] 2> 91870 INFO (OverseerStateUpdate-98723226606305289-127.0.0.1:34403_solr-n_0000000000) [n:127.0.0.1:34403_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:34403_solr
[junit4] 2> 91871 INFO (jetty-closer-534-thread-3) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7dc48f73{/solr,null,UNAVAILABLE}
[junit4] 2> 91872 INFO (zkCallback-554-thread-2-processing-n:127.0.0.1:42381_solr) [n:127.0.0.1:42381_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 91872 INFO (zkCallback-549-thread-2-processing-n:127.0.0.1:35431_solr) [n:127.0.0.1:35431_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35431_solr
[junit4] 2> 91872 INFO (zkCallback-549-thread-1-processing-n:127.0.0.1:35431_solr) [n:127.0.0.1:35431_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 91875 INFO (zkCallback-555-thread-1-processing-n:127.0.0.1:40095_solr) [n:127.0.0.1:40095_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 91878 INFO (coreCloseExecutor-199-thread-2) [n:127.0.0.1:42381_solr c:destination s:shard1 r:core_node3 x:destination_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.destination.shard1.leader, tag=249189608
[junit4] 2> 91878 INFO (coreCloseExecutor-199-thread-1) [n:127.0.0.1:42381_solr c:workQueue s:shard1 r:core_node3 x:workQueue_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.workQueue.shard1.leader, tag=477816353
[junit4] 2> 91879 INFO (coreCloseExecutor-198-thread-2) [n:127.0.0.1:40095_solr c:destination s:shard2 r:core_node4 x:destination_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.destination.shard2.leader, tag=1403652865
[junit4] 2> 91879 INFO (coreCloseExecutor-198-thread-1) [n:127.0.0.1:40095_solr c:collection1_collection s:shard2 r:core_node4 x:collection1_collection_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1_collection.shard2.leader, tag=1866508668
[junit4] 2> 91880 INFO (coreCloseExecutor-200-thread-2) [n:127.0.0.1:35431_solr c:workQueue s:shard2 r:core_node4 x:workQueue_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.workQueue.shard2.replica_n2, tag=658286134
[junit4] 2> 91880 INFO (coreCloseExecutor-200-thread-2) [n:127.0.0.1:35431_solr c:workQueue s:shard2 r:core_node4 x:workQueue_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@417711e1: rootName = solr_35431, domain = solr.core.workQueue.shard2.replica_n2, service url = null, agent id = null] for registry solr.core.workQueue.shard2.replica_n2 / com.codahale.metrics.MetricRegistry@21fbeb19
[junit4] 2> 91882 INFO (zkCallback-549-thread-2-processing-n:127.0.0.1:35431_solr) [n:127.0.0.1:35431_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (1)
[junit4] 2> 91887 INFO (coreCloseExecutor-200-thread-2) [n:127.0.0.1:35431_solr c:workQueue s:shard2 r:core_node4 x:workQueue_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.workQueue.shard2.leader, tag=658286134
[junit4] 2> 91887 INFO (coreCloseExecutor-200-thread-3) [n:127.0.0.1:35431_solr c:mainCorpus s:shard1 r:core_node3 x:mainCorpus_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.mainCorpus.shard1.replica_n1, tag=1219474880
[junit4] 2> 91887 INFO (coreCloseExecutor-200-thread-3) [n:127.0.0.1:35431_solr c:mainCorpus s:shard1 r:core_node3 x:mainCorpus_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@14834e1: rootName = solr_35431, domain = solr.core.mainCorpus.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.mainCorpus.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@a58943a
[junit4] 2> 91895 INFO (coreCloseExecutor-200-thread-1) [n:127.0.0.1:35431_solr c:collection1_collection s:shard1 r:core_node3 x:collection1_collection_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1_collection.shard1.leader, tag=1300940088
[junit4] 2> 91895 INFO (coreCloseExecutor-200-thread-3) [n:127.0.0.1:35431_solr c:mainCorpus s:shard1 r:core_node3 x:mainCorpus_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.mainCorpus.shard1.leader, tag=1219474880
[junit4] 2> 91895 INFO (coreCloseExecutor-200-thread-3) [n:127.0.0.1:35431_solr c:mainCorpus s:shard1 r:core_node3 x:mainCorpus_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
[junit4] 2> 91895 INFO (coreCloseExecutor-200-thread-3) [n:127.0.0.1:35431_solr c:mainCorpus s:shard1 r:core_node3 x:mainCorpus_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@a0ff4a commitCommandVersion:0
[junit4] 2> 91917 WARN (zkCallback-549-thread-2-processing-n:127.0.0.1:35431_solr) [n:127.0.0.1:35431_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 91917 INFO (jetty-closer-534-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6798f711{/solr,null,UNAVAILABLE}
[junit4] 2> 93382 WARN (zkCallback-554-thread-2-processing-n:127.0.0.1:42381_solr) [n:127.0.0.1:42381_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 93382 INFO (jetty-closer-534-thread-4) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3c69bff{/solr,null,UNAVAILABLE}
[junit4] 2> 93384 WARN (zkCallback-555-thread-1-processing-n:127.0.0.1:40095_solr) [n:127.0.0.1:40095_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 93384 INFO (jetty-closer-534-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@928215c{/solr,null,UNAVAILABLE}
[junit4] 2> 93385 ERROR (SUITE-StreamExpressionTest-seed#[67874D4D39A483A3]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 93385 INFO (SUITE-StreamExpressionTest-seed#[67874D4D39A483A3]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:39657 39657
[junit4] 2> 94808 INFO (Thread-36) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:39657 39657
[junit4] 2> 94809 WARN (Thread-36) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 71 /solr/aliases.json
[junit4] 2> 6 /solr/configs/conf
[junit4] 2> 4 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 71 /solr/clusterprops.json
[junit4] 2> 71 /solr/clusterstate.json
[junit4] 2> 20 /solr/collections/destination/state.json
[junit4] 2> 20 /solr/collections/parallelDestinationCollection/state.json
[junit4] 2> 20 /solr/collections/workQueue/state.json
[junit4] 2> 20 /solr/collections/parallelDestinationCollection1/state.json
[junit4] 2> 20 /solr/collections/destinationCollection/state.json
[junit4] 2> 20 /solr/collections/mainCorpus/state.json
[junit4] 2> 10 /solr/collections/collection1_collection/state.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 71 /solr/live_nodes
[junit4] 2> 71 /solr/collections
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_67874D4D39A483A3-001
[junit4] 2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=en-PH, timezone=AST
[junit4] 2> NOTE: Linux 4.10.0-33-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=8,threads=1,free=200604040,total=523239424
[junit4] 2> NOTE: All tests run in this JVM: [JdbcTest, TestV1toV2ApiMapper, DocumentAnalysisResponseTest, TestPolicy, StreamExpressionParserTest, FieldValueEvaluatorTest, JettyWebappTest, ConcurrentUpdateSolrClientBuilderTest, ShardParamsTest, StreamExpressionTest]
[junit4] Completed [89/145 (1!)] on J2 in 75.24s, 113 tests, 1 error, 9 skipped <<< FAILURES!
[...truncated 45939 lines...]
[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk1.8.0_144) - Build # 497
- Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/497/
Java: 64bit/jdk1.8.0_144 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
3 tests failed.
FAILED: org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest
Error Message:
Timeout waiting for all live and active
Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
at __randomizedtesting.SeedInfo.seed([5E0ED2692CDFE35A:DD788D9BFAA6EDFB]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest(TestCloudRecovery.java:184)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.TestTlogReplica.testRecovery
Error Message:
Replica core_node3 not up to date after 0 seconds expected:<4> but was:<0>
Stack Trace:
java.lang.AssertionError: Replica core_node3 not up to date after 0 seconds expected:<4> but was:<0>
at __randomizedtesting.SeedInfo.seed([5E0ED2692CDFE35A:9FFEABC5018F29FD]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.apache.solr.cloud.TestTlogReplica.waitForNumDocsInAllReplicas(TestTlogReplica.java:743)
at org.apache.solr.cloud.TestTlogReplica.waitForNumDocsInAllReplicas(TestTlogReplica.java:731)
at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:538)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.TestTlogReplica.testOnlyLeaderIndexes
Error Message:
IndexWriter at leader must see updates
Stack Trace:
java.lang.AssertionError: IndexWriter at leader must see updates
at __randomizedtesting.SeedInfo.seed([5E0ED2692CDFE35A:420FAFE4597A9DC9]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.TestTlogReplica.testOnlyLeaderIndexes(TestTlogReplica.java:469)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 13170 lines...]
[junit4] Suite: org.apache.solr.cloud.TestCloudRecovery
[junit4] 2> 1634044 INFO (SUITE-TestCloudRecovery-seed#[5E0ED2692CDFE35A]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_5E0ED2692CDFE35A-001/init-core-data-001
[junit4] 2> 1634045 WARN (SUITE-TestCloudRecovery-seed#[5E0ED2692CDFE35A]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=30 numCloses=30
[junit4] 2> 1634045 INFO (SUITE-TestCloudRecovery-seed#[5E0ED2692CDFE35A]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 1634046 INFO (SUITE-TestCloudRecovery-seed#[5E0ED2692CDFE35A]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 1634047 INFO (SUITE-TestCloudRecovery-seed#[5E0ED2692CDFE35A]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_5E0ED2692CDFE35A-001/tempDir-001
[junit4] 2> 1634048 INFO (SUITE-TestCloudRecovery-seed#[5E0ED2692CDFE35A]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1634048 INFO (Thread-4576) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1634048 INFO (Thread-4576) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1634050 ERROR (Thread-4576) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1634148 INFO (SUITE-TestCloudRecovery-seed#[5E0ED2692CDFE35A]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:34639
[junit4] 2> 1634152 INFO (jetty-launcher-1935-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 1634152 INFO (jetty-launcher-1935-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 1634153 INFO (jetty-launcher-1935-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@481cb578{/solr,null,AVAILABLE}
[junit4] 2> 1634153 INFO (jetty-launcher-1935-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7fa24ebc{/solr,null,AVAILABLE}
[junit4] 2> 1634154 INFO (jetty-launcher-1935-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@36b45661{SSL,[ssl, http/1.1]}{127.0.0.1:37257}
[junit4] 2> 1634154 INFO (jetty-launcher-1935-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@2fed51dd{SSL,[ssl, http/1.1]}{127.0.0.1:46449}
[junit4] 2> 1634154 INFO (jetty-launcher-1935-thread-1) [ ] o.e.j.s.Server Started @1635637ms
[junit4] 2> 1634154 INFO (jetty-launcher-1935-thread-2) [ ] o.e.j.s.Server Started @1635637ms
[junit4] 2> 1634154 INFO (jetty-launcher-1935-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37257}
[junit4] 2> 1634154 INFO (jetty-launcher-1935-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46449}
[junit4] 2> 1634154 ERROR (jetty-launcher-1935-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1634154 ERROR (jetty-launcher-1935-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1634154 INFO (jetty-launcher-1935-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 7.1.0
[junit4] 2> 1634154 INFO (jetty-launcher-1935-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 7.1.0
[junit4] 2> 1634154 INFO (jetty-launcher-1935-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1634154 INFO (jetty-launcher-1935-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1634154 INFO (jetty-launcher-1935-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 1634154 INFO (jetty-launcher-1935-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 1634154 INFO (jetty-launcher-1935-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-27T00:29:54.725Z
[junit4] 2> 1634154 INFO (jetty-launcher-1935-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-27T00:29:54.725Z
[junit4] 2> 1634155 INFO (jetty-launcher-1935-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1634155 INFO (jetty-launcher-1935-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1634164 INFO (jetty-launcher-1935-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34639/solr
[junit4] 2> 1634164 INFO (jetty-launcher-1935-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:34639/solr
[junit4] 2> 1634186 INFO (jetty-launcher-1935-thread-2) [n:127.0.0.1:46449_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1634187 INFO (jetty-launcher-1935-thread-1) [n:127.0.0.1:37257_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1634187 INFO (jetty-launcher-1935-thread-2) [n:127.0.0.1:46449_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46449_solr
[junit4] 2> 1634187 INFO (jetty-launcher-1935-thread-1) [n:127.0.0.1:37257_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37257_solr
[junit4] 2> 1634187 INFO (jetty-launcher-1935-thread-2) [n:127.0.0.1:46449_solr ] o.a.s.c.Overseer Overseer (id=98728161746616325-127.0.0.1:46449_solr-n_0000000000) starting
[junit4] 2> 1634187 INFO (zkCallback-1946-thread-1-processing-n:127.0.0.1:46449_solr) [n:127.0.0.1:46449_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1634188 INFO (zkCallback-1947-thread-1-processing-n:127.0.0.1:37257_solr) [n:127.0.0.1:37257_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1634190 INFO (jetty-launcher-1935-thread-2) [n:127.0.0.1:46449_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46449_solr
[junit4] 2> 1634190 INFO (zkCallback-1947-thread-1-processing-n:127.0.0.1:37257_solr) [n:127.0.0.1:37257_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1634190 INFO (zkCallback-1946-thread-2-processing-n:127.0.0.1:46449_solr) [n:127.0.0.1:46449_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1634228 INFO (jetty-launcher-1935-thread-1) [n:127.0.0.1:37257_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37257.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13afd938
[junit4] 2> 1634229 INFO (jetty-launcher-1935-thread-2) [n:127.0.0.1:46449_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46449.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13afd938
[junit4] 2> 1634233 INFO (jetty-launcher-1935-thread-1) [n:127.0.0.1:37257_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37257.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13afd938
[junit4] 2> 1634233 INFO (jetty-launcher-1935-thread-2) [n:127.0.0.1:46449_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46449.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13afd938
[junit4] 2> 1634233 INFO (jetty-launcher-1935-thread-1) [n:127.0.0.1:37257_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37257.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13afd938
[junit4] 2> 1634233 INFO (jetty-launcher-1935-thread-2) [n:127.0.0.1:46449_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46449.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13afd938
[junit4] 2> 1634234 INFO (jetty-launcher-1935-thread-1) [n:127.0.0.1:37257_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_5E0ED2692CDFE35A-001/tempDir-001/node1/.
[junit4] 2> 1634234 INFO (jetty-launcher-1935-thread-2) [n:127.0.0.1:46449_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_5E0ED2692CDFE35A-001/tempDir-001/node2/.
[junit4] 2> 1634259 INFO (SUITE-TestCloudRecovery-seed#[5E0ED2692CDFE35A]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1634259 INFO (SUITE-TestCloudRecovery-seed#[5E0ED2692CDFE35A]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:34639/solr ready
[junit4] 2> 1634275 INFO (qtp1848102950-13998) [n:127.0.0.1:46449_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=0&collection.configName=config&maxShardsPerNode=2&name=collection1&nrtReplicas=0&action=CREATE&numShards=2&tlogReplicas=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1634276 INFO (OverseerThreadFactory-4651-thread-1-processing-n:127.0.0.1:46449_solr) [n:127.0.0.1:46449_solr ] o.a.s.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 1634379 INFO (OverseerStateUpdate-98728161746616325-127.0.0.1:46449_solr-n_0000000000) [n:127.0.0.1:46449_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1_shard1_replica_t1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:37257/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 1634380 INFO (OverseerStateUpdate-98728161746616325-127.0.0.1:46449_solr-n_0000000000) [n:127.0.0.1:46449_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1_shard1_replica_t2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:46449/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 1634380 INFO (OverseerStateUpdate-98728161746616325-127.0.0.1:46449_solr-n_0000000000) [n:127.0.0.1:46449_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"collection1_shard2_replica_t4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:37257/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 1634381 INFO (OverseerStateUpdate-98728161746616325-127.0.0.1:46449_solr-n_0000000000) [n:127.0.0.1:46449_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"collection1_shard2_replica_t6",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:46449/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 1634587 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t6&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 1634587 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t2&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 1634587 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1634622 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 1634623 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1634623 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t4&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 1634726 INFO (zkCallback-1947-thread-1-processing-n:127.0.0.1:37257_solr) [n:127.0.0.1:37257_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1634726 INFO (zkCallback-1946-thread-2-processing-n:127.0.0.1:46449_solr) [n:127.0.0.1:46449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1634726 INFO (zkCallback-1947-thread-2-processing-n:127.0.0.1:37257_solr) [n:127.0.0.1:37257_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1635628 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 1635630 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 1635631 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 1635632 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 1635636 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.IndexSchema [collection1_shard1_replica_t2] Schema name=minimal
[junit4] 2> 1635636 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.IndexSchema [collection1_shard1_replica_t1] Schema name=minimal
[junit4] 2> 1635637 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.IndexSchema [collection1_shard2_replica_t6] Schema name=minimal
[junit4] 2> 1635638 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1635638 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t1' using configuration from collection collection1, trusted=true
[junit4] 2> 1635638 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1635639 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t2' using configuration from collection collection1, trusted=true
[junit4] 2> 1635639 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37257.solr.core.collection1.shard1.replica_t1' (registry 'solr.core.collection1.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13afd938
[junit4] 2> 1635639 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1635639 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SolrCore [[collection1_shard1_replica_t1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_5E0ED2692CDFE35A-001/tempDir-001/node1/collection1_shard1_replica_t1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_5E0ED2692CDFE35A-001/tempDir-001/node1/./collection1_shard1_replica_t1/data/]
[junit4] 2> 1635639 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1635639 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t6' using configuration from collection collection1, trusted=true
[junit4] 2> 1635639 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46449.solr.core.collection1.shard1.replica_t2' (registry 'solr.core.collection1.shard1.replica_t2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13afd938
[junit4] 2> 1635639 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1635639 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrCore [[collection1_shard1_replica_t2] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_5E0ED2692CDFE35A-001/tempDir-001/node2/collection1_shard1_replica_t2], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_5E0ED2692CDFE35A-001/tempDir-001/node2/./collection1_shard1_replica_t2/data/]
[junit4] 2> 1635639 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_46449.solr.core.collection1.shard2.replica_t6' (registry 'solr.core.collection1.shard2.replica_t6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13afd938
[junit4] 2> 1635639 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1635639 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrCore [[collection1_shard2_replica_t6] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_5E0ED2692CDFE35A-001/tempDir-001/node2/collection1_shard2_replica_t6], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_5E0ED2692CDFE35A-001/tempDir-001/node2/./collection1_shard2_replica_t6/data/]
[junit4] 2> 1635640 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.IndexSchema [collection1_shard2_replica_t4] Schema name=minimal
[junit4] 2> 1635642 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1635642 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t4' using configuration from collection collection1, trusted=true
[junit4] 2> 1635642 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37257.solr.core.collection1.shard2.replica_t4' (registry 'solr.core.collection1.shard2.replica_t4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@13afd938
[junit4] 2> 1635642 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1635642 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrCore [[collection1_shard2_replica_t4] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_5E0ED2692CDFE35A-001/tempDir-001/node1/collection1_shard2_replica_t4], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.TestCloudRecovery_5E0ED2692CDFE35A-001/tempDir-001/node1/./collection1_shard2_replica_t4/data/]
[junit4] 2> 1635705 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1635705 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1635705 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1635706 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1635706 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1635706 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1635706 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1635706 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1635707 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.SolrIndexSearcher Opening [Searcher@35a0df16[collection1_shard2_replica_t6] main]
[junit4] 2> 1635708 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@429f62ac[collection1_shard1_replica_t1] main]
[junit4] 2> 1635709 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 1635709 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 1635709 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 1635709 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 1635710 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1635710 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1635710 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1635710 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1635710 INFO (searcherExecutor-4656-thread-1-processing-n:127.0.0.1:37257_solr x:collection1_shard1_replica_t1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SolrCore [collection1_shard1_replica_t1] Registered new searcher Searcher@429f62ac[collection1_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1635711 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579650589688594432
[junit4] 2> 1635711 INFO (searcherExecutor-4658-thread-1-processing-n:127.0.0.1:46449_solr x:collection1_shard2_replica_t6 s:shard2 c:collection1 r:core_node8) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrCore [collection1_shard2_replica_t6] Registered new searcher Searcher@35a0df16[collection1_shard2_replica_t6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1635711 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579650589688594432
[junit4] 2> 1635711 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1635711 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1635713 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@701bb409[collection1_shard1_replica_t2] main]
[junit4] 2> 1635714 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
[junit4] 2> 1635714 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 1635714 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 1635715 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1635715 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579650589692788736
[junit4] 2> 1635716 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1635716 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1635716 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
[junit4] 2> 1635717 INFO (searcherExecutor-4657-thread-1-processing-n:127.0.0.1:46449_solr x:collection1_shard1_replica_t2 s:shard1 c:collection1 r:core_node5) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrCore [collection1_shard1_replica_t2] Registered new searcher Searcher@701bb409[collection1_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1635717 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1635717 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1635719 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.SolrIndexSearcher Opening [Searcher@2d57e1f7[collection1_shard2_replica_t4] main]
[junit4] 2> 1635719 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 1635719 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 1635720 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1635720 INFO (searcherExecutor-4659-thread-1-processing-n:127.0.0.1:37257_solr x:collection1_shard2_replica_t4 s:shard2 c:collection1 r:core_node7) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrCore [collection1_shard2_replica_t4] Registered new searcher Searcher@2d57e1f7[collection1_shard2_replica_t4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1635721 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579650589699080192
[junit4] 2> 1635817 INFO (zkCallback-1947-thread-2-processing-n:127.0.0.1:37257_solr) [n:127.0.0.1:37257_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1635817 INFO (zkCallback-1947-thread-1-processing-n:127.0.0.1:37257_solr) [n:127.0.0.1:37257_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1635817 INFO (zkCallback-1946-thread-2-processing-n:127.0.0.1:46449_solr) [n:127.0.0.1:46449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1636214 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1636214 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1636214 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:37257/solr/collection1_shard1_replica_t1/
[junit4] 2> 1636214 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.PeerSync PeerSync: core=collection1_shard1_replica_t1 url=https://127.0.0.1:37257/solr START replicas=[https://127.0.0.1:46449/solr/collection1_shard1_replica_t2/] nUpdates=100
[junit4] 2> 1636216 INFO (qtp1848102950-14073) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1636217 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1636217 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1636217 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:46449/solr/collection1_shard2_replica_t6/
[junit4] 2> 1636217 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.PeerSync PeerSync: core=collection1_shard2_replica_t6 url=https://127.0.0.1:46449/solr START replicas=[https://127.0.0.1:37257/solr/collection1_shard2_replica_t4/] nUpdates=100
[junit4] 2> 1636219 INFO (qtp1278211665-13999) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1636515 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.PeerSync PeerSync: core=collection1_shard1_replica_t1 url=https://127.0.0.1:37257/solr DONE. We have no versions. sync failed.
[junit4] 2> 1636515 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 1636515 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 1636515 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1636515 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ZkController collection1_shard1_replica_t1 stopping background replication from leader
[junit4] 2> 1636516 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:37257/solr/collection1_shard1_replica_t1/ shard1
[junit4] 2> 1636517 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.PeerSync PeerSync: core=collection1_shard2_replica_t6 url=https://127.0.0.1:46449/solr DONE. We have no versions. sync failed.
[junit4] 2> 1636517 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 1636517 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 1636518 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1636518 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.ZkController collection1_shard2_replica_t6 stopping background replication from leader
[junit4] 2> 1636519 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:46449/solr/collection1_shard2_replica_t6/ shard2
[junit4] 2> 1636620 INFO (zkCallback-1947-thread-1-processing-n:127.0.0.1:37257_solr) [n:127.0.0.1:37257_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1636620 INFO (zkCallback-1946-thread-2-processing-n:127.0.0.1:46449_solr) [n:127.0.0.1:46449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1636620 INFO (zkCallback-1947-thread-2-processing-n:127.0.0.1:37257_solr) [n:127.0.0.1:37257_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1636667 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1636667 INFO (qtp1278211665-14003) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2045
[junit4] 2> 1636669 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1636670 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t6&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2083
[junit4] 2> 1636719 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ZkController collection1_shard1_replica_t2 starting background replication from leader
[junit4] 2> 1636719 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
[junit4] 2> 1636720 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
[junit4] 2> 1636720 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1636720 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Replica core_node3 is leader but it's state is down, skipping replication
[junit4] 2> 1636721 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t2&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2134
[junit4] 2> 1636723 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ZkController collection1_shard2_replica_t4 starting background replication from leader
[junit4] 2> 1636727 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
[junit4] 2> 1636728 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
[junit4] 2> 1636728 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1636728 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher Replica core_node8 is leader but it's state is down, skipping replication
[junit4] 2> 1636729 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t4&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2106
[junit4] 2> 1636731 INFO (qtp1848102950-13998) [n:127.0.0.1:46449_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 1636831 INFO (zkCallback-1947-thread-1-processing-n:127.0.0.1:37257_solr) [n:127.0.0.1:37257_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1636831 INFO (zkCallback-1946-thread-2-processing-n:127.0.0.1:46449_solr) [n:127.0.0.1:46449_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1636831 INFO (zkCallback-1947-thread-2-processing-n:127.0.0.1:37257_solr) [n:127.0.0.1:37257_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1637731 INFO (qtp1848102950-13998) [n:127.0.0.1:46449_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=0&collection.configName=config&maxShardsPerNode=2&name=collection1&nrtReplicas=0&action=CREATE&numShards=2&tlogReplicas=2&wt=javabin&version=2} status=0 QTime=3456
[junit4] 2> 1637731 INFO (SUITE-TestCloudRecovery-seed#[5E0ED2692CDFE35A]-worker) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
[junit4] 2> 1637732 INFO (SUITE-TestCloudRecovery-seed#[5E0ED2692CDFE35A]-worker) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
[junit4] 2> 1637749 INFO (TEST-TestCloudRecovery.leaderRecoverFromLogOnStartupTest-seed#[5E0ED2692CDFE35A]) [ ] o.a.s.SolrTestCaseJ4 ###Starting leaderRecoverFromLogOnStartupTest
[junit4] 2> 1637756 INFO (qtp1848102950-14006) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t2] webapp=/solr path=/update params={update.distrib=FROMLEADER&_version_=-1579650591828738048&distrib.from=https://127.0.0.1:37257/solr/collection1_shard1_replica_t1/&wt=javabin&version=2}{deleteByQuery=*:* (-1579650591828738048)} 0 0
[junit4] 2> 1637759 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t4] webapp=/solr path=/update params={update.distrib=FROMLEADER&_version_=-1579650591826640896&distrib.from=https://127.0.0.1:46449/solr/collection1_shard2_replica_t6/&wt=javabin&version=2}{deleteByQuery=*:* (-1579650591826640896)} 0 2
[junit4] 2> 1637760 INFO (qtp1278211665-14075) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t1] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:46449/solr/collection1_shard2_replica_t6/&wt=javabin&version=2}{deleteByQuery=*:* (-1579650591828738048)} 0 7
[junit4] 2> 1637760 INFO (qtp1848102950-14009) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t6] webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1579650591826640896)} 0 9
[junit4] 2> 1637761 INFO (qtp1278211665-14076) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579650591838175232,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1637761 INFO (qtp1278211665-14076) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1637762 INFO (qtp1278211665-14076) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1637762 INFO (qtp1278211665-14076) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:46449/solr/collection1_shard2_replica_t6/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 1637763 INFO (qtp1848102950-14004) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.TestInjection Start waiting for replica in sync with leader
[junit4] 2> 1637768 INFO (qtp1278211665-13999) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.TestInjection Start waiting for replica in sync with leader
[junit4] 2> 1637770 INFO (qtp1848102950-14002) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579650591847612416,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1637770 INFO (qtp1848102950-14002) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1637771 INFO (qtp1848102950-14002) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1637771 INFO (qtp1848102950-14002) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t6] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:46449/solr/collection1_shard2_replica_t6/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 1637773 INFO (qtp1278211665-14007) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=2
[junit4] 2> 1637773 INFO (qtp1848102950-14009) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1638277 INFO (qtp1848102950-13998) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1638277 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1638278 INFO (OverseerCollectionConfigSetProcessor-98728161746616325-127.0.0.1:46449_solr-n_0000000000) [n:127.0.0.1:46449_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1638781 INFO (qtp1848102950-13998) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1638781 INFO (qtp1278211665-14007) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1639285 INFO (qtp1278211665-14075) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1639285 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1639720 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Last replication failed, so I'll force replication
[junit4] 2> 1639720 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Updated masterUrl to https://127.0.0.1:37257/solr/collection1_shard1_replica_t1/
[junit4] 2> 1639720 INFO (qtp1278211665-14000) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1639721 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1639721 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1639721 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 1639721 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 1639721 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 1639721 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1639721 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@2d3fd229 commitCommandVersion:0
[junit4] 2> 1639728 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher Last replication failed, so I'll force replication
[junit4] 2> 1639728 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher Updated masterUrl to https://127.0.0.1:46449/solr/collection1_shard2_replica_t6/
[junit4] 2> 1639728 INFO (qtp1848102950-14073) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1639729 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1639729 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1639729 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 1639729 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 1639729 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 1639729 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1639729 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@76cde383 commitCommandVersion:0
[junit4] 2> 1639742 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@27c7b959[collection1_shard1_replica_t2] main]
[junit4] 2> 1639742 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1639742 INFO (searcherExecutor-4657-thread-1-processing-n:127.0.0.1:46449_solr x:collection1_shard1_replica_t2 s:shard1 c:collection1 r:core_node5) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrCore [collection1_shard1_replica_t2] Registered new searcher Searcher@27c7b959[collection1_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1639746 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.SolrIndexSearcher Opening [Searcher@7c3588f3[collection1_shard2_replica_t4] main]
[junit4] 2> 1639746 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1639747 INFO (searcherExecutor-4659-thread-1-processing-n:127.0.0.1:37257_solr x:collection1_shard2_replica_t4 s:shard2 c:collection1 r:core_node7) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrCore [collection1_shard2_replica_t4] Registered new searcher Searcher@7c3588f3[collection1_shard2_replica_t4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1639789 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1639789 INFO (qtp1278211665-14007) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1640293 INFO (qtp1278211665-14075) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1640293 INFO (qtp1848102950-14012) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1640797 INFO (qtp1278211665-14007) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1640797 INFO (qtp1848102950-14006) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1641301 INFO (qtp1848102950-14009) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1641301 INFO (qtp1278211665-14075) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1641804 INFO (qtp1278211665-14000) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1641805 INFO (qtp1848102950-14012) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1642308 INFO (qtp1848102950-14073) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1642308 INFO (qtp1278211665-14075) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1642720 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1642720 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1642720 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1642720 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 1642720 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's version: 1506472200292
[junit4] 2> 1642728 INFO (qtp1848102950-14006) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1642729 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1642729 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1642729 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 1642729 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher Slave's version: 1506472200300
[junit4] 2> 1642812 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1642813 INFO (qtp1848102950-14006) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1643316 INFO (qtp1278211665-14007) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1643316 INFO (qtp1848102950-14006) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1643820 INFO (qtp1278211665-13996) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1643821 INFO (qtp1848102950-14012) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1644322 INFO (qtp1278211665-14007) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1644322 INFO (qtp1848102950-14009) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1644826 INFO (qtp1278211665-14075) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1644826 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1645329 INFO (qtp1278211665-14076) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1645329 INFO (qtp1848102950-14012) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1645722 INFO (qtp1278211665-14000) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1645722 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1645722 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1645722 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 1645722 INFO (indexFetcher-4677-thread-1) [n:127.0.0.1:46449_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.IndexFetcher Slave's version: 1506472200292
[junit4] 2> 1645729 INFO (qtp1848102950-14006) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1645729 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1645729 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1645729 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 1645729 INFO (indexFetcher-4679-thread-1) [n:127.0.0.1:37257_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.IndexFetcher Slave's version: 1506472200300
[junit4] 2> 1645834 INFO (qtp1278211665-14076) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1645834 INFO (qtp1848102950-14006) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1646336 INFO (qtp1278211665-14075) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1646336 INFO (qtp1848102950-14010) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1646840 INFO (qtp1278211665-14076) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1646840 INFO (qtp1848102950-14006) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1647344 INFO (qtp1278211665-14007) [n:127.0.0.1:37257_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1647344 INFO (qtp1848102950-14002) [n:127.0.0.1:46449_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1647848 INFO (qtp1848102950-14009) [n:127.0.0.1:46449_solr c:coll
[...truncated too long message...]
3]
[junit4] 2> 2045186 WARN (updateExecutor-5560-thread-2-processing-n:127.0.0.1:41543_solr x:tlog_replica_test_recovery_shard1_replica_t1 s:shard1 c:tlog_replica_test_recovery r:core_node3) [n:127.0.0.1:41543_solr c:tlog_replica_test_recovery s:shard1 r:core_node3 x:tlog_replica_test_recovery_shard1_replica_t1] o.a.s.u.DefaultSolrCoreState Skipping recovery because Solr is shutdown
[junit4] 2> 2045186 INFO (jetty-closer-5527-thread-1) [ ] o.a.s.c.Overseer Overseer (id=98728182499835921-127.0.0.1:41543_solr-n_0000000004) closing
[junit4] 2> 2045186 INFO (OverseerStateUpdate-98728182499835921-127.0.0.1:41543_solr-n_0000000004) [n:127.0.0.1:41543_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:41543_solr
[junit4] 2> 2045187 WARN (zkCallback-5563-thread-3-processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 2045187 INFO (jetty-closer-5527-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@29b8d0bb{/solr,null,UNAVAILABLE}
[junit4] 2> 2045188 ERROR (SUITE-TestTlogReplica-seed#[5E0ED2692CDFE35A]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2045188 INFO (SUITE-TestTlogReplica-seed#[5E0ED2692CDFE35A]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35949 35949
[junit4] 2> 2051372 INFO (Thread-7197) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35949 35949
[junit4] 2> 2051374 WARN (Thread-7197) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 21 /solr/configs/conf
[junit4] 2> 8 /solr/aliases.json
[junit4] 2> 7 /solr/security.json
[junit4] 2> 3 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 27 /solr/collections/tlog_replica_test_out_of_order_db_qwith_in_place_updates/state.json
[junit4] 2> 25 /solr/collections/tlog_replica_test_kill_leader/state.json
[junit4] 2> 24 /solr/collections/tlog_replica_test_remove_leader/state.json
[junit4] 2> 24 /solr/collections/tlog_replica_test_create_delete/state.json
[junit4] 2> 23 /solr/collections/tlog_replica_test_add_remove_tlog_replica/state.json
[junit4] 2> 23 /solr/collections/tlog_replica_test_kill_tlog_replica/state.json
[junit4] 2> 22 /solr/collections/tlog_replica_test_basic_leader_election/state.json
[junit4] 2> 15 /solr/collections/tlog_replica_test_recovery/state.json
[junit4] 2> 15 /solr/collections/tlog_replica_test_real_time_get/state.json
[junit4] 2> 15 /solr/collections/tlog_replica_test_only_leader_indexes/state.json
[junit4] 2> 13 /solr/collections/tlog_replica_test_delete_by_id/state.json
[junit4] 2> 8 /solr/clusterprops.json
[junit4] 2> 8 /solr/clusterstate.json
[junit4] 2> 6 /solr/collections/tlog_replica_test_add_docs/state.json
[junit4] 2> 2 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard2/election/98728182499835915-core_node15-n_0000000002
[junit4] 2> 2 /solr/collections/tlog_replica_test_recovery/leader_elect/shard1/election/98728182499835915-core_node4-n_0000000000
[junit4] 2> 2 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard1/election/98728182499835915-core_node3-n_0000000000
[junit4] 2> 2 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard2/election/98728182499835921-core_node14-n_0000000001
[junit4] 2> 2 /solr/overseer_elect/election/98728182499835921-127.0.0.1:41543_solr-n_0000000004
[junit4] 2> 2 /solr/overseer_elect/election/98728182499835915-127.0.0.1:40689_solr-n_0000000002
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 8 /solr/collections
[junit4] 2> 4 /solr/overseer/queue
[junit4] 2> 4 /solr/overseer/collection-queue-work
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 2 /solr/overseer/queue-work
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.TestTlogReplica_5E0ED2692CDFE35A-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {foo=FSTOrd50, title_s=FSTOrd50, id=PostingsFormat(name=LuceneVarGapFixedInterval), foo_s=FSTOrd50}, docValues:{_version_=DocValuesFormat(name=Direct), id=DocValuesFormat(name=Asserting), inplace_updatable_int=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=1982, maxMBSortInHeap=5.497048573913103, sim=RandomSimilarity(queryNorm=true): {}, locale=ar-LY, timezone=America/Kentucky/Monticello
[junit4] 2> NOTE: Linux 4.10.0-33-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=8,threads=1,free=103775320,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [SuggesterTest, ShowFileRequestHandlerTest, TestQuerySenderListener, CircularListTest, DeleteLastCustomShardedReplicaTest, NumericFieldsTest, XsltUpdateRequestHandlerTest, DirectoryFactoryTest, TermVectorComponentDistributedTest, TestDistributedMissingSort, TestCollectionAPI, TestLMJelinekMercerSimilarityFactory, CachingDirectoryFactoryTest, TestRandomFlRTGCloud, StatsComponentTest, TestBinaryResponseWriter, SpatialHeatmapFacetsTest, TestCorePropertiesReload, TestSerializedLuceneMatchVersion, TestReRankQParserPlugin, ClusterStateTest, TestSearcherReuse, URLClassifyProcessorTest, AddSchemaFieldsUpdateProcessorFactoryTest, TestSolrCoreProperties, JvmMetricsTest, UnloadDistributedZkTest, TestQueryWrapperFilter, PolyFieldTest, SpatialRPTFieldTypeTest, ChaosMonkeySafeLeaderWithPullReplicasTest, ManagedSchemaRoundRobinCloudTest, TestDeleteCollectionOnDownNodes, TestAuthenticationFramework, MigrateRouteKeyTest, BlockJoinFacetRandomTest, TestSuggestSpellingConverter, TestSubQueryTransformer, PingRequestHandlerTest, TermsComponentTest, TestReload, ResponseLogComponentTest, TemplateUpdateProcessorTest, HdfsChaosMonkeyNothingIsSafeTest, ZkStateWriterTest, TestSchemaResource, TestInitQParser, PathHierarchyTokenizerFactoryTest, TestFastWriter, TestObjectReleaseTracker, SharedFSAutoReplicaFailoverTest, TestCloudNestedDocsSort, TestTolerantUpdateProcessorRandomCloud, LeaderElectionTest, TestCollectionAPIs, TestNRTOpen, DistributedFacetPivotSmallTest, DocumentBuilderTest, SolrTestCaseJ4Test, TestManagedSynonymFilterFactory, V2ApiIntegrationTest, TestSolrCloudWithSecureImpersonation, SolrCLIZkUtilsTest, SpellPossibilityIteratorTest, ExplicitHLLTest, RecoveryAfterSoftCommitTest, TestFieldTypeCollectionResource, TestHighFrequencyDictionaryFactory, BadIndexSchemaTest, TestRetrieveFieldsOptimizer, SignatureUpdateProcessorFactoryTest, BufferStoreTest, TestStandardQParsers, TestCloudSchemaless, TestFieldCacheWithThreads, TestZkAclsWithHadoopAuth, FastVectorHighlighterTest, TestBulkSchemaAPI, SolrJmxReporterTest, RecoveryZkTest, MoveReplicaHDFSUlogDirTest, ConjunctionSolrSpellCheckerTest, SolrGangliaReporterTest, HdfsTlogReplayBufferedWhileIndexingTest, TestSSLRandomization, DistributedVersionInfoTest, TestStressReorder, TestScoreJoinQPScore, TestEmbeddedSolrServerSchemaAPI, OverseerTest, TestExceedMaxTermLength, MinimalSchemaTest, TestSizeLimitedDistributedMap, HdfsNNFailoverTest, TestPointFields, UUIDUpdateProcessorFallbackTest, DeleteShardTest, DistributedFacetPivotLargeTest, TestDelegationWithHadoopAuth, TestStressLiveNodes, SimplePostToolTest, TestCloudJSONFacetJoinDomain, DirectUpdateHandlerTest, TestPolicyCloud, TestXmlQParser, TestExactStatsCache, MultiTermTest, TestAddFieldRealTimeGet, CopyFieldTest, TestLockTree, TestLegacyField, TestImplicitCoreProperties, TestSolrCloudSnapshots, TestRecovery, QueryResultKeyTest, NotRequiredUniqueKeyTest, TestAnalyzedSuggestions, CdcrRequestHandlerTest, TestSortByMinMaxFunction, TestCodecSupport, TestSimpleTextCodec, StressHdfsTest, BasicDistributedZkTest, BasicDistributedZk2Test, FullSolrCloudDistribCmdsTest, TestZkChroot, ShardRoutingCustomTest, DistributedSpellCheckComponentTest, ZkControllerTest, HardAutoCommitTest, TestGroupingSearch, TestIndexSearcher, HighlighterTest, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SpellCheckCollatorTest, SuggesterFSTTest, TestFoldingMultitermQuery, DocValuesTest, QueryEqualityTest, StatelessScriptUpdateProcessorFactoryTest, CacheHeaderTest, IndexBasedSpellCheckerTest, TestWriterPerf, PrimitiveFieldTypeTest, TestOmitPositions, RequestHandlersTest, TestIndexingPerformance, TestSolrQueryParser, RegexBoostProcessorTest, TestJmxIntegration, UpdateRequestProcessorFactoryTest, QueryParsingTest, UniqFieldsUpdateProcessorFactoryTest, TestPHPSerializedResponseWriter, CSVRequestHandlerTest, TestComponentsName, SearchHandlerTest, TestLFUCache, ChaosMonkeyNothingIsSafeTest, AssignTest, CdcrBootstrapTest, ConcurrentDeleteAndCreateCollectionTest, ConfigSetsAPITest, ConnectionManagerTest, CustomCollectionTest, DeleteStatusTest, DocValuesNotIndexedTest, ForceLeaderTest, NodeMutatorTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, OverseerStatusTest, RemoteQueryErrorTest, SolrCloudExampleTest, TestCloudPivotFacet, TestCloudPseudoReturnFields, TestCloudSearcherWarming, TestCollectionsAPIViaSolrCloudCluster, TestConfigSetsAPIExclusivity, TestConfigSetsAPIZkFailure, TestCryptoKeys, TestExclusionRuleCollectionAccess, TestLeaderElectionWithEmptyReplica, TestLeaderElectionZkExpiry, TestMiniSolrCloudClusterSSL, TestRebalanceLeaders, TestReplicaProperties, TestRequestForwarding, TestTlogReplica]
[junit4] Completed [560/732 (2!)] on J2 in 100.41s, 13 tests, 2 failures <<< FAILURES!
[...truncated 47305 lines...]
[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-9) - Build # 496 - Still
Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/496/
Java: 64bit/jdk-9 -XX:-UseCompressedOops -XX:+UseG1GC --illegal-access=deny
1 tests failed.
FAILED: org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test
Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node5:{"core":"c8n_1x3_lf_shard1_replica_n2","base_url":"http://127.0.0.1:45805","node_name":"127.0.0.1:45805_","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/16)={ "pullReplicas":"0", "replicationFactor":"1", "shards":{"shard1":{ "range":"80000000-7fffffff", "state":"active", "replicas":{ "core_node4":{ "core":"c8n_1x3_lf_shard1_replica_n1", "base_url":"http://127.0.0.1:34373", "node_name":"127.0.0.1:34373_", "state":"down", "type":"NRT"}, "core_node5":{ "core":"c8n_1x3_lf_shard1_replica_n2", "base_url":"http://127.0.0.1:45805", "node_name":"127.0.0.1:45805_", "state":"active", "type":"NRT", "leader":"true"}, "core_node6":{ "state":"down", "base_url":"http://127.0.0.1:37453", "core":"c8n_1x3_lf_shard1_replica_n3", "node_name":"127.0.0.1:37453_", "type":"NRT"}}}}, "router":{"name":"compositeId"}, "maxShardsPerNode":"1", "autoAddReplicas":"false", "nrtReplicas":"3", "tlogReplicas":"0"}
Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node5:{"core":"c8n_1x3_lf_shard1_replica_n2","base_url":"http://127.0.0.1:45805","node_name":"127.0.0.1:45805_","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/16)={
"pullReplicas":"0",
"replicationFactor":"1",
"shards":{"shard1":{
"range":"80000000-7fffffff",
"state":"active",
"replicas":{
"core_node4":{
"core":"c8n_1x3_lf_shard1_replica_n1",
"base_url":"http://127.0.0.1:34373",
"node_name":"127.0.0.1:34373_",
"state":"down",
"type":"NRT"},
"core_node5":{
"core":"c8n_1x3_lf_shard1_replica_n2",
"base_url":"http://127.0.0.1:45805",
"node_name":"127.0.0.1:45805_",
"state":"active",
"type":"NRT",
"leader":"true"},
"core_node6":{
"state":"down",
"base_url":"http://127.0.0.1:37453",
"core":"c8n_1x3_lf_shard1_replica_n3",
"node_name":"127.0.0.1:37453_",
"type":"NRT"}}}},
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"false",
"nrtReplicas":"3",
"tlogReplicas":"0"}
at __randomizedtesting.SeedInfo.seed([BA55801CAB16FDDC:3201BFC605EA9024]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:169)
at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:56)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
Build Log:
[...truncated 11801 lines...]
[junit4] Suite: org.apache.solr.cloud.LeaderFailoverAfterPartitionTest
[junit4] 2> 238129 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[BA55801CAB16FDDC]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/init-core-data-001
[junit4] 2> 238129 WARN (SUITE-LeaderFailoverAfterPartitionTest-seed#[BA55801CAB16FDDC]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
[junit4] 2> 238129 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[BA55801CAB16FDDC]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 238130 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[BA55801CAB16FDDC]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl="https://issues.apache.org/jira/browse/SOLR-5776")
[junit4] 2> 238130 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[BA55801CAB16FDDC]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
[junit4] 2> 238131 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 238131 INFO (Thread-462) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 238131 INFO (Thread-462) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 238132 ERROR (Thread-462) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 238231 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.ZkTestServer start zk server on port:44205
[junit4] 2> 238239 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 238240 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 238241 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 238242 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 238242 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 238243 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 238243 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 238244 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 238244 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 238245 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 238245 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 238247 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
[junit4] 2> 238328 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 238329 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4bd0a014{/,null,AVAILABLE}
[junit4] 2> 238330 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@7dfbc60a{HTTP/1.1,[http/1.1]}{127.0.0.1:36443}
[junit4] 2> 238331 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.Server Started @239725ms
[junit4] 2> 238331 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, solr.data.dir=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/tempDir-001/control/data, hostPort=34373, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/control-001/cores}
[junit4] 2> 238331 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 238331 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 238331 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 238331 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 238331 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T21:03:37.601398Z
[junit4] 2> 238333 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 238333 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/control-001/solr.xml
[junit4] 2> 238336 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 238338 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44205/solr
[junit4] 2> 238374 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 238374 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34373_
[junit4] 2> 238374 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.Overseer Overseer (id=98727350593257476-127.0.0.1:34373_-n_0000000000) starting
[junit4] 2> 238381 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34373_
[junit4] 2> 238381 INFO (zkCallback-224-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 238421 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 238430 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 238430 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 238432 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/control-001/cores
[junit4] 2> 238448 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 238448 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44205/solr ready
[junit4] 2> 238449 INFO (SocketProxy-Acceptor-34373) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=34420,localport=34373], receiveBufferSize:531000
[junit4] 2> 238450 INFO (SocketProxy-Acceptor-34373) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36443,localport=38306], receiveBufferSize=530904
[junit4] 2> 238452 INFO (qtp1546507956-1757) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:34373_&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 238463 INFO (OverseerThreadFactory-614-thread-1) [ ] o.a.s.c.CreateCollectionCmd Create collection control_collection
[junit4] 2> 238568 INFO (SocketProxy-Acceptor-34373) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=34424,localport=34373], receiveBufferSize:531000
[junit4] 2> 238568 INFO (SocketProxy-Acceptor-34373) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36443,localport=38310], receiveBufferSize=530904
[junit4] 2> 238569 INFO (qtp1546507956-1759) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 238569 INFO (qtp1546507956-1759) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 238673 INFO (zkCallback-224-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 239582 INFO (qtp1546507956-1759) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 239599 INFO (qtp1546507956-1759) [ ] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
[junit4] 2> 239659 INFO (qtp1546507956-1759) [ ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 239667 INFO (qtp1546507956-1759) [ ] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
[junit4] 2> 239667 INFO (qtp1546507956-1759) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 239668 INFO (qtp1546507956-1759) [ ] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 239668 INFO (qtp1546507956-1759) [ ] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/control-001/cores/control_collection_shard1_replica_n1/data/]
[junit4] 2> 239669 INFO (qtp1546507956-1759) [ ] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=53.5078125, floorSegmentMB=1.6025390625, forceMergeDeletesPctAllowed=28.268212800252712, segmentsPerTier=17.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 239671 WARN (qtp1546507956-1759) [ ] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 239698 INFO (qtp1546507956-1759) [ ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 239698 INFO (qtp1546507956-1759) [ ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 239699 INFO (qtp1546507956-1759) [ ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 239699 INFO (qtp1546507956-1759) [ ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 239700 INFO (qtp1546507956-1759) [ ] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=27.3857421875, floorSegmentMB=1.517578125, forceMergeDeletesPctAllowed=15.99246503078511, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 239700 INFO (qtp1546507956-1759) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@53919ab7[control_collection_shard1_replica_n1] main]
[junit4] 2> 239701 INFO (qtp1546507956-1759) [ ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 239701 INFO (qtp1546507956-1759) [ ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 239701 INFO (qtp1546507956-1759) [ ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 239702 INFO (searcherExecutor-617-thread-1) [ ] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@53919ab7[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 239702 INFO (qtp1546507956-1759) [ ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579637611138383872
[junit4] 2> 239705 INFO (qtp1546507956-1759) [ ] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 239705 INFO (qtp1546507956-1759) [ ] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 239705 INFO (qtp1546507956-1759) [ ] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:34373/control_collection_shard1_replica_n1/
[junit4] 2> 239705 INFO (qtp1546507956-1759) [ ] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 239705 INFO (qtp1546507956-1759) [ ] o.a.s.c.SyncStrategy http://127.0.0.1:34373/control_collection_shard1_replica_n1/ has no replicas
[junit4] 2> 239705 INFO (qtp1546507956-1759) [ ] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 239706 INFO (qtp1546507956-1759) [ ] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:34373/control_collection_shard1_replica_n1/ shard1
[junit4] 2> 239808 INFO (zkCallback-224-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 239857 INFO (qtp1546507956-1759) [ ] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 239858 INFO (qtp1546507956-1759) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1289
[junit4] 2> 239861 INFO (qtp1546507956-1757) [ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 239961 INFO (zkCallback-224-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 240465 INFO (OverseerCollectionConfigSetProcessor-98727350593257476-127.0.0.1:34373_-n_0000000000) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 240861 INFO (qtp1546507956-1757) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:34373_&wt=javabin&version=2} status=0 QTime=2409
[junit4] 2> 240868 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 240868 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44205/solr ready
[junit4] 2> 240869 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 240870 INFO (SocketProxy-Acceptor-34373) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=34432,localport=34373], receiveBufferSize:531000
[junit4] 2> 240870 INFO (SocketProxy-Acceptor-34373) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=36443,localport=38318], receiveBufferSize=530904
[junit4] 2> 240871 INFO (qtp1546507956-1757) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 240874 INFO (OverseerThreadFactory-614-thread-2) [ ] o.a.s.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 240875 WARN (OverseerThreadFactory-614-thread-2) [ ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
[junit4] 2> 241080 INFO (qtp1546507956-1757) [ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 241080 INFO (qtp1546507956-1757) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=208
[junit4] 2> 241159 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-1-001 of type NRT
[junit4] 2> 241160 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 241162 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7f33b678{/,null,AVAILABLE}
[junit4] 2> 241162 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3a7e4ddc{HTTP/1.1,[http/1.1]}{127.0.0.1:37495}
[junit4] 2> 241162 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.Server Started @242556ms
[junit4] 2> 241162 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, solrconfig=solrconfig.xml, solr.data.dir=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/tempDir-001/jetty1, hostPort=43579, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-1-001/cores}
[junit4] 2> 241162 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 241162 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 241162 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 241162 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 241162 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T21:03:40.432952Z
[junit4] 2> 241165 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 241165 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-1-001/solr.xml
[junit4] 2> 241168 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 241169 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44205/solr
[junit4] 2> 241173 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 241174 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 241174 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43579_
[junit4] 2> 241175 INFO (zkCallback-224-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 241175 INFO (zkCallback-236-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 241175 INFO (zkCallback-231-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 241215 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 241220 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 241220 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 241221 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-1-001/cores
[junit4] 2> 241234 INFO (qtp1546507956-1758) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:43579_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 241236 INFO (OverseerCollectionConfigSetProcessor-98727350593257476-127.0.0.1:34373_-n_0000000000) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 241236 INFO (OverseerThreadFactory-614-thread-3) [ ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:43579_ for creating new replica
[junit4] 2> 241237 INFO (SocketProxy-Acceptor-43579) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=57480,localport=43579], receiveBufferSize:531000
[junit4] 2> 241237 INFO (SocketProxy-Acceptor-43579) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37495,localport=39536], receiveBufferSize=530904
[junit4] 2> 241238 INFO (qtp1520957024-1805) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 241238 INFO (qtp1520957024-1805) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 241341 INFO (zkCallback-236-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 242252 INFO (qtp1520957024-1805) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 242269 INFO (qtp1520957024-1805) [ ] o.a.s.s.IndexSchema [collection1_shard2_replica_n41] Schema name=test
[junit4] 2> 242354 INFO (qtp1520957024-1805) [ ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 242362 INFO (qtp1520957024-1805) [ ] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n41' using configuration from collection collection1, trusted=true
[junit4] 2> 242362 INFO (qtp1520957024-1805) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n41' (registry 'solr.core.collection1.shard2.replica_n41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 242362 INFO (qtp1520957024-1805) [ ] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 242363 INFO (qtp1520957024-1805) [ ] o.a.s.c.SolrCore [[collection1_shard2_replica_n41] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-1-001/cores/collection1_shard2_replica_n41], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-1-001/cores/collection1_shard2_replica_n41/data/]
[junit4] 2> 242364 INFO (qtp1520957024-1805) [ ] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=53.5078125, floorSegmentMB=1.6025390625, forceMergeDeletesPctAllowed=28.268212800252712, segmentsPerTier=17.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 242366 WARN (qtp1520957024-1805) [ ] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 242393 INFO (qtp1520957024-1805) [ ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 242393 INFO (qtp1520957024-1805) [ ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 242394 INFO (qtp1520957024-1805) [ ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 242394 INFO (qtp1520957024-1805) [ ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 242395 INFO (qtp1520957024-1805) [ ] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=27.3857421875, floorSegmentMB=1.517578125, forceMergeDeletesPctAllowed=15.99246503078511, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 242395 INFO (qtp1520957024-1805) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@466a16e1[collection1_shard2_replica_n41] main]
[junit4] 2> 242396 INFO (qtp1520957024-1805) [ ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 242396 INFO (qtp1520957024-1805) [ ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 242396 INFO (qtp1520957024-1805) [ ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 242397 INFO (searcherExecutor-628-thread-1) [ ] o.a.s.c.SolrCore [collection1_shard2_replica_n41] Registered new searcher Searcher@466a16e1[collection1_shard2_replica_n41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 242397 INFO (qtp1520957024-1805) [ ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579637613964296192
[junit4] 2> 242400 INFO (qtp1520957024-1805) [ ] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 242400 INFO (qtp1520957024-1805) [ ] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 242400 INFO (qtp1520957024-1805) [ ] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:43579/collection1_shard2_replica_n41/
[junit4] 2> 242400 INFO (qtp1520957024-1805) [ ] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 242400 INFO (qtp1520957024-1805) [ ] o.a.s.c.SyncStrategy http://127.0.0.1:43579/collection1_shard2_replica_n41/ has no replicas
[junit4] 2> 242400 INFO (qtp1520957024-1805) [ ] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 242401 INFO (qtp1520957024-1805) [ ] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:43579/collection1_shard2_replica_n41/ shard2
[junit4] 2> 242502 INFO (zkCallback-236-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 242551 INFO (qtp1520957024-1805) [ ] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 242553 INFO (qtp1520957024-1805) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1314
[junit4] 2> 242554 INFO (qtp1546507956-1758) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:43579_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1319
[junit4] 2> 242632 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-2-001 of type NRT
[junit4] 2> 242632 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 242633 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@723aa652{/,null,AVAILABLE}
[junit4] 2> 242633 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@fed0899{HTTP/1.1,[http/1.1]}{127.0.0.1:44539}
[junit4] 2> 242633 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.Server Started @244028ms
[junit4] 2> 242634 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, solrconfig=solrconfig.xml, solr.data.dir=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/tempDir-001/jetty2, hostPort=37453, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-2-001/cores, replicaType=NRT}
[junit4] 2> 242634 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 242634 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 242634 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 242634 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 242634 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T21:03:41.904600Z
[junit4] 2> 242636 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 242636 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-2-001/solr.xml
[junit4] 2> 242639 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 242641 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44205/solr
[junit4] 2> 242645 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 242646 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 242646 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37453_
[junit4] 2> 242647 INFO (zkCallback-231-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 242647 INFO (zkCallback-236-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 242647 INFO (zkCallback-242-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 242647 INFO (zkCallback-224-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 242677 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 242683 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 242683 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 242684 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-2-001/cores
[junit4] 2> 242703 INFO (qtp1546507956-1753) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:37453_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 242704 INFO (OverseerCollectionConfigSetProcessor-98727350593257476-127.0.0.1:34373_-n_0000000000) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 242704 INFO (OverseerThreadFactory-614-thread-4) [ ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:37453_ for creating new replica
[junit4] 2> 242705 INFO (SocketProxy-Acceptor-37453) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=38796,localport=37453], receiveBufferSize:531000
[junit4] 2> 242708 INFO (SocketProxy-Acceptor-37453) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=44539,localport=58330], receiveBufferSize=530904
[junit4] 2> 242709 INFO (qtp2097374211-1837) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 242710 INFO (qtp2097374211-1837) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 242812 INFO (zkCallback-242-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 242812 INFO (zkCallback-236-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 243721 INFO (qtp2097374211-1837) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 243740 INFO (qtp2097374211-1837) [ ] o.a.s.s.IndexSchema [collection1_shard1_replica_n43] Schema name=test
[junit4] 2> 243813 INFO (qtp2097374211-1837) [ ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 243822 INFO (qtp2097374211-1837) [ ] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n43' using configuration from collection collection1, trusted=true
[junit4] 2> 243822 INFO (qtp2097374211-1837) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n43' (registry 'solr.core.collection1.shard1.replica_n43') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 243822 INFO (qtp2097374211-1837) [ ] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 243822 INFO (qtp2097374211-1837) [ ] o.a.s.c.SolrCore [[collection1_shard1_replica_n43] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-2-001/cores/collection1_shard1_replica_n43], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-2-001/cores/collection1_shard1_replica_n43/data/]
[junit4] 2> 243824 INFO (qtp2097374211-1837) [ ] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=53.5078125, floorSegmentMB=1.6025390625, forceMergeDeletesPctAllowed=28.268212800252712, segmentsPerTier=17.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 243825 WARN (qtp2097374211-1837) [ ] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 243854 INFO (qtp2097374211-1837) [ ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 243854 INFO (qtp2097374211-1837) [ ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 243856 INFO (qtp2097374211-1837) [ ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 243856 INFO (qtp2097374211-1837) [ ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 243856 INFO (qtp2097374211-1837) [ ] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=27.3857421875, floorSegmentMB=1.517578125, forceMergeDeletesPctAllowed=15.99246503078511, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 243856 INFO (qtp2097374211-1837) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@2218d77f[collection1_shard1_replica_n43] main]
[junit4] 2> 243857 INFO (qtp2097374211-1837) [ ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 243857 INFO (qtp2097374211-1837) [ ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 243858 INFO (qtp2097374211-1837) [ ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 243858 INFO (searcherExecutor-639-thread-1) [ ] o.a.s.c.SolrCore [collection1_shard1_replica_n43] Registered new searcher Searcher@2218d77f[collection1_shard1_replica_n43] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 243858 INFO (qtp2097374211-1837) [ ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579637615496265728
[junit4] 2> 243861 INFO (qtp2097374211-1837) [ ] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 243861 INFO (qtp2097374211-1837) [ ] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 243861 INFO (qtp2097374211-1837) [ ] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:37453/collection1_shard1_replica_n43/
[junit4] 2> 243861 INFO (qtp2097374211-1837) [ ] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 243861 INFO (qtp2097374211-1837) [ ] o.a.s.c.SyncStrategy http://127.0.0.1:37453/collection1_shard1_replica_n43/ has no replicas
[junit4] 2> 243861 INFO (qtp2097374211-1837) [ ] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 243862 INFO (qtp2097374211-1837) [ ] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:37453/collection1_shard1_replica_n43/ shard1
[junit4] 2> 243964 INFO (zkCallback-236-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 243964 INFO (zkCallback-242-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 244013 INFO (qtp2097374211-1837) [ ] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 244015 INFO (qtp2097374211-1837) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1305
[junit4] 2> 244018 INFO (qtp1546507956-1753) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:37453_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1314
[junit4] 2> 244117 INFO (zkCallback-242-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 244117 INFO (zkCallback-236-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 244119 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-3-001 of type NRT
[junit4] 2> 244119 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 244120 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@16d1ecc4{/,null,AVAILABLE}
[junit4] 2> 244120 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@7d60e2ee{HTTP/1.1,[http/1.1]}{127.0.0.1:38533}
[junit4] 2> 244121 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.Server Started @245515ms
[junit4] 2> 244121 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, solrconfig=solrconfig.xml, solr.data.dir=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/tempDir-001/jetty3, hostPort=45805, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-3-001/cores, replicaType=NRT}
[junit4] 2> 244121 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 244121 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 244121 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 244121 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 244121 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T21:03:43.391661Z
[junit4] 2> 244124 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 244124 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-3-001/solr.xml
[junit4] 2> 244127 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 244128 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44205/solr
[junit4] 2> 244133 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 244133 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 244134 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45805_
[junit4] 2> 244134 INFO (zkCallback-231-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 244134 INFO (zkCallback-236-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 244135 INFO (zkCallback-248-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 244135 INFO (zkCallback-224-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 244135 INFO (zkCallback-242-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 244164 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 244170 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 244170 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 244171 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-3-001/cores
[junit4] 2> 244195 INFO (qtp1546507956-1754) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:45805_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 244196 INFO (OverseerCollectionConfigSetProcessor-98727350593257476-127.0.0.1:34373_-n_0000000000) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 244196 INFO (OverseerThreadFactory-614-thread-5) [ ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:45805_ for creating new replica
[junit4] 2> 244197 INFO (SocketProxy-Acceptor-45805) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=56056,localport=45805], receiveBufferSize:531000
[junit4] 2> 244197 INFO (SocketProxy-Acceptor-45805) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=38533,localport=41822], receiveBufferSize=530904
[junit4] 2> 244198 INFO (qtp1523545027-1870) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n45&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 244199 INFO (qtp1523545027-1870) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 244302 INFO (zkCallback-242-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 244302 INFO (zkCallback-248-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 244302 INFO (zkCallback-236-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 245214 INFO (qtp1523545027-1870) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 245229 INFO (qtp1523545027-1870) [ ] o.a.s.s.IndexSchema [collection1_shard2_replica_n45] Schema name=test
[junit4] 2> 245289 INFO (qtp1523545027-1870) [ ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 245300 INFO (qtp1523545027-1870) [ ] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n45' using configuration from collection collection1, trusted=true
[junit4] 2> 245300 INFO (qtp1523545027-1870) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n45' (registry 'solr.core.collection1.shard2.replica_n45') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@75027c81
[junit4] 2> 245300 INFO (qtp1523545027-1870) [ ] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 245300 INFO (qtp1523545027-1870) [ ] o.a.s.c.SolrCore [[collection1_shard2_replica_n45] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-3-001/cores/collection1_shard2_replica_n45], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001/shard-3-001/cores/collection1_shard2_replica_n45/data/]
[junit4] 2> 245302 INFO (qtp1523545027-1870) [ ] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=20, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=53.5078125, floorSegmentMB=1.6025390625, forceMergeDeletesPctAllowed=28.268212800252712, segmentsPerTier=17.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 245303 WARN (qtp1523545027-1870) [ ] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 245335 INFO (qtp1523545027-1870) [ ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 245335 INFO (qtp1523545027-1870) [ ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 245336 INFO (qtp1523545027-1870) [ ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 245336 INFO (qtp1523545027-1870) [ ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 245337 INFO (qtp1523545027-1870) [ ] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=25, maxMergeAtOnceExplicit=48, maxMergedSegmentMB=27.3857421875, floorSegmentMB=1.517578125, forceMergeDeletesPctAllowed=15.99246503078511, segmentsPerTier=46.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 245337 INFO (qtp1523545027-1870) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@12055ec7[collection1_shard2_replica_n45] main]
[junit4] 2> 245338 INFO (qtp1523545027-1870) [ ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 245338 INFO (qtp1523545027-1870) [ ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 245339 INFO (qtp1523545027-1870) [ ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 245339 INFO (qtp1523545027-1870) [ ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579637617049206784
[junit4] 2> 245339 INFO (searcherExecutor-650-thread-1) [ ] o.a.s.c.SolrCore [collection1_shard2_replica_n45] Registered new searcher Searcher@12055ec7[collection1_shard2_replica_n45] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 245341 INFO (qtp1523545027-1870) [ ] o.a.s.c.ZkController Core needs to recover:collection1_shard2_replica_n45
[junit4] 2> 245341 INFO (updateExecutor-245-thread-1) [ ] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 245341 INFO (qtp1523545027-1870) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n45&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1142
[junit4] 2> 245341 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 245341 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 245342 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1_shard2_replica_n45]
[junit4] 2> 245342 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 245342 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.c.RecoveryStrategy Publishing state of core [collection1_shard2_replica_n45] as recovering, leader is [http://127.0.0.1:43579/collection1_shard2_replica_n41/] and I am [http://127.0.0.1:45805/collection1_shard2_replica_n45/]
[junit4] 2> 245343 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.c.RecoveryStrategy Sending prep recovery command to [http://127.0.0.1:43579]; [WaitForState: action=PREPRECOVERY&core=collection1_shard2_replica_n41&nodeName=127.0.0.1:45805_&coreNodeName=core_node46&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
[junit4] 2> 245343 INFO (qtp1546507956-1754) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:45805_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1148
[junit4] 2> 245343 INFO (SocketProxy-Acceptor-43579) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=57514,localport=43579], receiveBufferSize:531000
[junit4] 2> 245344 INFO (SocketProxy-Acceptor-43579) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37495,localport=39570], receiveBufferSize=530904
[junit4] 2> 245345 INFO (qtp1520957024-1807) [ ] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node46, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true, maxTime: 183 s
[junit4] 2> 245345 INFO (qtp1520957024-1807) [ ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard2, thisCore=collection1_shard2_replica_n41, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=down, localState=active, nodeName=127.0.0.1:45805_, coreNodeName=core_node46, onlyIfActiveCheckResult=false, nodeProps: core_node46:{"core":"collection1_shard2_replica_n45","base_url":"http://127.0.0.1:45805","node_name":"127.0.0.1:45805_","state":"down","type":"NRT"}
[junit4] 2> 245346 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.SolrTestCaseJ4 ###Starting test
[junit4] 2> 245346 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Wait for recoveries to finish - wait 30000 for each attempt
[junit4] 2> 245346 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30000
[junit4] 2> 245444 INFO (zkCallback-236-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 245444 INFO (zkCallback-242-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 245444 INFO (zkCallback-248-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 246197 INFO (OverseerCollectionConfigSetProcessor-98727350593257476-127.0.0.1:34373_-n_0000000000) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000008 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 246345 INFO (qtp1520957024-1807) [ ] o.a.s.h.a.PrepRecoveryOp In WaitForState(recovering): collection=collection1, shard=shard2, thisCore=collection1_shard2_replica_n41, leaderDoesNotNeedRecovery=false, isLeader? true, live=true, checkLive=true, currentState=recovering, localState=active, nodeName=127.0.0.1:45805_, coreNodeName=core_node46, onlyIfActiveCheckResult=false, nodeProps: core_node46:{"core":"collection1_shard2_replica_n45","base_url":"http://127.0.0.1:45805","node_name":"127.0.0.1:45805_","state":"recovering","type":"NRT"}
[junit4] 2> 246346 INFO (qtp1520957024-1807) [ ] o.a.s.h.a.PrepRecoveryOp Waited coreNodeName: core_node46, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4] 2> 246346 INFO (qtp1520957024-1807) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:45805_&onlyIfLeaderActive=true&core=collection1_shard2_replica_n41&coreNodeName=core_node46&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=1000
[junit4] 2> 246846 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.c.RecoveryStrategy Attempting to PeerSync from [http://127.0.0.1:43579/collection1_shard2_replica_n41/] - recoveringAfterStartup=[true]
[junit4] 2> 246847 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.u.PeerSync PeerSync: core=collection1_shard2_replica_n45 url=http://127.0.0.1:45805 START replicas=[http://127.0.0.1:43579/collection1_shard2_replica_n41/] nUpdates=100
[junit4] 2> 246848 INFO (SocketProxy-Acceptor-43579) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=57526,localport=43579], receiveBufferSize:531000
[junit4] 2> 246848 INFO (SocketProxy-Acceptor-43579) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37495,localport=39582], receiveBufferSize=530904
[junit4] 2> 246851 INFO (qtp1520957024-1807) [ ] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 246851 INFO (qtp1520957024-1807) [ ] o.a.s.c.S.Request [collection1_shard2_replica_n41] webapp= path=/get params={distrib=false&qt=/get&getFingerprint=9223372036854775807&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 246852 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.u.IndexFingerprint IndexFingerprint millis:0.0 result:{maxVersionSpecified=9223372036854775807, maxVersionEncountered=0, maxInHash=0, versionsHash=0, numVersions=0, numDocs=0, maxDoc=0}
[junit4] 2> 246852 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.u.PeerSync We are already in sync. No need to do a PeerSync
[junit4] 2> 246852 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 246852 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 246853 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 246853 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.c.RecoveryStrategy PeerSync stage of recovery was successful.
[junit4] 2> 246853 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.c.RecoveryStrategy Replaying updates buffered during PeerSync.
[junit4] 2> 246853 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.c.RecoveryStrategy No replay needed.
[junit4] 2> 246853 INFO (recoveryExecutor-246-thread-1) [ ] o.a.s.c.RecoveryStrategy Registering as Active after recovery.
[junit4] 2> 246957 INFO (zkCallback-242-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 246958 INFO (zkCallback-236-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 246958 INFO (zkCallback-248-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 247348 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
[junit4] 2> 247350 INFO (qtp1546507956-1758) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579637619157893120,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 247351 INFO (qtp1546507956-1758) [ ] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 247351 INFO (qtp1546507956-1758) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 247351 INFO (qtp1546507956-1758) [ ] o.a.s.u.p.LogUpdateProcessorFactory [control_collection_shard1_replica_n1] webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 1
[junit4] 2> 247353 INFO (SocketProxy-Acceptor-43579) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=57532,localport=43579], receiveBufferSize:531000
[junit4] 2> 247354 INFO (SocketProxy-Acceptor-43579) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37495,localport=39588], receiveBufferSize=530904
[junit4] 2> 247358 INFO (SocketProxy-Acceptor-45805) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=56084,localport=45805], receiveBufferSize:531000
[junit4] 2> 247359 INFO (SocketProxy-Acceptor-37453) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=38842,localport=37453], receiveBufferSize:531000
[junit4] 2> 247359 INFO (SocketProxy-Acceptor-43579) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=57538,localport=43579], receiveBufferSize:531000
[junit4] 2> 247360 INFO (SocketProxy-Acceptor-37453) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=44539,localport=58380], receiveBufferSize=530904
[junit4] 2> 247360 INFO (SocketProxy-Acceptor-45805) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=38533,localport=41854], receiveBufferSize=530904
[junit4] 2> 247362 INFO (SocketProxy-Acceptor-43579) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37495,localport=39600], receiveBufferSize=530904
[junit4] 2> 247364 INFO (qtp1520957024-1807) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579637619172573184,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 247364 INFO (qtp2097374211-1839) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579637619172573184,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 247364 INFO (qtp1523545027-1872) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579637619172573184,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 247364 INFO (qtp1520957024-1807) [ ] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 247365 INFO (qtp2097374211-1839) [ ] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 247365 INFO (qtp1523545027-1872) [ ] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 247366 INFO (qtp2097374211-1839) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 247366 INFO (qtp1520957024-1807) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 247366 INFO (qtp2097374211-1839) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n43] webapp= path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:43579/collection1_shard2_replica_n41/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 2
[junit4] 2> 247366 INFO (qtp1520957024-1807) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n41] webapp= path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:43579/collection1_shard2_replica_n41/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 2
[junit4] 2> 247366 INFO (qtp1523545027-1872) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 247367 INFO (qtp1523545027-1872) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n45] webapp= path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:43579/collection1_shard2_replica_n41/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 2
[junit4] 2> 247369 INFO (qtp1520957024-1801) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n41] webapp= path=/update params={_stateVer_=collection1:8&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 13
[junit4] 2> 247372 INFO (SocketProxy-Acceptor-43579) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=57548,localport=43579], receiveBufferSize:531000
[junit4] 2> 247374 INFO (SocketProxy-Acceptor-43579) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=37495,localport=39604], receiveBufferSize=530904
[junit4] 2> 247377 INFO (qtp1520957024-1805) [ ] o.a.s.c.S.Request [collection1_shard2_replica_n41] webapp= path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=2
[junit4] 2> 247378 INFO (SocketProxy-Acceptor-45805) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=56100,localport=45805], receiveBufferSize:531000
[junit4] 2> 247380 INFO (SocketProxy-Acceptor-45805) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=38533,localport=41866], receiveBufferSize=530904
[junit4] 2> 247381 INFO (qtp1523545027-1870) [ ] o.a.s.c.S.Request [collection1_shard2_replica_n45] webapp= path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 247383 INFO (SocketProxy-Acceptor-37453) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=38860,localport=37453], receiveBufferSize:531000
[junit4] 2> 247383 INFO (SocketProxy-Acceptor-37453) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=44539,localport=58394], receiveBufferSize=530904
[junit4] 2> 247385 INFO (qtp2097374211-1837) [ ] o.a.s.c.S.Request [collection1_shard1_replica_n43] webapp= path=/select params={q=*:*&distrib=false&tests=checkShardConsistency&rows=0&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 249388 INFO (SocketProxy-Acceptor-37453) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=38868,localport=37453], receiveBufferSize:531000
[junit4] 2> 249389 INFO (SocketProxy-Acceptor-37453) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=44539,localport=58402], receiveBufferSize=530904
[junit4] 2> 249390 INFO (qtp2097374211-1839) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&maxShardsPerNode=1&collection.configName=conf1&nrtReplicas=3&name=c8n_1x3_lf&action=CREATE&numShards=1&tlogReplicas=0&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 249392 INFO (OverseerThreadFactory-614-thread-5) [ ] o.a.s.c.CreateCollectionCmd Create collection c8n_1x3_lf
[junit4] 2> 249505 INFO (qtp1546507956-1752) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=c8n_1x3_lf_shard1_replica_n1&action=CREATE&numShards=1&collection=c8n_1x3_lf&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 249506 INFO (qtp1523545027-1871) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=c8n_1x3_lf_shard1_replica_n2&action=CREATE&numShards=1&collection=c8n_1x3_lf&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 249510 INFO (qtp2097374211-1838) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=c8n_1x3_lf_shard1_replica_n3&action=CREATE&numShards=1&collection=c8n_1x3_lf&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 249615 INFO (zkCallback-248-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/c8n_1x3_lf/state.json] for collection [c8n_1x3_lf] has occurred - updating... (live nodes size: [4])
[junit4] 2> 249615 INFO (zkCallback-224-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/c8n_1x3_lf/state.json] for collection [c8n_1x3_lf] has occurred - updating... (live nodes size: [4])
[junit4] 2> 249615 INFO (zkCallback-242-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/c8n_1x3_lf/state.json] for collection [c8n_1x3_lf] has occurred - updating... (live nodes size: [4])
[junit4] 2> 250521 INFO (qtp1546507956-1752) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 250524 INFO (qtp1523545027-1871) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 250525 INFO (qtp2097374211-1838) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 250538 INFO (qtp1546507956-1752) [ ] o.a.s.s.IndexSchema [c8n_1x3_lf_shard1_replica_n1] Schema name=test
[junit4] 2> 250540 INFO (qtp2097374211-1838) [ ] o.a.s.s.IndexSchema [c8n_1x3_lf_shard1_replica_n3] Schema name=test
[junit4] 2> 250542 INFO (qtp1523545027-1871) [ ] o.a.s.s.IndexSchema [c8n_1x3_lf_shard1_replica_n2] Schema name=test
[junit4] 2> 250640 INFO (qtp2097374211-1838) [ ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 250641 INFO (qtp1546507956-1752) [ ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 250743 INFO (qtp1523545027-1871) [ ] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 250749 INFO (qtp1546507956-1752) [ ] o.a.s.c.CoreContainer Creating SolrCore 'c8n_1x3_lf_shard1_replica_n1' using configuration from collection c8n_1x3_lf, trusted=true
[junit4] 2> 250750 INFO (qtp1546507956-1752) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.c8n_1x3_lf.shard1.replica_n1' (
[...truncated too long message...]
annelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2>
[junit4] 2> 366908 INFO (qtp1523545027-1868) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:37453_&onlyIfLeaderActive=true&core=c8n_1x3_lf_shard1_replica_n2&coreNodeName=core_node6&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=400 QTime=95023
[junit4] 2> 368843 WARN (zkCallback-248-thread-3) [ ] o.a.s.c.SyncStrategy Closed, skipping sync up.
[junit4] 2> 368843 INFO (zkCallback-248-thread-3) [ ] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 368843 INFO (zkCallback-248-thread-3) [ ] o.a.s.c.SolrCore [collection1_shard2_replica_n45] CLOSING SolrCore org.apache.solr.core.SolrCore@7099e28
[junit4] 2> 368843 INFO (zkCallback-248-thread-3) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1.shard2.replica_n45, tag=118070824
[junit4] 2> 368843 INFO (zkCallback-248-thread-3) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4de3d459: rootName = null, domain = solr.core.collection1.shard2.replica_n45, service url = null, agent id = null] for registry solr.core.collection1.shard2.replica_n45 / com.codahale.metrics.MetricRegistry@3b918bc9
[junit4] 2> 368849 INFO (zkCallback-248-thread-3) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=118070824
[junit4] 2> 368850 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@7d60e2ee{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 368851 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@16d1ecc4{/,null,UNAVAILABLE}
[junit4] 2> 368851 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 368851 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44205 44205
[junit4] 2> 374627 INFO (Thread-462) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44205 44205
[junit4] 2> 374628 WARN (Thread-462) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/aliases.json
[junit4] 2> 4 /solr/security.json
[junit4] 2> 4 /solr/configs/conf1
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/clusterprops.json
[junit4] 2> 6 /solr/clusterstate.json
[junit4] 2> 3 /solr/collections/collection1/state.json
[junit4] 2> 3 /solr/collections/c8n_1x3_lf/state.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/live_nodes
[junit4] 2> 6 /solr/collections
[junit4] 2> 3 /solr/overseer/queue
[junit4] 2> 3 /solr/overseer/collection-queue-work
[junit4] 2> 3 /solr/overseer/queue-work
[junit4] 2>
[junit4] 2> 374628 WARN (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.SocketProxy Closing 6 connections to: http://127.0.0.1:45805/, target: http://127.0.0.1:38533/
[junit4] 2> 374628 WARN (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.SocketProxy Closing 0 connections to: http://127.0.0.1:37453/, target: http://127.0.0.1:44539/
[junit4] 2> 374628 WARN (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.SocketProxy Closing 0 connections to: http://127.0.0.1:34373/, target: http://127.0.0.1:36443/
[junit4] 2> 374628 WARN (TEST-LeaderFailoverAfterPartitionTest.test-seed#[BA55801CAB16FDDC]) [ ] o.a.s.c.SocketProxy Closing 6 connections to: http://127.0.0.1:43579/, target: http://127.0.0.1:37495/
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=LeaderFailoverAfterPartitionTest -Dtests.method=test -Dtests.seed=BA55801CAB16FDDC -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=lb-LU -Dtests.timezone=Asia/Yekaterinburg -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 137s J1 | LeaderFailoverAfterPartitionTest.test <<<
[junit4] > Throwable #1: java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node5:{"core":"c8n_1x3_lf_shard1_replica_n2","base_url":"http://127.0.0.1:45805","node_name":"127.0.0.1:45805_","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/16)={
[junit4] > "pullReplicas":"0",
[junit4] > "replicationFactor":"1",
[junit4] > "shards":{"shard1":{
[junit4] > "range":"80000000-7fffffff",
[junit4] > "state":"active",
[junit4] > "replicas":{
[junit4] > "core_node4":{
[junit4] > "core":"c8n_1x3_lf_shard1_replica_n1",
[junit4] > "base_url":"http://127.0.0.1:34373",
[junit4] > "node_name":"127.0.0.1:34373_",
[junit4] > "state":"down",
[junit4] > "type":"NRT"},
[junit4] > "core_node5":{
[junit4] > "core":"c8n_1x3_lf_shard1_replica_n2",
[junit4] > "base_url":"http://127.0.0.1:45805",
[junit4] > "node_name":"127.0.0.1:45805_",
[junit4] > "state":"active",
[junit4] > "type":"NRT",
[junit4] > "leader":"true"},
[junit4] > "core_node6":{
[junit4] > "state":"down",
[junit4] > "base_url":"http://127.0.0.1:37453",
[junit4] > "core":"c8n_1x3_lf_shard1_replica_n3",
[junit4] > "node_name":"127.0.0.1:37453_",
[junit4] > "type":"NRT"}}}},
[junit4] > "router":{"name":"compositeId"},
[junit4] > "maxShardsPerNode":"1",
[junit4] > "autoAddReplicas":"false",
[junit4] > "nrtReplicas":"3",
[junit4] > "tlogReplicas":"0"}
[junit4] > at __randomizedtesting.SeedInfo.seed([BA55801CAB16FDDC:3201BFC605EA9024]:0)
[junit4] > at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:169)
[junit4] > at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:56)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:564)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_BA55801CAB16FDDC-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {multiDefault=Lucene50(blocksize=128), a_t=Lucene50(blocksize=128), id=PostingsFormat(name=LuceneVarGapDocFreqInterval), text=PostingsFormat(name=Asserting)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Asserting), _version_=DocValuesFormat(name=Direct), range_facet_i_dv=DocValuesFormat(name=Memory), intDvoDefault=DocValuesFormat(name=Asserting), timestamp=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=373, maxMBSortInHeap=7.732662459660973, sim=RandomSimilarity(queryNorm=true): {}, locale=lb-LU, timezone=Asia/Yekaterinburg
[junit4] 2> NOTE: Linux 4.10.0-33-generic amd64/Oracle Corporation 9 (64-bit)/cpus=8,threads=1,free=283157784,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [ClusterStateUpdateTest, HdfsThreadLeakTest, InfoHandlerTest, QueryParsingTest, HdfsDirectoryFactoryTest, TestNoOpRegenerator, TestAuthorizationFramework, TestEmbeddedSolrServerConstructors, HdfsWriteToMultipleCollectionsTest, CollectionsAPISolrJTest, ZkCLITest, TestCloudPseudoReturnFields, TermVectorComponentTest, ScriptEngineTest, TestOrdValues, TestAnalyzeInfixSuggestions, SpellCheckComponentTest, CoreAdminHandlerTest, TestMaxScoreQueryParser, TestXIncludeConfig, DirectSolrSpellCheckerTest, LeaderFailoverAfterPartitionTest]
[junit4] Completed [106/732 (1!)] on J1 in 136.52s, 1 test, 1 failure <<< FAILURES!
[...truncated 41643 lines...]
[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk1.8.0_144) - Build # 495
- Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/495/
Java: 64bit/jdk1.8.0_144 -XX:+UseCompressedOops -XX:+UseSerialGC
2 tests failed.
FAILED: org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test
Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node5:{"core":"c8n_1x3_lf_shard1_replica_n2","base_url":"http://127.0.0.1:37181/cg_xl/de","node_name":"127.0.0.1:37181_cg_xl%2Fde","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/16)={ "pullReplicas":"0", "replicationFactor":"1", "shards":{"shard1":{ "range":"80000000-7fffffff", "state":"active", "replicas":{ "core_node4":{ "state":"down", "base_url":"http://127.0.0.1:44669/cg_xl/de", "core":"c8n_1x3_lf_shard1_replica_n1", "node_name":"127.0.0.1:44669_cg_xl%2Fde", "type":"NRT"}, "core_node5":{ "core":"c8n_1x3_lf_shard1_replica_n2", "base_url":"http://127.0.0.1:37181/cg_xl/de", "node_name":"127.0.0.1:37181_cg_xl%2Fde", "state":"active", "type":"NRT", "leader":"true"}, "core_node6":{ "core":"c8n_1x3_lf_shard1_replica_n3", "base_url":"http://127.0.0.1:43341/cg_xl/de", "node_name":"127.0.0.1:43341_cg_xl%2Fde", "state":"down", "type":"NRT"}}}}, "router":{"name":"compositeId"}, "maxShardsPerNode":"1", "autoAddReplicas":"false", "nrtReplicas":"3", "tlogReplicas":"0"}
Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node5:{"core":"c8n_1x3_lf_shard1_replica_n2","base_url":"http://127.0.0.1:37181/cg_xl/de","node_name":"127.0.0.1:37181_cg_xl%2Fde","state":"active","type":"NRT","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/16)={
"pullReplicas":"0",
"replicationFactor":"1",
"shards":{"shard1":{
"range":"80000000-7fffffff",
"state":"active",
"replicas":{
"core_node4":{
"state":"down",
"base_url":"http://127.0.0.1:44669/cg_xl/de",
"core":"c8n_1x3_lf_shard1_replica_n1",
"node_name":"127.0.0.1:44669_cg_xl%2Fde",
"type":"NRT"},
"core_node5":{
"core":"c8n_1x3_lf_shard1_replica_n2",
"base_url":"http://127.0.0.1:37181/cg_xl/de",
"node_name":"127.0.0.1:37181_cg_xl%2Fde",
"state":"active",
"type":"NRT",
"leader":"true"},
"core_node6":{
"core":"c8n_1x3_lf_shard1_replica_n3",
"base_url":"http://127.0.0.1:43341/cg_xl/de",
"node_name":"127.0.0.1:43341_cg_xl%2Fde",
"state":"down",
"type":"NRT"}}}},
"router":{"name":"compositeId"},
"maxShardsPerNode":"1",
"autoAddReplicas":"false",
"nrtReplicas":"3",
"tlogReplicas":"0"}
at __randomizedtesting.SeedInfo.seed([2701EFB75CFA281E:AF55D06DF20645E6]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:169)
at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:56)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.MissingSegmentRecoveryTest.testLeaderRecovery
Error Message:
expected:<10> but was:<0>
Stack Trace:
java.lang.AssertionError: expected:<10> but was:<0>
at __randomizedtesting.SeedInfo.seed([2701EFB75CFA281E:775477B405DB9E03]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.MissingSegmentRecoveryTest.testLeaderRecovery(MissingSegmentRecoveryTest.java:108)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12224 lines...]
[junit4] Suite: org.apache.solr.cloud.LeaderFailoverAfterPartitionTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/init-core-data-001
[junit4] 2> 669682 WARN (SUITE-LeaderFailoverAfterPartitionTest-seed#[2701EFB75CFA281E]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
[junit4] 2> 669682 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[2701EFB75CFA281E]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 669683 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[2701EFB75CFA281E]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
[junit4] 2> 669683 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[2701EFB75CFA281E]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 669683 INFO (SUITE-LeaderFailoverAfterPartitionTest-seed#[2701EFB75CFA281E]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /cg_xl/de
[junit4] 2> 669684 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 669685 INFO (Thread-1199) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 669685 INFO (Thread-1199) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 669687 ERROR (Thread-1199) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 669785 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.ZkTestServer start zk server on port:35575
[junit4] 2> 669810 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 669811 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 669811 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 669812 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 669813 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 669813 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 669814 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 669815 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 669816 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 669817 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 669817 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.AbstractZkTestCase put /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 669818 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
[junit4] 2> 669887 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 669905 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1382c5d8{/cg_xl/de,null,AVAILABLE}
[junit4] 2> 669906 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@12854bd3{HTTP/1.1,[http/1.1]}{127.0.0.1:45113}
[junit4] 2> 669906 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.Server Started @671179ms
[junit4] 2> 669906 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/tempDir-001/control/data, hostContext=/cg_xl/de, hostPort=42871, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/control-001/cores}
[junit4] 2> 669907 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 669907 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 669907 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 669907 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 669907 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T18:20:14.509Z
[junit4] 2> 669910 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 669910 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/control-001/solr.xml
[junit4] 2> 669913 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 669924 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35575/solr
[junit4] 2> 669959 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 669959 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42871_cg_xl%2Fde
[junit4] 2> 669960 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.Overseer Overseer (id=98726708136378372-127.0.0.1:42871_cg_xl%2Fde-n_0000000000) starting
[junit4] 2> 669965 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42871_cg_xl%2Fde
[junit4] 2> 669967 INFO (zkCallback-814-thread-1-processing-n:127.0.0.1:42871_cg_xl%2Fde) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 670031 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 670041 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 670041 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 670043 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/control-001/cores
[junit4] 2> 670069 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 670069 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35575/solr ready
[junit4] 2> 670070 INFO (SocketProxy-Acceptor-42871) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=37684,localport=42871], receiveBufferSize:531000
[junit4] 2> 670084 INFO (SocketProxy-Acceptor-42871) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=45113,localport=48706], receiveBufferSize=530904
[junit4] 2> 670088 INFO (qtp1788938286-5339) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42871_cg_xl%252Fde&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 670092 INFO (OverseerThreadFactory-2253-thread-1-processing-n:127.0.0.1:42871_cg_xl%2Fde) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.CreateCollectionCmd Create collection control_collection
[junit4] 2> 670204 INFO (SocketProxy-Acceptor-42871) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=37688,localport=42871], receiveBufferSize:531000
[junit4] 2> 670213 INFO (SocketProxy-Acceptor-42871) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=45113,localport=48710], receiveBufferSize=530904
[junit4] 2> 670213 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 670213 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 670319 INFO (zkCallback-814-thread-1-processing-n:127.0.0.1:42871_cg_xl%2Fde) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 671229 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 671246 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
[junit4] 2> 671316 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 671326 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
[junit4] 2> 671327 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 671327 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 671327 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/control-001/cores/control_collection_shard1_replica_n1/data/]
[junit4] 2> 671328 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=30, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6667788805352822]
[junit4] 2> 671333 WARN (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 671357 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 671357 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 671358 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 671358 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 671359 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=55.2275390625, floorSegmentMB=1.16015625, forceMergeDeletesPctAllowed=3.4894320416219573, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 671359 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6327a33f[control_collection_shard1_replica_n1] main]
[junit4] 2> 671359 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 671360 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 671360 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 671361 INFO (searcherExecutor-2256-thread-1-processing-n:127.0.0.1:42871_cg_xl%2Fde x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@6327a33f[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 671361 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579627331938418688
[junit4] 2> 671364 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 671364 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 671364 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:42871/cg_xl/de/control_collection_shard1_replica_n1/
[junit4] 2> 671365 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 671365 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:42871/cg_xl/de/control_collection_shard1_replica_n1/ has no replicas
[junit4] 2> 671365 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 671366 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:42871/cg_xl/de/control_collection_shard1_replica_n1/ shard1
[junit4] 2> 671467 INFO (zkCallback-814-thread-1-processing-n:127.0.0.1:42871_cg_xl%2Fde) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 671517 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 671519 INFO (qtp1788938286-5341) [n:127.0.0.1:42871_cg_xl%2Fde c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1306
[junit4] 2> 671521 INFO (qtp1788938286-5339) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 671622 INFO (zkCallback-814-thread-1-processing-n:127.0.0.1:42871_cg_xl%2Fde) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 672094 INFO (OverseerCollectionConfigSetProcessor-98726708136378372-127.0.0.1:42871_cg_xl%2Fde-n_0000000000) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 672522 INFO (qtp1788938286-5339) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:42871_cg_xl%252Fde&wt=javabin&version=2} status=0 QTime=2433
[junit4] 2> 672526 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 672526 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35575/solr ready
[junit4] 2> 672526 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 672527 INFO (SocketProxy-Acceptor-42871) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=37694,localport=42871], receiveBufferSize:531000
[junit4] 2> 672527 INFO (SocketProxy-Acceptor-42871) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=45113,localport=48716], receiveBufferSize=530904
[junit4] 2> 672527 INFO (qtp1788938286-5339) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 672528 INFO (OverseerThreadFactory-2253-thread-2-processing-n:127.0.0.1:42871_cg_xl%2Fde) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 672529 WARN (OverseerThreadFactory-2253-thread-2-processing-n:127.0.0.1:42871_cg_xl%2Fde) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
[junit4] 2> 672731 INFO (qtp1788938286-5339) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 672731 INFO (qtp1788938286-5339) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=203
[junit4] 2> 672788 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-1-001 of type NRT
[junit4] 2> 672789 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 672790 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@28e6a550{/cg_xl/de,null,AVAILABLE}
[junit4] 2> 672790 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@33bdc6d9{HTTP/1.1,[http/1.1]}{127.0.0.1:34787}
[junit4] 2> 672790 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.Server Started @674062ms
[junit4] 2> 672790 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/cg_xl/de, hostPort=43341, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-1-001/cores}
[junit4] 2> 672790 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 672790 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 672790 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 672790 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 672791 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T18:20:17.392Z
[junit4] 2> 672797 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 672797 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-1-001/solr.xml
[junit4] 2> 672799 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 672805 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35575/solr
[junit4] 2> 672815 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 672816 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 672817 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43341_cg_xl%2Fde
[junit4] 2> 672817 INFO (zkCallback-814-thread-1-processing-n:127.0.0.1:42871_cg_xl%2Fde) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 672818 INFO (zkCallback-821-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 672818 INFO (zkCallback-826-thread-1-processing-n:127.0.0.1:43341_cg_xl%2Fde) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 672884 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 672891 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 672891 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 672892 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-1-001/cores
[junit4] 2> 672912 INFO (qtp1788938286-5340) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:43341_cg_xl%252Fde&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 672913 INFO (OverseerCollectionConfigSetProcessor-98726708136378372-127.0.0.1:42871_cg_xl%2Fde-n_0000000000) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 672913 INFO (OverseerThreadFactory-2253-thread-3-processing-n:127.0.0.1:42871_cg_xl%2Fde) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:43341_cg_xl%2Fde for creating new replica
[junit4] 2> 672914 INFO (SocketProxy-Acceptor-43341) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=52998,localport=43341], receiveBufferSize:531000
[junit4] 2> 672916 INFO (SocketProxy-Acceptor-43341) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=34787,localport=58934], receiveBufferSize=530904
[junit4] 2> 672917 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 672917 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 673021 INFO (zkCallback-826-thread-1-processing-n:127.0.0.1:43341_cg_xl%2Fde) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 673927 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 673950 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema [collection1_shard2_replica_n41] Schema name=test
[junit4] 2> 674039 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 674050 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n41' using configuration from collection collection1, trusted=true
[junit4] 2> 674050 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n41' (registry 'solr.core.collection1.shard2.replica_n41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 674050 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 674050 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [[collection1_shard2_replica_n41] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-1-001/cores/collection1_shard2_replica_n41], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-1-001/cores/collection1_shard2_replica_n41/data/]
[junit4] 2> 674053 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=30, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6667788805352822]
[junit4] 2> 674054 WARN (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 674088 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 674088 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 674089 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 674089 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 674090 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=55.2275390625, floorSegmentMB=1.16015625, forceMergeDeletesPctAllowed=3.4894320416219573, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 674090 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.s.SolrIndexSearcher Opening [Searcher@11668ecc[collection1_shard2_replica_n41] main]
[junit4] 2> 674091 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 674091 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 674091 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 674092 INFO (searcherExecutor-2267-thread-1-processing-n:127.0.0.1:43341_cg_xl%2Fde x:collection1_shard2_replica_n41 s:shard2 c:collection1) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [collection1_shard2_replica_n41] Registered new searcher Searcher@11668ecc[collection1_shard2_replica_n41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 674092 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579627334802079744
[junit4] 2> 674095 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 674095 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 674095 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:43341/cg_xl/de/collection1_shard2_replica_n41/
[junit4] 2> 674095 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 674095 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy http://127.0.0.1:43341/cg_xl/de/collection1_shard2_replica_n41/ has no replicas
[junit4] 2> 674095 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 674097 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:43341/cg_xl/de/collection1_shard2_replica_n41/ shard2
[junit4] 2> 674198 INFO (zkCallback-826-thread-1-processing-n:127.0.0.1:43341_cg_xl%2Fde) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 674198 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 674201 INFO (qtp650724821-5387) [n:127.0.0.1:43341_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1284
[junit4] 2> 674203 INFO (qtp1788938286-5340) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:43341_cg_xl%252Fde&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1291
[junit4] 2> 674204 INFO (OverseerCollectionConfigSetProcessor-98726708136378372-127.0.0.1:42871_cg_xl%2Fde-n_0000000000) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 674283 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-2-001 of type NRT
[junit4] 2> 674284 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 674290 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@51eb2c08{/cg_xl/de,null,AVAILABLE}
[junit4] 2> 674290 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@7a723169{HTTP/1.1,[http/1.1]}{127.0.0.1:43715}
[junit4] 2> 674290 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.Server Started @675563ms
[junit4] 2> 674290 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/tempDir-001/jetty2, solrconfig=solrconfig.xml, hostContext=/cg_xl/de, hostPort=37181, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-2-001/cores}
[junit4] 2> 674291 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 674292 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 674292 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 674292 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 674292 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T18:20:18.894Z
[junit4] 2> 674301 INFO (zkCallback-826-thread-1-processing-n:127.0.0.1:43341_cg_xl%2Fde) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 674303 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 674303 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-2-001/solr.xml
[junit4] 2> 674307 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 674311 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35575/solr
[junit4] 2> 674353 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:37181_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 674354 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:37181_cg_xl%2Fde ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 674356 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:37181_cg_xl%2Fde ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37181_cg_xl%2Fde
[junit4] 2> 674358 INFO (zkCallback-821-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 674358 INFO (zkCallback-826-thread-1-processing-n:127.0.0.1:43341_cg_xl%2Fde) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 674358 INFO (zkCallback-814-thread-1-processing-n:127.0.0.1:42871_cg_xl%2Fde) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 674361 INFO (zkCallback-832-thread-1-processing-n:127.0.0.1:37181_cg_xl%2Fde) [n:127.0.0.1:37181_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 674412 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:37181_cg_xl%2Fde ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 674423 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:37181_cg_xl%2Fde ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 674423 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:37181_cg_xl%2Fde ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 674424 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:37181_cg_xl%2Fde ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-2-001/cores
[junit4] 2> 674462 INFO (qtp1788938286-5335) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:37181_cg_xl%252Fde&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 674464 INFO (OverseerThreadFactory-2253-thread-4-processing-n:127.0.0.1:42871_cg_xl%2Fde) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:37181_cg_xl%2Fde for creating new replica
[junit4] 2> 674466 INFO (SocketProxy-Acceptor-37181) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=48790,localport=37181], receiveBufferSize:531000
[junit4] 2> 674474 INFO (SocketProxy-Acceptor-37181) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43715,localport=46462], receiveBufferSize=530904
[junit4] 2> 674475 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 674475 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 674584 INFO (zkCallback-832-thread-1-processing-n:127.0.0.1:37181_cg_xl%2Fde) [n:127.0.0.1:37181_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 674584 INFO (zkCallback-826-thread-1-processing-n:127.0.0.1:43341_cg_xl%2Fde) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 675486 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 675495 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema [collection1_shard1_replica_n43] Schema name=test
[junit4] 2> 675585 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 675596 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n43' using configuration from collection collection1, trusted=true
[junit4] 2> 675597 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n43' (registry 'solr.core.collection1.shard1.replica_n43') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 675597 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 675597 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [[collection1_shard1_replica_n43] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-2-001/cores/collection1_shard1_replica_n43], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-2-001/cores/collection1_shard1_replica_n43/data/]
[junit4] 2> 675598 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=30, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6667788805352822]
[junit4] 2> 675601 WARN (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 675626 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 675626 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 675627 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 675627 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 675628 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=55.2275390625, floorSegmentMB=1.16015625, forceMergeDeletesPctAllowed=3.4894320416219573, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 675628 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.s.SolrIndexSearcher Opening [Searcher@4c55882f[collection1_shard1_replica_n43] main]
[junit4] 2> 675629 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 675630 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 675630 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 675631 INFO (searcherExecutor-2278-thread-1-processing-n:127.0.0.1:37181_cg_xl%2Fde x:collection1_shard1_replica_n43 s:shard1 c:collection1) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [collection1_shard1_replica_n43] Registered new searcher Searcher@4c55882f[collection1_shard1_replica_n43] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 675631 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579627336415838208
[junit4] 2> 675634 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 675634 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 675634 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:37181/cg_xl/de/collection1_shard1_replica_n43/
[junit4] 2> 675634 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 675634 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy http://127.0.0.1:37181/cg_xl/de/collection1_shard1_replica_n43/ has no replicas
[junit4] 2> 675634 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 675635 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:37181/cg_xl/de/collection1_shard1_replica_n43/ shard1
[junit4] 2> 675736 INFO (zkCallback-826-thread-1-processing-n:127.0.0.1:43341_cg_xl%2Fde) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 675736 INFO (zkCallback-832-thread-1-processing-n:127.0.0.1:37181_cg_xl%2Fde) [n:127.0.0.1:37181_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 675785 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 675786 INFO (qtp1202144587-5419) [n:127.0.0.1:37181_cg_xl%2Fde c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1311
[junit4] 2> 675787 INFO (qtp1788938286-5335) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:37181_cg_xl%252Fde&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1324
[junit4] 2> 675843 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-3-001 of type NRT
[junit4] 2> 675844 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 675844 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4b246f04{/cg_xl/de,null,AVAILABLE}
[junit4] 2> 675845 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@623d2bed{HTTP/1.1,[http/1.1]}{127.0.0.1:43059}
[junit4] 2> 675845 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.e.j.s.Server Started @677117ms
[junit4] 2> 675845 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/tempDir-001/jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/cg_xl/de, hostPort=44669, coreRootDirectory=/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-3-001/cores}
[junit4] 2> 675845 ERROR (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 675845 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 675845 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 675845 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 675845 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T18:20:20.447Z
[junit4] 2> 675850 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 675850 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-3-001/solr.xml
[junit4] 2> 675852 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 675854 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35575/solr
[junit4] 2> 675859 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:44669_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 675860 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:44669_cg_xl%2Fde ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 675860 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:44669_cg_xl%2Fde ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44669_cg_xl%2Fde
[junit4] 2> 675861 INFO (zkCallback-821-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 675861 INFO (zkCallback-814-thread-1-processing-n:127.0.0.1:42871_cg_xl%2Fde) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 675861 INFO (zkCallback-832-thread-1-processing-n:127.0.0.1:37181_cg_xl%2Fde) [n:127.0.0.1:37181_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 675861 INFO (zkCallback-826-thread-1-processing-n:127.0.0.1:43341_cg_xl%2Fde) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 675861 INFO (zkCallback-838-thread-1-processing-n:127.0.0.1:44669_cg_xl%2Fde) [n:127.0.0.1:44669_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 675922 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:44669_cg_xl%2Fde ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 675929 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:44669_cg_xl%2Fde ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 675930 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:44669_cg_xl%2Fde ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 675931 INFO (TEST-LeaderFailoverAfterPartitionTest.test-seed#[2701EFB75CFA281E]) [n:127.0.0.1:44669_cg_xl%2Fde ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-3-001/cores
[junit4] 2> 675960 INFO (qtp1788938286-5336) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:44669_cg_xl%252Fde&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 675961 INFO (OverseerCollectionConfigSetProcessor-98726708136378372-127.0.0.1:42871_cg_xl%2Fde-n_0000000000) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 675961 INFO (OverseerThreadFactory-2253-thread-5-processing-n:127.0.0.1:42871_cg_xl%2Fde) [n:127.0.0.1:42871_cg_xl%2Fde ] o.a.s.c.AddReplicaCmd Node Identified 127.0.0.1:44669_cg_xl%2Fde for creating new replica
[junit4] 2> 675962 INFO (zkCallback-826-thread-1-processing-n:127.0.0.1:43341_cg_xl%2Fde) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 675962 INFO (zkCallback-832-thread-1-processing-n:127.0.0.1:37181_cg_xl%2Fde) [n:127.0.0.1:37181_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 675962 INFO (SocketProxy-Acceptor-44669) [ ] o.a.s.c.SocketProxy accepted Socket[addr=/127.0.0.1,port=51242,localport=44669], receiveBufferSize:531000
[junit4] 2> 675963 INFO (SocketProxy-Acceptor-44669) [ ] o.a.s.c.SocketProxy proxy connection Socket[addr=/127.0.0.1,port=43059,localport=33770], receiveBufferSize=530904
[junit4] 2> 675964 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n45&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 675964 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 676066 INFO (zkCallback-832-thread-1-processing-n:127.0.0.1:37181_cg_xl%2Fde) [n:127.0.0.1:37181_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 676066 INFO (zkCallback-838-thread-1-processing-n:127.0.0.1:44669_cg_xl%2Fde) [n:127.0.0.1:44669_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 676066 INFO (zkCallback-826-thread-1-processing-n:127.0.0.1:43341_cg_xl%2Fde) [n:127.0.0.1:43341_cg_xl%2Fde ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 676972 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 676979 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.s.IndexSchema [collection1_shard2_replica_n45] Schema name=test
[junit4] 2> 677055 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 677074 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n45' using configuration from collection collection1, trusted=true
[junit4] 2> 677075 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n45' (registry 'solr.core.collection1.shard2.replica_n45') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1c77916
[junit4] 2> 677075 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 677075 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.c.SolrCore [[collection1_shard2_replica_n45] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-3-001/cores/collection1_shard2_replica_n45], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.cloud.LeaderFailoverAfterPartitionTest_2701EFB75CFA281E-001/shard-3-001/cores/collection1_shard2_replica_n45/data/]
[junit4] 2> 677076 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=30, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6667788805352822]
[junit4] 2> 677078 WARN (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 677102 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 677102 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 677103 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 677103 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 677104 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=47, maxMergedSegmentMB=55.2275390625, floorSegmentMB=1.16015625, forceMergeDeletesPctAllowed=3.4894320416219573, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
[junit4] 2> 677104 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.s.SolrIndexSearcher Opening [Searcher@76c20ce1[collection1_shard2_replica_n45] main]
[junit4] 2> 677104 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 677105 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 677105 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 677106 INFO (searcherExecutor-2289-thread-1-processing-n:127.0.0.1:44669_cg_xl%2Fde x:collection1_shard2_replica_n45 s:shard2 c:collection1) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.c.SolrCore [collection1_shard2_replica_n45] Registered new searcher Searcher@76c20ce1[collection1_shard2_replica_n45] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 677106 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579627337962487808
[junit4] 2> 677108 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.c.ZkController Core needs to recover:collection1_shard2_replica_n45
[junit4] 2> 677109 INFO (updateExecutor-835-thread-1-processing-n:127.0.0.1:44669_cg_xl%2Fde x:collection1_shard2_replica_n45 s:shard2 c:collection1) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 677109 INFO (qtp87005549-5451) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n45&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1145
[junit4] 2> 677109 INFO (recoveryExecutor-836-thread-1-processing-n:127.0.0.1:44669_cg_xl%2Fde x:collection1_shard2_replica_n45 s:shard2 c:collection1 r:core_node46) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 677109 INFO (recoveryExecutor-836-thread-1-processing-n:127.0.0.1:44669_cg_xl%2Fde x:collection1_shard2_replica_n45 s:shard2 c:collection1 r:core_node46) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 677109 INFO (recoveryExecutor-836-thread-1-processing-n:127.0.0.1:44669_cg_xl%2Fde x:collection1_shard2_replica_n45 s:shard2 c:collection1 r:core_node46) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1_shard2_replica_n45]
[junit4] 2> 677109 INFO (recoveryExecutor-836-thread-1-processing-n:127.0.0.1:44669_cg_xl%2Fde x:collection1_shard2_replica_n45 s:shard2 c:collection1 r:core_node46) [n:127.0.0.1:44669_cg_xl%2Fde c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 677109 I
[...truncated too long message...]
l=[false]
[junit4] 2> 1585920 INFO (recoveryExecutor-2020-thread-1-processing-n:127.0.0.1:36187_solr x:MissingSegmentRecoveryTest_shard1_replica_n1 s:shard1 c:MissingSegmentRecoveryTest r:core_node3) [n:127.0.0.1:36187_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node3 x:MissingSegmentRecoveryTest_shard1_replica_n1] o.a.s.c.SolrCore [MissingSegmentRecoveryTest_shard1_replica_n1] CLOSING SolrCore org.apache.solr.core.SolrCore@24e7f923
[junit4] 2> 1585920 INFO (recoveryExecutor-2020-thread-1-processing-n:127.0.0.1:36187_solr x:MissingSegmentRecoveryTest_shard1_replica_n1 s:shard1 c:MissingSegmentRecoveryTest r:core_node3) [n:127.0.0.1:36187_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node3 x:MissingSegmentRecoveryTest_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.MissingSegmentRecoveryTest.shard1.replica_n1, tag=619182371
[junit4] 2> 1585920 INFO (recoveryExecutor-2020-thread-1-processing-n:127.0.0.1:36187_solr x:MissingSegmentRecoveryTest_shard1_replica_n1 s:shard1 c:MissingSegmentRecoveryTest r:core_node3) [n:127.0.0.1:36187_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node3 x:MissingSegmentRecoveryTest_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2173983e: rootName = solr_36187, domain = solr.core.MissingSegmentRecoveryTest.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.MissingSegmentRecoveryTest.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@178ab23c
[junit4] 2> 1585926 INFO (recoveryExecutor-2020-thread-1-processing-n:127.0.0.1:36187_solr x:MissingSegmentRecoveryTest_shard1_replica_n1 s:shard1 c:MissingSegmentRecoveryTest r:core_node3) [n:127.0.0.1:36187_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node3 x:MissingSegmentRecoveryTest_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.MissingSegmentRecoveryTest.shard1.leader, tag=619182371
[junit4] 2> 1585927 WARN (recoveryExecutor-2020-thread-1-processing-n:127.0.0.1:36187_solr x:MissingSegmentRecoveryTest_shard1_replica_n1 s:shard1 c:MissingSegmentRecoveryTest r:core_node3) [n:127.0.0.1:36187_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node3 x:MissingSegmentRecoveryTest_shard1_replica_n1] o.a.s.c.RecoveryStrategy Stopping recovery for core=[MissingSegmentRecoveryTest_shard1_replica_n1] coreNodeName=[core_node3]
[junit4] 2> 1585928 WARN (updateExecutor-2019-thread-2-processing-n:127.0.0.1:36187_solr x:MissingSegmentRecoveryTest_shard1_replica_n1 s:shard1 c:MissingSegmentRecoveryTest r:core_node3) [n:127.0.0.1:36187_solr c:MissingSegmentRecoveryTest s:shard1 r:core_node3 x:MissingSegmentRecoveryTest_shard1_replica_n1] o.a.s.u.DefaultSolrCoreState Skipping recovery because Solr is shutdown
[junit4] 2> 1585929 INFO (jetty-closer-2000-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4bc4afe8{/solr,null,UNAVAILABLE}
[junit4] 2> 1585929 ERROR (SUITE-MissingSegmentRecoveryTest-seed#[2701EFB75CFA281E]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1585929 INFO (SUITE-MissingSegmentRecoveryTest-seed#[2701EFB75CFA281E]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42925 42925
[junit4] 2> 1585997 INFO (Thread-6876) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42925 42925
[junit4] 2> 1585997 WARN (Thread-6876) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/aliases.json
[junit4] 2> 3 /solr/security.json
[junit4] 2> 3 /solr/configs/conf
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 18 /solr/collections/MissingSegmentRecoveryTest/state.json
[junit4] 2> 4 /solr/clusterstate.json
[junit4] 2> 4 /solr/clusterprops.json
[junit4] 2> 2 /solr/overseer_elect/election/98726767540961286-127.0.0.1:35487_solr-n_0000000000
[junit4] 2> 2 /solr/collections/MissingSegmentRecoveryTest/leader_elect/shard1/election/98726767540961286-core_node4-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 4 /solr/collections
[junit4] 2> 3 /solr/live_nodes
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.MissingSegmentRecoveryTest_2701EFB75CFA281E-001
[junit4] 2> Sep 26, 2017 6:35:30 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=false): {}, locale=mt-MT, timezone=America/Merida
[junit4] 2> NOTE: Linux 4.10.0-33-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=8,threads=1,free=153305152,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [SolrIndexMetricsTest, SharedFSAutoReplicaFailoverUtilsTest, TestPhraseSuggestions, SolrMetricReporterTest, TestGraphTermsQParserPlugin, TestMultiValuedNumericRangeQuery, TestSimpleQParserPlugin, TestPullReplica, TestIndexingPerformance, LeaderElectionContextKeyTest, UpdateLogTest, TestFunctionQuery, SparseHLLTest, TestCustomDocTransformer, TestMultiWordSynonyms, SuggesterWFSTTest, CursorPagingTest, FieldMutatingUpdateProcessorTest, TestCloudSearcherWarming, JSONWriterTest, HdfsBasicDistributedZk2Test, TestCustomStream, TestRecoveryHdfs, TestJsonRequest, DefaultValueUpdateProcessorTest, LeaderInitiatedRecoveryOnCommitTest, TestTrackingShardHandlerFactory, TestCloudPivotFacet, SuggestComponentContextFilterQueryTest, TestRandomRequestDistribution, TestConfigsApi, TestRandomCollapseQParserPlugin, MoveReplicaTest, TestRemoteStreaming, V2StandaloneTest, TestEmbeddedSolrServerAdminHandler, TestCustomSort, TestUninvertingReader, TestMinMaxOnMultiValuedField, TestSolrQueryResponse, TestSchemaManager, DocumentAnalysisRequestHandlerTest, TestCollationFieldDocValues, TestUniqueKeyFieldResource, TestLazyCores, CdcrUpdateLogTest, ChangedSchemaMergeTest, RequestHandlersTest, TestPHPSerializedResponseWriter, IndexBasedSpellCheckerTest, TestRandomFaceting, TestTlogReplica, AsyncCallRequestStatusResponseTest, RemoteQueryErrorTest, TestRequestForwarding, TestExtendedDismaxParser, ImplicitSnitchTest, DistributedExpandComponentTest, StatsReloadRaceTest, TestWordDelimiterFilterFactory, ChaosMonkeyNothingIsSafeTest, OverseerTaskQueueTest, ExitableDirectoryReaderTest, TestGraphMLResponseWriter, TestManagedSchemaAPI, TestQuerySenderNoQuery, TestSolrConfigHandler, RestartWhileUpdatingTest, SolrCloudExampleTest, TestRandomDVFaceting, TestConfigSetsAPIZkFailure, TestBulkSchemaConcurrent, SolrGraphiteReporterTest, MetricsConfigTest, LeaderElectionIntegrationTest, ReplicaListTransformerTest, SpellingQueryConverterTest, TestRawResponseWriter, TestConfig, TestLegacyNumericRangeQueryBuilder, SpellCheckCollatorWithCollapseTest, ExternalFileFieldSortTest, SynonymTokenizerTest, TestSegmentSorting, TestNumericTokenStream, TestHdfsUpdateLog, TestSchemaSimilarityResource, CdcrReplicationDistributedZkTest, TestLegacyFieldReuse, FieldAnalysisRequestHandlerTest, DistribJoinFromCollectionTest, TestFieldTypeResource, BasicDistributedZkTest, TestFastOutputStream, AliasIntegrationTest, TestInfoStreamLogging, CursorMarkTest, TestUseDocValuesAsStored2, TestBadConfig, WrapperMergePolicyFactoryTest, TestLRUStatsCache, FullHLLTest, TestBinaryField, SaslZkACLProviderTest, ConfigureRecoveryStrategyTest, TestImpersonationWithHadoopAuth, TestRuleBasedAuthorizationPlugin, PreAnalyzedUpdateProcessorTest, CoreMergeIndexesAdminHandlerTest, TestLMDirichletSimilarityFactory, DistributedSpellCheckComponentTest, TestSubQueryTransformerCrossCore, TestIBSimilarityFactory, TestXmlQParserPlugin, QueryElevationComponentTest, TestClusterStateMutator, TestReplicationHandlerBackup, DistanceUnitsTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, ShardRoutingTest, BasicZkTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ZkCLITest, TestDistributedGrouping, ZkControllerTest, SolrCoreTest, PeerSyncTest, ShowFileRequestHandlerTest, SolrIndexSplitterTest, SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, SolrRequestParserTest, TestFoldingMultitermQuery, DocValuesTest, SchemaVersionSpecificBehaviorTest, DirectUpdateHandlerOptimizeTest, StatelessScriptUpdateProcessorFactoryTest, DebugComponentTest, DisMaxRequestHandlerTest, TestQueryTypes, TestOmitPositions, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, MoreLikeThisHandlerTest, FastVectorHighlighterTest, TestCollationField, TestCSVResponseWriter, QueryParsingTest, UniqFieldsUpdateProcessorFactoryTest, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, CSVRequestHandlerTest, TestComponentsName, TestLFUCache, OutputWriterTest, DirectSolrConnectionTest, TestLuceneMatchVersion, TestXIncludeConfig, EchoParamsTest, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, URLClassifyProcessorTest, TestFastLRUCache, PreAnalyzedFieldTest, PrimUtilsTest, DOMUtilTest, ClusterStateTest, TestLRUCache, ZkNodePropsTest, CircularListTest, TestRTGBase, SolrTestCaseJ4Test, TestCrossCoreJoin, TestCursorMarkWithoutUniqueKey, TestHighlightDedupGrouping, ActionThrottleTest, AssignBackwardCompatibilityTest, ConfigSetsAPITest, ConnectionManagerTest, CreateCollectionCleanupTest, DeleteInactiveReplicaTest, DeleteLastCustomShardedReplicaTest, DeleteNodeTest, DeleteStatusTest, DistributedVersionInfoTest, MissingSegmentRecoveryTest]
[junit4] Completed [517/732 (2!)] on J0 in 10.29s, 1 test, 1 failure <<< FAILURES!
[...truncated 47350 lines...]
[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk1.8.0_144) - Build # 494
- Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/494/
Java: 64bit/jdk1.8.0_144 -XX:-UseCompressedOops -XX:+UseG1GC
1 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth
Error Message:
2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth: 1) Thread[id=19322, name=jetty-launcher-2855-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105) at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288) at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279) at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41) at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244) at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44) at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61) at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505) 2) Thread[id=19317, name=jetty-launcher-2855-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277) at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323) at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105) at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288) at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279) at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41) at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244) at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44) at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61) at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth:
1) Thread[id=19322, name=jetty-launcher-2855-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
2) Thread[id=19317, name=jetty-launcher-2855-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
at __randomizedtesting.SeedInfo.seed([A2FF6DE13CDE0217]:0)
Build Log:
[...truncated 13802 lines...]
[junit4] Suite: org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_A2FF6DE13CDE0217-001/init-core-data-001
[junit4] 2> 2120923 INFO (SUITE-TestImpersonationWithHadoopAuth-seed#[A2FF6DE13CDE0217]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 2120925 INFO (SUITE-TestImpersonationWithHadoopAuth-seed#[A2FF6DE13CDE0217]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 2120925 INFO (SUITE-TestImpersonationWithHadoopAuth-seed#[A2FF6DE13CDE0217]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 2120947 INFO (SUITE-TestImpersonationWithHadoopAuth-seed#[A2FF6DE13CDE0217]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_A2FF6DE13CDE0217-001/tempDir-001
[junit4] 2> 2120947 INFO (SUITE-TestImpersonationWithHadoopAuth-seed#[A2FF6DE13CDE0217]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 2120948 INFO (Thread-5399) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2120948 INFO (Thread-5399) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 2120950 ERROR (Thread-5399) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2121048 INFO (SUITE-TestImpersonationWithHadoopAuth-seed#[A2FF6DE13CDE0217]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:45373
[junit4] 2> 2121058 INFO (jetty-launcher-2855-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 2121058 INFO (jetty-launcher-2855-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 2121059 INFO (jetty-launcher-2855-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3b367515{/solr,null,AVAILABLE}
[junit4] 2> 2121059 INFO (jetty-launcher-2855-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@22e6e184{/solr,null,AVAILABLE}
[junit4] 2> 2121061 INFO (jetty-launcher-2855-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@73e1442e{SSL,[ssl, http/1.1]}{127.0.0.1:44079}
[junit4] 2> 2121062 INFO (jetty-launcher-2855-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@5e378797{SSL,[ssl, http/1.1]}{127.0.0.1:40037}
[junit4] 2> 2121062 INFO (jetty-launcher-2855-thread-2) [ ] o.e.j.s.Server Started @2122621ms
[junit4] 2> 2121062 INFO (jetty-launcher-2855-thread-1) [ ] o.e.j.s.Server Started @2122621ms
[junit4] 2> 2121062 INFO (jetty-launcher-2855-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44079}
[junit4] 2> 2121062 INFO (jetty-launcher-2855-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40037}
[junit4] 2> 2121062 ERROR (jetty-launcher-2855-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2121062 ERROR (jetty-launcher-2855-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2121062 INFO (jetty-launcher-2855-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 2121062 INFO (jetty-launcher-2855-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 2121062 INFO (jetty-launcher-2855-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2121063 INFO (jetty-launcher-2855-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2121063 INFO (jetty-launcher-2855-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 2121063 INFO (jetty-launcher-2855-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 2121063 INFO (jetty-launcher-2855-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T15:43:44.538Z
[junit4] 2> 2121063 INFO (jetty-launcher-2855-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T15:43:44.538Z
[junit4] 2> 2121065 INFO (jetty-launcher-2855-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 2121065 INFO (jetty-launcher-2855-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 2121075 INFO (jetty-launcher-2855-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45373/solr
[junit4] 2> 2121075 INFO (jetty-launcher-2855-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45373/solr
[junit4] 2> 2121079 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ebedbb9280003, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 2121114 INFO (jetty-launcher-2855-thread-2) [n:127.0.0.1:44079_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2121114 INFO (jetty-launcher-2855-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2121115 INFO (jetty-launcher-2855-thread-2) [n:127.0.0.1:44079_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:44079_solr
[junit4] 2> 2121115 INFO (jetty-launcher-2855-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40037_solr
[junit4] 2> 2121115 INFO (jetty-launcher-2855-thread-2) [n:127.0.0.1:44079_solr ] o.a.s.c.Overseer Overseer (id=98726092762251270-127.0.0.1:44079_solr-n_0000000000) starting
[junit4] 2> 2121115 INFO (zkCallback-2867-thread-1-processing-n:127.0.0.1:44079_solr) [n:127.0.0.1:44079_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2121115 INFO (zkCallback-2866-thread-1-processing-n:127.0.0.1:40037_solr) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2121117 INFO (jetty-launcher-2855-thread-2) [n:127.0.0.1:44079_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44079_solr
[junit4] 2> 2121117 INFO (zkCallback-2867-thread-1-processing-n:127.0.0.1:44079_solr) [n:127.0.0.1:44079_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2121117 INFO (zkCallback-2866-thread-1-processing-n:127.0.0.1:40037_solr) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2121156 INFO (jetty-launcher-2855-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
[junit4] 2> 2121156 INFO (jetty-launcher-2855-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.s.HadoopAuthPlugin Params: {proxyuser.anyHostUsersGroup.hosts=*, proxyuser.wrongHost.hosts=1.1.1.1.1.1, proxyuser.bogusGroup.groups=__some_bogus_group, proxyuser.localHostAnyGroup.groups=*, cookie.domain=127.0.0.1, proxyuser.wrongHost.groups=*, type=simple, proxyuser.noHosts.groups=*, proxyuser.anyHostUsersGroup.groups=jenkins, proxyuser.noGroups.hosts=*, proxyuser.anyHostAnyUser.groups=*, delegation-token.token-kind=solr-dt, proxyuser.bogusGroup.hosts=*, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, zk-dt-secret-manager.enable=true, proxyuser.localHostAnyGroup.hosts=localhost.localdomain,localhost,127.0.0.1, proxyuser.anyHostAnyUser.hosts=*, signer.secret.provider=zookeeper}
[junit4] 2> 2121157 INFO (jetty-launcher-2855-thread-1) [n:127.0.0.1:40037_solr ] o.a.c.f.i.CuratorFrameworkImpl Starting
[junit4] 2> 2121158 INFO (jetty-launcher-2855-thread-1-EventThread) [n:127.0.0.1:40037_solr ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
[junit4] 2> 2121159 INFO (jetty-launcher-2855-thread-2) [n:127.0.0.1:44079_solr ] o.a.s.c.CoreContainer Initializing authentication plugin: org.apache.solr.security.HadoopAuthPlugin
[junit4] 2> 2121159 INFO (jetty-launcher-2855-thread-2) [n:127.0.0.1:44079_solr ] o.a.s.s.HadoopAuthPlugin Params: {proxyuser.anyHostUsersGroup.hosts=*, proxyuser.wrongHost.hosts=1.1.1.1.1.1, proxyuser.bogusGroup.groups=__some_bogus_group, proxyuser.localHostAnyGroup.groups=*, cookie.domain=127.0.0.1, proxyuser.wrongHost.groups=*, type=simple, proxyuser.noHosts.groups=*, proxyuser.anyHostUsersGroup.groups=jenkins, proxyuser.noGroups.hosts=*, proxyuser.anyHostAnyUser.groups=*, delegation-token.token-kind=solr-dt, proxyuser.bogusGroup.hosts=*, zk-dt-secret-manager.znodeWorkingPath=solr/security/zkdtsm, signer.secret.provider.zookeeper.path=/token, zk-dt-secret-manager.enable=true, proxyuser.localHostAnyGroup.hosts=localhost.localdomain,localhost,127.0.0.1, proxyuser.anyHostAnyUser.hosts=*, signer.secret.provider=zookeeper}
[junit4] 2> 2121159 INFO (jetty-launcher-2855-thread-2) [n:127.0.0.1:44079_solr ] o.a.c.f.i.CuratorFrameworkImpl Starting
[junit4] 2> 2121160 INFO (jetty-launcher-2855-thread-2-EventThread) [n:127.0.0.1:44079_solr ] o.a.c.f.s.ConnectionStateManager State change: CONNECTED
[junit4] 2> 2121166 INFO (jetty-launcher-2855-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
[junit4] 2> 2121166 INFO (jetty-launcher-2855-thread-2) [n:127.0.0.1:44079_solr ] o.a.s.c.CoreContainer PKIAuthenticationPlugin is managing internode requests
[junit4] 2> 2121191 INFO (jetty-launcher-2855-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40037.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4ad2ef8f
[junit4] 2> 2121193 INFO (jetty-launcher-2855-thread-2) [n:127.0.0.1:44079_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44079.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4ad2ef8f
[junit4] 2> 2121198 INFO (jetty-launcher-2855-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40037.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4ad2ef8f
[junit4] 2> 2121198 INFO (jetty-launcher-2855-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40037.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4ad2ef8f
[junit4] 2> 2121198 INFO (jetty-launcher-2855-thread-2) [n:127.0.0.1:44079_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44079.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4ad2ef8f
[junit4] 2> 2121199 INFO (jetty-launcher-2855-thread-2) [n:127.0.0.1:44079_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_44079.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4ad2ef8f
[junit4] 2> 2121199 INFO (jetty-launcher-2855-thread-1) [n:127.0.0.1:40037_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_A2FF6DE13CDE0217-001/tempDir-001/node1/.
[junit4] 2> 2121199 INFO (jetty-launcher-2855-thread-2) [n:127.0.0.1:44079_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J1/temp/solr.security.hadoop.TestImpersonationWithHadoopAuth_A2FF6DE13CDE0217-001/tempDir-001/node2/.
[junit4] 2> 2121224 INFO (SUITE-TestImpersonationWithHadoopAuth-seed#[A2FF6DE13CDE0217]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2121224 INFO (SUITE-TestImpersonationWithHadoopAuth-seed#[A2FF6DE13CDE0217]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45373/solr ready
[junit4] 2> 2121238 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyValidateAnyHostAnyUser-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateAnyHostAnyUser
[junit4] 2> 2121255 INFO (qtp657045337-19271) [n:127.0.0.1:44079_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&user.name=anyHostAnyUser&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2121255 INFO (qtp657045337-19271) [n:127.0.0.1:44079_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&user.name=anyHostAnyUser&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2121256 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyValidateAnyHostAnyUser-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateAnyHostAnyUser
[junit4] 2> 2121270 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyValidateGroup-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateGroup
[junit4] 2> 2121278 INFO (qtp657045337-19280) [n:127.0.0.1:44079_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=jenkins&action=LIST&user.name=anyHostUsersGroup&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2121278 INFO (qtp657045337-19280) [n:127.0.0.1:44079_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=jenkins&action=LIST&user.name=anyHostUsersGroup&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2121278 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyValidateGroup-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateGroup
[junit4] 2> 2121292 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyNullProxyUser-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNullProxyUser
[junit4] 2> 2121294 INFO (qtp657045337-19282) [ ] o.a.s.s.SolrDispatchFilter Error authenticating
[junit4] 2> java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
[junit4] 2> at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:195)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
[junit4] 2> at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
[junit4] 2> at org.apache.solr.security.HadoopAuthFilter.doFilter(HadoopAuthFilter.java:110)
[junit4] 2> at org.apache.solr.security.HadoopAuthPlugin.doAuthenticate(HadoopAuthPlugin.java:245)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 2121295 WARN (qtp657045337-19282) [ ] o.e.j.s.ServletHandler /solr/admin/collections
[junit4] 2> org.apache.solr.common.SolrException: Error during request authentication,
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:457)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:347)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> Caused by: java.lang.IllegalArgumentException: userName is NULL, empty or contains a '&'
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.checkForIllegalArgument(AuthToken.java:87)
[junit4] 2> at org.apache.hadoop.security.authentication.util.AuthToken.<init>(AuthToken.java:70)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationToken.<init>(AuthenticationToken.java:58)
[junit4] 2> at org.apache.hadoop.security.authentication.server.PseudoAuthenticationHandler.authenticate(PseudoAuthenticationHandler.java:195)
[junit4] 2> at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:348)
[junit4] 2> at org.apache.solr.security.RequestContinuesRecorderAuthenticationHandler.authenticate(RequestContinuesRecorderAuthenticationHandler.java:69)
[junit4] 2> at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:518)
[junit4] 2> at org.apache.solr.security.HadoopAuthFilter.doFilter(HadoopAuthFilter.java:110)
[junit4] 2> at org.apache.solr.security.HadoopAuthPlugin.doAuthenticate(HadoopAuthPlugin.java:245)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:451)
[junit4] 2> ... 29 more
[junit4] 2> 2121295 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyNullProxyUser-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNullProxyUser
[junit4] 2> 2121310 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyValidateHost-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyValidateHost
[junit4] 2> 2121313 INFO (qtp657045337-19282) [n:127.0.0.1:44079_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :list with params doAs=bar&action=LIST&user.name=localHostAnyGroup&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2121313 INFO (qtp657045337-19282) [n:127.0.0.1:44079_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={doAs=bar&action=LIST&user.name=localHostAnyGroup&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2121313 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyValidateHost-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyValidateHost
[junit4] 2> 2121330 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigHosts-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigHosts
[junit4] 2> 2121334 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigHosts-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigHosts
[junit4] 2> 2121351 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidGroup-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidGroup
[junit4] 2> 2121358 WARN (qtp657045337-19280) [n:127.0.0.1:44079_solr ] o.a.h.s.ShellBasedUnixGroupsMapping got exception trying to get groups for user bar: id: ‘bar’: no such user
[junit4] 2> id: ‘bar’: no such user
[junit4] 2>
[junit4] 2> 2121360 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidGroup-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidGroup
[junit4] 2> 2121376 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigGroups-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyNoConfigGroups
[junit4] 2> 2121382 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyNoConfigGroups-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyNoConfigGroups
[junit4] 2> 2121399 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyWrongHost-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyWrongHost
[junit4] 2> 2136415 WARN (jetty-launcher-2855-thread-1-SendThread(127.0.0.1:45373)) [n:127.0.0.1:40037_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15240ms for sessionid 0x15ebedbb9280007
[junit4] 2> 2136415 WARN (jetty-launcher-2855-thread-2-SendThread(127.0.0.1:45373)) [n:127.0.0.1:44079_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15196ms for sessionid 0x15ebedbb9280006
[junit4] 2> 2136415 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ebedbb9280007, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 2136415 WARN (jetty-launcher-2855-thread-2-SendThread(127.0.0.1:45373)) [n:127.0.0.1:44079_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15240ms for sessionid 0x15ebedbb9280008
[junit4] 2> 2136415 WARN (jetty-launcher-2855-thread-1-SendThread(127.0.0.1:45373)) [n:127.0.0.1:40037_solr ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 15216ms for sessionid 0x15ebedbb9280005
[junit4] 2> 2136416 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ebedbb9280006, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 2136416 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ebedbb9280008, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 2136417 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ebedbb9280005, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 2136418 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyWrongHost-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyWrongHost
[junit4] 2> 2136454 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidProxyUser-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testProxyInvalidProxyUser
[junit4] 2> 2136460 INFO (TEST-TestImpersonationWithHadoopAuth.testProxyInvalidProxyUser-seed#[A2FF6DE13CDE0217]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testProxyInvalidProxyUser
[junit4] IGNOR/A 0.00s J1 | TestImpersonationWithHadoopAuth.testForwarding
[junit4] > Assumption #1: 'awaitsfix' test group is disabled (@AwaitsFix(bugUrl=https://issues.apache.org/jira/browse/HADOOP-9893))
[junit4] 2> 2136516 INFO (jetty-launcher-2855-thread-1-EventThread) [n:127.0.0.1:40037_solr ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
[junit4] 2> 2136516 WARN (zkCallback-2867-thread-2-processing-n:127.0.0.1:44079_solr) [n:127.0.0.1:44079_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@71c31545 name: ZooKeeperConnection Watcher:127.0.0.1:45373/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 2136516 WARN (zkCallback-2867-thread-2-processing-n:127.0.0.1:44079_solr) [n:127.0.0.1:44079_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 2136518 INFO (jetty-launcher-2855-thread-2-EventThread) [n:127.0.0.1:44079_solr ] o.a.c.f.s.ConnectionStateManager State change: SUSPENDED
[junit4] 2> 2136520 INFO (jetty-closer-2856-thread-2) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@5e378797{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 2136521 INFO (jetty-closer-2856-thread-2) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1688162843
[junit4] 2> 2136521 INFO (jetty-closer-2856-thread-1) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@73e1442e{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 2136521 INFO (jetty-closer-2856-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 2136522 WARN (zkCallback-2866-thread-1-processing-n:127.0.0.1:40037_solr) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@793c94d6 name: ZooKeeperConnection Watcher:127.0.0.1:45373/solr got event WatchedEvent state:Disconnected type:None path:null path: null type: None
[junit4] 2> 2136527 WARN (zkCallback-2866-thread-1-processing-n:127.0.0.1:40037_solr) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ConnectionManager zkClient has disconnected
[junit4] 2> 2136522 INFO (jetty-closer-2856-thread-1) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1722974918
[junit4] 2> 2136527 INFO (jetty-closer-2856-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3d0b0754: rootName = solr_40037, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@43bb7726
[junit4] 2> 2136528 INFO (jetty-closer-2856-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 2136528 INFO (jetty-closer-2856-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6ea64424: rootName = solr_44079, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@438c94f7
[junit4] 2> 2136536 INFO (jetty-closer-2856-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 2136536 INFO (jetty-closer-2856-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@59ae1512: rootName = solr_40037, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@6a9a1fc4
[junit4] 2> 2136536 INFO (jetty-closer-2856-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 2136536 INFO (jetty-closer-2856-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1e244678: rootName = solr_44079, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@6a9a1fc4
[junit4] 2> 2136542 INFO (jetty-closer-2856-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 2136542 INFO (jetty-closer-2856-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@54406d6c: rootName = solr_40037, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@43513be7
[junit4] 2> 2136542 INFO (jetty-closer-2856-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 2136543 INFO (jetty-closer-2856-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3a70abef: rootName = solr_44079, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@43513be7
[junit4] 2> 2138175 INFO (jetty-closer-2856-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 2138175 INFO (jetty-closer-2856-thread-2) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 2138176 WARN (zkCallback-2866-thread-2-processing-n:127.0.0.1:40037_solr) [n:127.0.0.1:40037_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 2138176 ERROR (Thread[Thread-5416,5,TGRP-TestImpersonationWithHadoopAuth]) [n:127.0.0.1:40037_solr ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
[junit4] 2> 2138241 INFO (jetty-closer-2856-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@22e6e184{/solr,null,UNAVAILABLE}
[junit4] 2> 2138271 INFO (zkCallback-2867-thread-2-processing-n:127.0.0.1:44079_solr) [n:127.0.0.1:44079_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 2138272 INFO (jetty-closer-2856-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 2138272 INFO (jetty-closer-2856-thread-1) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 2138272 INFO (jetty-closer-2856-thread-1) [ ] o.a.s.c.Overseer Overseer (id=98726092762251270-127.0.0.1:44079_solr-n_0000000000) closing
[junit4] 2> 2138272 INFO (OverseerStateUpdate-98726092762251270-127.0.0.1:44079_solr-n_0000000000) [n:127.0.0.1:44079_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:44079_solr
[junit4] 2> 2138273 WARN (zkCallback-2867-thread-2-processing-n:127.0.0.1:44079_solr) [n:127.0.0.1:44079_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 2138273 ERROR (Thread[Thread-5418,5,TGRP-TestImpersonationWithHadoopAuth]) [n:127.0.0.1:44079_solr ] o.a.h.s.t.d.AbstractDelegationTokenSecretManager ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
[junit4] 2> 2138275 INFO (jetty-closer-2856-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3b367515{/solr,null,UNAVAILABLE}
[junit4] 2> 2138276 ERROR (SUITE-TestImpersonationWithHadoopAuth-seed#[A2FF6DE13CDE0217]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2138276 INFO (SUITE-TestImpersonationWithHadoopAuth-seed#[A2FF6DE13CDE0217]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45373 45373
[junit4] 2> 2143292 INFO (Thread-5399) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45373 45373
[junit4] 2> 2143294 WARN (Thread-5399) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/aliases.json
[junit4] 2> 2 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/clusterstate.json
[junit4] 2> 3 /solr/clusterprops.json
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMKeyIdRoot
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_1
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_3
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_2
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot/DK_4
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMSeqNumRoot
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/collections
[junit4] 2> 3 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMMasterKeyRoot
[junit4] 2> 2 /solr/security/zkdtsm/ZKDTSMRoot/ZKDTSMTokensRoot
[junit4] 2>
[junit4] 2> 2143294 INFO (SUITE-TestImpersonationWithHadoopAuth-seed#[A2FF6DE13CDE0217]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
[junit4] 2> Sep 26, 2017 3:44:06 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
[junit4] 2> Sep 26, 2017 3:44:26 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> SEVERE: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth:
[junit4] 2> 1) Thread[id=19322, name=jetty-launcher-2855-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
[junit4] 2> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
[junit4] 2> at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
[junit4] 2> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] 2> at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] 2> 2) Thread[id=19317, name=jetty-launcher-2855-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
[junit4] 2> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
[junit4] 2> at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
[junit4] 2> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] 2> at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] 2> Sep 26, 2017 3:44:26 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=19322, name=jetty-launcher-2855-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
[junit4] 2> 2) Thread[id=19317, name=jetty-launcher-2855-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
[junit4] 2> 2163504 ERROR (jetty-launcher-2855-thread-2-EventThread) [n:127.0.0.1:44079_solr ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
[junit4] 2> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
[junit4] 2> at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
[junit4] 2> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] 2> at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] 2> 2163504 ERROR (jetty-launcher-2855-thread-1-EventThread) [n:127.0.0.1:40037_solr ] o.a.c.f.i.CuratorFrameworkImpl Watcher exception
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1039)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
[junit4] 2> at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
[junit4] 2> at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
[junit4] 2> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] 2> at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] 2> at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] 2> at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] 2> Sep 26, 2017 3:44:26 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: All leaked threads terminated.
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70), sim=RandomSimilarity(queryNorm=false): {}, locale=be, timezone=CST6CDT
[junit4] 2> NOTE: Linux 4.10.0-33-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=8,threads=1,free=255311920,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [MetricUtilsTest, ZkSolrClientTest, SpatialFilterTest, CurrencyFieldTypeTest, TestFastLRUCache, TestLRUCache, ConnectionManagerTest, HdfsLockFactoryTest, TestXIncludeConfig, TestSha256AuthenticationProvider, TestFieldSortValues, OverriddenZkACLAndCredentialsProvidersTest, SmileWriterTest, HdfsThreadLeakTest, ClusterStateUpdateTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestLocalFSCloudBackupRestore, GraphQueryTest, DistanceFunctionTest, TestNumericTerms32, TestDFISimilarityFactory, LeaderFailoverAfterPartitionTest, TestHashQParserPlugin, TestSystemIdResolver, TestShortCircuitedRequests, SolrJmxReporterCloudTest, ChaosMonkeySafeLeaderTest, StatelessScriptUpdateProcessorFactoryTest, SOLR749Test, ShardSplitTest, SecurityConfHandlerTest, TestDownShardTolerantSearch, SpellCheckComponentTest, TestApiFramework, DistributedQueryComponentCustomSortTest, CheckHdfsIndexTest, BaseCdcrDistributedZkTest, BadCopyFieldTest, AlternateDirectoryTest, TestConfigSetsAPI, LoggingHandlerTest, TestDocBasedVersionConstraints, SolrXmlInZkTest, TestCryptoKeys, BitVectorTest, AnalysisAfterCoreReloadTest, TestInitParams, TestSQLHandlerNonCloud, TermVectorComponentTest, TolerantUpdateProcessorTest, RuleEngineTest, HdfsBasicDistributedZkTest, TestNumericRangeQuery32, HdfsWriteToMultipleCollectionsTest, TestClusterProperties, TestHttpShardHandlerFactory, SchemaVersionSpecificBehaviorTest, MoreLikeThisHandlerTest, TestDistributedGrouping, AutoScalingHandlerTest, ScriptEngineTest, TestSchemaNameResource, TestQueryTypes, SortSpecParsingTest, TriLevelCompositeIdRoutingTest, TestTolerantUpdateProcessorCloud, DocValuesTest, DistributedFacetExistsSmallTest, CoreAdminHandlerTest, AtomicUpdatesTest, CSVRequestHandlerTest, TestSchemaVersionResource, ShufflingReplicaListTransformerTest, BasicFunctionalityTest, ShardRoutingCustomTest, TestMaxTokenLenTokenizer, TestAtomicUpdateErrorCases, TestAnalyzeInfixSuggestions, TestStressRecovery, IndexSchemaTest, SuggesterTSTTest, RollingRestartTest, CollectionsAPISolrJTest, TestMacroExpander, ZkControllerTest, CoreSorterTest, TestFreeTextSuggestions, TestExactSharedStatsCache, AnalyticsMergeStrategyTest, TestRTGBase, CleanupOldIndexTest, TestJavabinTupleStreamParser, SimpleCollectionCreateDeleteTest, ConnectionReuseTest, TestSolrXml, ConvertedLegacyTest, TestManagedSchemaThreadSafety, ReplaceNodeTest, ZkCLITest, RequestLoggingTest, TestSolrCloudWithKerberosAlt, TestHalfAndHalfDocValues, FileUtilsTest, TestPerFieldSimilarity, TestChildDocTransformer, TestFiltering, TestEmbeddedSolrServerConstructors, TestValueSourceCache, BasicAuthIntegrationTest, DistributedFacetPivotSmallAdvancedTest, CollectionsAPIDistributedZkTest, TestLegacyNumericUtils, TestSlowCompositeReaderWrapper, TestSolrJ, MultiThreadedOCPTest, TestRebalanceLeaders, ForceLeaderTest, TestSimpleTrackingShardHandler, TestFieldCacheReopen, RemoteQueryErrorTest, SharedFSAutoReplicaFailoverTest, TestAuthenticationFramework, TestExclusionRuleCollectionAccess, TestHdfsCloudBackupRestore, TestLeaderElectionZkExpiry, TestOnReconnectListenerSupport, TestPullReplicaErrorHandling, TestRandomRequestDistribution, TestRequestForwarding, TestSSLRandomization, TestStressCloudBlindAtomicUpdates, TestTolerantUpdateProcessorRandomCloud, TestPolicyCloud, ZkStateReaderTest, ZkStateWriterTest, ImplicitSnitchTest, BlobRepositoryCloudTest, DirectoryFactoryTest, ExitableDirectoryReaderTest, HdfsDirectoryFactoryTest, TestInfoStreamLogging, TestReloadAndDeleteDocs, TestBlobHandler, TestConfigReload, TestSolrConfigHandlerConcurrent, CoreAdminCreateDiscoverTest, CoreAdminRequestStatusTest, CoreMergeIndexesAdminHandlerTest, InfoHandlerTest, MetricsHandlerTest, SegmentsInfoRequestHandlerTest, StatsReloadRaceTest, DistributedFacetPivotWhiteBoxTest, TestExpandComponent, TestPivotHelperCode, TestTrackingShardHandlerFactory, JSONWriterTest, TestBinaryResponseWriter, TestCustomDocTransformer, TestExportWriter, TestGraphMLResponseWriter, TestRawTransformer, TestRetrieveFieldsOptimizer, TestSolrQueryResponse, TestSubQueryTransformer, TestSubQueryTransformerDistrib, TestRestManager, TestCopyFieldCollectionResource, TestDynamicFieldCollectionResource, TestFieldResource, TestSchemaSimilarityResource, TestManagedStopFilterFactory, TestManagedSynonymFilterFactory, DocValuesMissingTest, EnumFieldTest, PreAnalyzedFieldManagedSchemaCloudTest, TestCollationFieldDocValues, TestManagedSchema, TestManagedSchemaAPI, TestSchemalessBufferedUpdates, TestUseDocValuesAsStored2, LargeFieldTest, RankQueryTest, TestCollapseQParserPlugin, TestCustomSort, TestInitQParser, TestNoOpRegenerator, TestPayloadScoreQParserPlugin, TestSmileRequest, TestJsonFacetRefinement, BlockJoinFacetRandomTest, TestCloudNestedDocsSort, TestScoreJoinQPNoScore, PKIAuthenticationIntegrationTest, TestDelegationWithHadoopAuth, TestImpersonationWithHadoopAuth]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestImpersonationWithHadoopAuth -Dtests.seed=A2FF6DE13CDE0217 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=be -Dtests.timezone=CST6CDT -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J1 | TestImpersonationWithHadoopAuth (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.security.hadoop.TestImpersonationWithHadoopAuth:
[junit4] > 1) Thread[id=19322, name=jetty-launcher-2855-thread-2-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
[junit4] > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
[junit4] > at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
[junit4] > at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] > at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] > 2) Thread[id=19317, name=jetty-launcher-2855-thread-1-EventThread, state=TIMED_WAITING, group=TGRP-TestImpersonationWithHadoopAuth]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
[junit4] > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
[junit4] > at org.apache.curator.CuratorZookeeperClient.internalBlockUntilConnectedOrTimedOut(CuratorZookeeperClient.java:323)
[junit4] > at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:105)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
[junit4] > at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue.readValue(SharedValue.java:244)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue.access$100(SharedValue.java:44)
[junit4] > at org.apache.curator.framework.recipes.shared.SharedValue$1.process(SharedValue.java:61)
[junit4] > at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:67)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:530)
[junit4] > at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:505)
[junit4] > at __randomizedtesting.SeedInfo.seed([A2FF6DE13CDE0217]:0)
[junit4] Completed [711/732 (1!)] on J1 in 42.60s, 10 tests, 1 error, 1 skipped <<< FAILURES!
[...truncated 46727 lines...]
Re: [JENKINS] Lucene-Solr-7.x-Linux (32bit/jdk1.8.0_144) - Build #
493 - Still Unstable!
Posted by Joel Bernstein <jo...@gmail.com>.
I just updated https://issues.apache.org/jira/browse/SOLR-11392. The latest
comment shows a possible problem with the Solr Cloud test frame work.
I'll ping Dat and Shalin on the ticket.
Joel Bernstein
http://joelsolr.blogspot.com/
On Tue, Sep 26, 2017 at 9:05 AM, Policeman Jenkins Server <
jenkins@thetaphi.de> wrote:
> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/493/
> Java: 32bit/jdk1.8.0_144 -server -XX:+UseConcMarkSweepGC
>
> 1 tests failed.
> FAILED: org.apache.solr.client.solrj.io.stream.StreamExpressionTest.
> testParallelExecutorStream
>
> Error Message:
> Error from server at https://127.0.0.1:40441/solr/
> workQueue_shard2_replica_n3: Expected mime type application/octet-stream
> but got text/html. <html> <head> <meta http-equiv="Content-Type"
> content="text/html;charset=ISO-8859-1"/> <title>Error 404 </title>
> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing
> /solr/workQueue_shard2_replica_n3/update. Reason: <pre> Can not find:
> /solr/workQueue_shard2_replica_n3/update</pre></p> <hr /><a href="
> http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
> </body> </html>
>
> Stack Trace:
> org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error
> from server at https://127.0.0.1:40441/solr/workQueue_shard2_replica_n3:
> Expected mime type application/octet-stream but got text/html. <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
> <title>Error 404 </title>
> </head>
> <body>
> <h2>HTTP ERROR: 404</h2>
> <p>Problem accessing /solr/workQueue_shard2_replica_n3/update. Reason:
> <pre> Can not find: /solr/workQueue_shard2_replica_n3/update</pre></p>
> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty://
> 9.3.20.v20170531</a><hr/>
> </body>
> </html>
>
> at __randomizedtesting.SeedInfo.seed([1A8F22F34F79AFDB:
> A79857EA76559286]:0)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.
> directUpdate(CloudSolrClient.java:539)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.
> sendRequest(CloudSolrClient.java:993)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.
> requestWithRetryOnStaleState(CloudSolrClient.java:862)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(
> CloudSolrClient.java:793)
> at org.apache.solr.client.solrj.SolrRequest.process(
> SolrRequest.java:178)
> at org.apache.solr.client.solrj.request.UpdateRequest.commit(
> UpdateRequest.java:233)
> at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.
> testParallelExecutorStream(StreamExpressionTest.java:7333)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(
> RandomizedRunner.java:1737)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(
> RandomizedRunner.java:934)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(
> RandomizedRunner.java:970)
> at com.carrotsearch.randomizedtesting.
> RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
> at com.carrotsearch.randomizedtesting.rules.
> SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.
> java:57)
> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(
> TestRuleSetupTeardownChained.java:49)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(
> AbstractBeforeAfterRule.java:45)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(
> TestRuleThreadAndTestName.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures
> $1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleMarkFailure$1.
> evaluate(TestRuleMarkFailure.java:47)
> at com.carrotsearch.randomizedtesting.rules.
> StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$
> StatementRunner.run(ThreadLeakControl.java:368)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.
> forkTimeoutingTask(ThreadLeakControl.java:817)
> at com.carrotsearch.randomizedtesting.
> ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.
> runSingleTest(RandomizedRunner.java:943)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(
> RandomizedRunner.java:829)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(
> RandomizedRunner.java:879)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(
> RandomizedRunner.java:890)
> at com.carrotsearch.randomizedtesting.rules.
> StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.
> SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.
> java:57)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(
> AbstractBeforeAfterRule.java:45)
> at com.carrotsearch.randomizedtesting.rules.
> StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(
> TestRuleStoreClassName.java:41)
> at com.carrotsearch.randomizedtesting.rules.
> NoShadowingOrOverridesOnMethodsRule$1.evaluate(
> NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.
> NoShadowingOrOverridesOnMethodsRule$1.evaluate(
> NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.
> StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.
> StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.
> StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(
> TestRuleAssertionsRequired.java:53)
> at org.apache.lucene.util.TestRuleMarkFailure$1.
> evaluate(TestRuleMarkFailure.java:47)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures
> $1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(
> TestRuleIgnoreTestSuites.java:54)
> at com.carrotsearch.randomizedtesting.rules.
> StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$
> StatementRunner.run(ThreadLeakControl.java:368)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error from server at https://127.0.0.1:40441/solr/
> workQueue_shard2_replica_n3: Expected mime type application/octet-stream
> but got text/html. <html>
> <head>
> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
> <title>Error 404 </title>
> </head>
> <body>
> <h2>HTTP ERROR: 404</h2>
> <p>Problem accessing /solr/workQueue_shard2_replica_n3/update. Reason:
> <pre> Can not find: /solr/workQueue_shard2_replica_n3/update</pre></p>
> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty://
> 9.3.20.v20170531</a><hr/>
> </body>
> </html>
>
> at org.apache.solr.client.solrj.impl.HttpSolrClient.
> executeMethod(HttpSolrClient.java:591)
> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> HttpSolrClient.java:253)
> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(
> HttpSolrClient.java:242)
> at org.apache.solr.client.solrj.impl.LBHttpSolrClient.
> doRequest(LBHttpSolrClient.java:483)
> at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(
> LBHttpSolrClient.java:413)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$
> directUpdate$0(CloudSolrClient.java:516)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$
> MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:624)
> ... 1 more
>
>
>
>
> Build Log:
> [...truncated 13910 lines...]
> [junit4] JVM J0: stdout was not empty, see: /home/jenkins/workspace/
> Lucene-Solr-7.x-Linux/solr/build/solr-core/test/temp/
> junit4-J0-20170926_120641_4815724741315114230021.sysout
> [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
> [junit4] Default case invoked for:
> [junit4] opcode = 0, "Node"
> [junit4] Default case invoked for:
> [junit4] opcode = 0, "Node"
> [junit4] Default case invoked for:
> [junit4] opcode = 201, "Phi"
> [junit4] <<< JVM J0: EOF ----
>
> [...truncated 887 lines...]
> [junit4] Suite: org.apache.solr.client.solrj.
> io.stream.StreamExpressionTest
> [junit4] 2> Creating dataDir: /home/jenkins/workspace/
> Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J0/temp/
> solr.client.solrj.io.stream.StreamExpressionTest_
> 1A8F22F34F79AFDB-001/init-core-data-001
> [junit4] 2> 89531 WARN (SUITE-StreamExpressionTest-
> seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.SolrTestCaseJ4
> startTrackingSearchers: numOpens=4 numCloses=4
> [junit4] 2> 89531 INFO (SUITE-StreamExpressionTest-
> seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.SolrTestCaseJ4 Using
> PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
> [junit4] 2> 89532 INFO (SUITE-StreamExpressionTest-
> seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized
> ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=,
> ssl=NaN, value=NaN, clientAuth=NaN)
> [junit4] 2> 89533 INFO (SUITE-StreamExpressionTest-
> seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom
> sanity checks: test.solr.allowed.securerandom=null &
> java.security.egd=file:/dev/./urandom
> [junit4] 2> 89533 INFO (SUITE-StreamExpressionTest-
> seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.MiniSolrCloudCluster
> Starting cluster of 4 servers in /home/jenkins/workspace/
> Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J0/temp/
> solr.client.solrj.io.stream.StreamExpressionTest_
> 1A8F22F34F79AFDB-001/tempDir-001
> [junit4] 2> 89534 INFO (SUITE-StreamExpressionTest-
> seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK
> TEST SERVER
> [junit4] 2> 89534 INFO (Thread-313) [ ] o.a.s.c.ZkTestServer
> client port:0.0.0.0/0.0.0.0:0
> [junit4] 2> 89534 INFO (Thread-313) [ ] o.a.s.c.ZkTestServer
> Starting server
> [junit4] 2> 89535 ERROR (Thread-313) [ ] o.a.z.s.ZooKeeperServer
> ZKShutdownHandler is not registered, so ZooKeeper server won't take any
> action on ERROR or SHUTDOWN server state changes
> [junit4] 2> 89634 INFO (SUITE-StreamExpressionTest-
> seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.ZkTestServer start zk
> server on port:40651
> [junit4] 2> 89648 INFO (jetty-launcher-207-thread-2) [ ]
> o.e.j.s.Server jetty-9.3.20.v20170531
> [junit4] 2> 89648 INFO (jetty-launcher-207-thread-1) [ ]
> o.e.j.s.Server jetty-9.3.20.v20170531
> [junit4] 2> 89650 INFO (jetty-launcher-207-thread-3) [ ]
> o.e.j.s.Server jetty-9.3.20.v20170531
> [junit4] 2> 89651 INFO (jetty-launcher-207-thread-4) [ ]
> o.e.j.s.Server jetty-9.3.20.v20170531
> [junit4] 2> 89662 INFO (jetty-launcher-207-thread-3) [ ]
> o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@
> 1602ddd{/solr,null,AVAILABLE}
> [junit4] 2> 89668 INFO (jetty-launcher-207-thread-4) [ ]
> o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@
> 8a9d8b{/solr,null,AVAILABLE}
> [junit4] 2> 89668 INFO (jetty-launcher-207-thread-3) [ ]
> o.e.j.s.AbstractConnector Started ServerConnector@fb5d1c{SSL,[ssl,
> http/1.1]}{127.0.0.1:38431}
> [junit4] 2> 89668 INFO (jetty-launcher-207-thread-3) [ ]
> o.e.j.s.Server Started @91096ms
> [junit4] 2> 89668 INFO (jetty-launcher-207-thread-3) [ ]
> o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr,
> hostPort=38431}
> [junit4] 2> 89669 ERROR (jetty-launcher-207-thread-3) [ ]
> o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may
> be missing or incomplete.
> [junit4] 2> 89669 INFO (jetty-launcher-207-thread-3) [ ]
> o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢
> version 7.1.0
> [junit4] 2> 89669 INFO (jetty-launcher-207-thread-3) [ ]
> o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port
> null
> [junit4] 2> 89669 INFO (jetty-launcher-207-thread-3) [ ]
> o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default
> config dir: null
> [junit4] 2> 89669 INFO (jetty-launcher-207-thread-3) [ ]
> o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
> 2017-09-26T12:57:48.132Z
> [junit4] 2> 89671 INFO (jetty-launcher-207-thread-1) [ ]
> o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@
> 53b3f5{/solr,null,AVAILABLE}
> [junit4] 2> 89671 INFO (jetty-launcher-207-thread-2) [ ]
> o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@
> 72f089{/solr,null,AVAILABLE}
> [junit4] 2> 89671 INFO (jetty-launcher-207-thread-2) [ ]
> o.e.j.s.AbstractConnector Started ServerConnector@db9c64{SSL,[ssl,
> http/1.1]}{127.0.0.1:41543}
> [junit4] 2> 89671 INFO (jetty-launcher-207-thread-2) [ ]
> o.e.j.s.Server Started @91099ms
> [junit4] 2> 89671 INFO (jetty-launcher-207-thread-2) [ ]
> o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr,
> hostPort=41543}
> [junit4] 2> 89672 INFO (jetty-launcher-207-thread-1) [ ]
> o.e.j.s.AbstractConnector Started ServerConnector@127ade3{SSL,[ssl,
> http/1.1]}{127.0.0.1:40441}
> [junit4] 2> 89672 INFO (jetty-launcher-207-thread-1) [ ]
> o.e.j.s.Server Started @91099ms
> [junit4] 2> 89672 INFO (jetty-launcher-207-thread-1) [ ]
> o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr,
> hostPort=40441}
> [junit4] 2> 89672 ERROR (jetty-launcher-207-thread-2) [ ]
> o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may
> be missing or incomplete.
> [junit4] 2> 89672 ERROR (jetty-launcher-207-thread-1) [ ]
> o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may
> be missing or incomplete.
> [junit4] 2> 89672 INFO (jetty-launcher-207-thread-4) [ ]
> o.e.j.s.AbstractConnector Started ServerConnector@f4f4e7{SSL,[ssl,
> http/1.1]}{127.0.0.1:38423}
> [junit4] 2> 89672 INFO (jetty-launcher-207-thread-1) [ ]
> o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢
> version 7.1.0
> [junit4] 2> 89672 INFO (jetty-launcher-207-thread-4) [ ]
> o.e.j.s.Server Started @91099ms
> [junit4] 2> 89672 INFO (jetty-launcher-207-thread-1) [ ]
> o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port
> null
> [junit4] 2> 89672 INFO (jetty-launcher-207-thread-4) [ ]
> o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr,
> hostPort=38423}
> [junit4] 2> 89672 INFO (jetty-launcher-207-thread-1) [ ]
> o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default
> config dir: null
> [junit4] 2> 89676 INFO (jetty-launcher-207-thread-1) [ ]
> o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
> 2017-09-26T12:57:48.139Z
> [junit4] 2> 89676 ERROR (jetty-launcher-207-thread-4) [ ]
> o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may
> be missing or incomplete.
> [junit4] 2> 89676 INFO (jetty-launcher-207-thread-4) [ ]
> o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢
> version 7.1.0
> [junit4] 2> 89676 INFO (jetty-launcher-207-thread-4) [ ]
> o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port
> null
> [junit4] 2> 89676 INFO (jetty-launcher-207-thread-4) [ ]
> o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default
> config dir: null
> [junit4] 2> 89676 INFO (jetty-launcher-207-thread-4) [ ]
> o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
> 2017-09-26T12:57:48.139Z
> [junit4] 2> 89672 INFO (jetty-launcher-207-thread-2) [ ]
> o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢
> version 7.1.0
> [junit4] 2> 89679 INFO (jetty-launcher-207-thread-2) [ ]
> o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port
> null
> [junit4] 2> 89679 INFO (jetty-launcher-207-thread-2) [ ]
> o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default
> config dir: null
> [junit4] 2> 89679 INFO (jetty-launcher-207-thread-2) [ ]
> o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time:
> 2017-09-26T12:57:48.142Z
> [junit4] 2> 89696 INFO (jetty-launcher-207-thread-3) [ ]
> o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
> [junit4] 2> 89699 INFO (jetty-launcher-207-thread-1) [ ]
> o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
> [junit4] 2> 89699 INFO (jetty-launcher-207-thread-4) [ ]
> o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
> [junit4] 2> 89735 INFO (jetty-launcher-207-thread-3) [ ]
> o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40651/solr
> [junit4] 2> 89736 INFO (jetty-launcher-207-thread-2) [ ]
> o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
> [junit4] 2> 89738 INFO (jetty-launcher-207-thread-1) [ ]
> o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40651/solr
> [junit4] 2> 89744 INFO (jetty-launcher-207-thread-4) [ ]
> o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40651/solr
> [junit4] 2> 89770 INFO (jetty-launcher-207-thread-2) [ ]
> o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40651/solr
> [junit4] 2> 89807 WARN (jetty-launcher-207-thread-4)
> [n:127.0.0.1:38423_solr ] o.a.s.c.ZkController The _default configset
> could not be uploaded. Please provide 'solr.default.confdir' parameter that
> points to a configset intended to be the default. Current
> 'solr.default.confdir' value: null
> [junit4] 2> 89807 WARN (jetty-launcher-207-thread-3)
> [n:127.0.0.1:38431_solr ] o.a.s.c.ZkController The _default configset
> could not be uploaded. Please provide 'solr.default.confdir' parameter that
> points to a configset intended to be the default. Current
> 'solr.default.confdir' value: null
> [junit4] 2> 89807 WARN (jetty-launcher-207-thread-2)
> [n:127.0.0.1:41543_solr ] o.a.s.c.ZkController The _default configset
> could not be uploaded. Please provide 'solr.default.confdir' parameter that
> points to a configset intended to be the default. Current
> 'solr.default.confdir' value: null
> [junit4] 2> 89808 WARN (jetty-launcher-207-thread-1)
> [n:127.0.0.1:40441_solr ] o.a.s.c.ZkController The _default configset
> could not be uploaded. Please provide 'solr.default.confdir' parameter that
> points to a configset intended to be the default. Current
> 'solr.default.confdir' value: null
> [junit4] 2> 89814 INFO (jetty-launcher-207-thread-3)
> [n:127.0.0.1:38431_solr ] o.a.s.c.Overseer Overseer (id=null) closing
> [junit4] 2> 89814 INFO (jetty-launcher-207-thread-4)
> [n:127.0.0.1:38423_solr ] o.a.s.c.Overseer Overseer (id=null) closing
> [junit4] 2> 89814 INFO (jetty-launcher-207-thread-1)
> [n:127.0.0.1:40441_solr ] o.a.s.c.Overseer Overseer (id=null) closing
> [junit4] 2> 89814 INFO (jetty-launcher-207-thread-2)
> [n:127.0.0.1:41543_solr ] o.a.s.c.Overseer Overseer (id=null) closing
> [junit4] 2> 89814 INFO (jetty-launcher-207-thread-3)
> [n:127.0.0.1:38431_solr ] o.a.s.c.OverseerElectionContext I am going
> to be the leader 127.0.0.1:38431_solr
> [junit4] 2> 89815 INFO (jetty-launcher-207-thread-4)
> [n:127.0.0.1:38423_solr ] o.a.s.c.ZkController Register node as live
> in ZooKeeper:/live_nodes/127.0.0.1:38423_solr
> [junit4] 2> 89815 INFO (jetty-launcher-207-thread-2)
> [n:127.0.0.1:41543_solr ] o.a.s.c.ZkController Register node as live
> in ZooKeeper:/live_nodes/127.0.0.1:41543_solr
> [junit4] 2> 89815 INFO (jetty-launcher-207-thread-1)
> [n:127.0.0.1:40441_solr ] o.a.s.c.ZkController Register node as live
> in ZooKeeper:/live_nodes/127.0.0.1:40441_solr
> [junit4] 2> 89815 INFO (jetty-launcher-207-thread-3)
> [n:127.0.0.1:38431_solr ] o.a.s.c.Overseer Overseer
> (id=98725440257851403-127.0.0.1:38431_solr-n_0000000000) starting
> [junit4] 2> 89822 INFO (zkCallback-229-thread-1-
> processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
> [junit4] 2> 89822 INFO (zkCallback-227-thread-1-
> processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
> [junit4] 2> 89834 INFO (zkCallback-226-thread-1-
> processing-n:127.0.0.1:38423_solr) [n:127.0.0.1:38423_solr ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
> [junit4] 2> 89835 INFO (zkCallback-228-thread-1-
> processing-n:127.0.0.1:40441_solr) [n:127.0.0.1:40441_solr ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
> [junit4] 2> 89861 INFO (jetty-launcher-207-thread-3)
> [n:127.0.0.1:38431_solr ] o.a.s.c.ZkController Register node as live
> in ZooKeeper:/live_nodes/127.0.0.1:38431_solr
> [junit4] 2> 89861 INFO (zkCallback-229-thread-1-
> processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
> [junit4] 2> 89861 INFO (zkCallback-228-thread-1-
> processing-n:127.0.0.1:40441_solr) [n:127.0.0.1:40441_solr ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
> [junit4] 2> 89861 INFO (zkCallback-227-thread-1-
> processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
> [junit4] 2> 89863 INFO (zkCallback-226-thread-1-
> processing-n:127.0.0.1:38423_solr) [n:127.0.0.1:38423_solr ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
> [junit4] 2> 89924 INFO (jetty-launcher-207-thread-2)
> [n:127.0.0.1:41543_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr_41543.solr.node' (registry 'solr.node') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
> [junit4] 2> 89935 INFO (jetty-launcher-207-thread-2)
> [n:127.0.0.1:41543_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr_41543.solr.jvm' (registry 'solr.jvm') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
> [junit4] 2> 89935 INFO (jetty-launcher-207-thread-2)
> [n:127.0.0.1:41543_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr_41543.solr.jetty' (registry 'solr.jetty') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
> [junit4] 2> 89936 INFO (jetty-launcher-207-thread-2)
> [n:127.0.0.1:41543_solr ] o.a.s.c.CorePropertiesLocator Found 0 core
> definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/
> build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.
> StreamExpressionTest_1A8F22F34F79AFDB-001/tempDir-001/node2/.
> [junit4] 2> 89939 INFO (jetty-launcher-207-thread-1)
> [n:127.0.0.1:40441_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr_40441.solr.node' (registry 'solr.node') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
> [junit4] 2> 89947 INFO (jetty-launcher-207-thread-1)
> [n:127.0.0.1:40441_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr_40441.solr.jvm' (registry 'solr.jvm') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
> [junit4] 2> 89947 INFO (jetty-launcher-207-thread-1)
> [n:127.0.0.1:40441_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr_40441.solr.jetty' (registry 'solr.jetty') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
> [junit4] 2> 89957 INFO (jetty-launcher-207-thread-1)
> [n:127.0.0.1:40441_solr ] o.a.s.c.CorePropertiesLocator Found 0 core
> definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/
> build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.
> StreamExpressionTest_1A8F22F34F79AFDB-001/tempDir-001/node1/.
> [junit4] 2> 90056 INFO (jetty-launcher-207-thread-3)
> [n:127.0.0.1:38431_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr_38431.solr.node' (registry 'solr.node') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
> [junit4] 2> 90072 INFO (jetty-launcher-207-thread-3)
> [n:127.0.0.1:38431_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr_38431.solr.jvm' (registry 'solr.jvm') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
> [junit4] 2> 90072 INFO (jetty-launcher-207-thread-3)
> [n:127.0.0.1:38431_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr_38431.solr.jetty' (registry 'solr.jetty') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
> [junit4] 2> 90075 INFO (jetty-launcher-207-thread-3)
> [n:127.0.0.1:38431_solr ] o.a.s.c.CorePropertiesLocator Found 0 core
> definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/
> build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.
> StreamExpressionTest_1A8F22F34F79AFDB-001/tempDir-001/node3/.
> [junit4] 2> 90077 INFO (jetty-launcher-207-thread-4)
> [n:127.0.0.1:38423_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr_38423.solr.node' (registry 'solr.node') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
> [junit4] 2> 90082 INFO (jetty-launcher-207-thread-4)
> [n:127.0.0.1:38423_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr_38423.solr.jvm' (registry 'solr.jvm') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
> [junit4] 2> 90082 INFO (jetty-launcher-207-thread-4)
> [n:127.0.0.1:38423_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for
> 'solr_38423.solr.jetty' (registry 'solr.jetty') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
> [junit4] 2> 90083 INFO (jetty-launcher-207-thread-4)
> [n:127.0.0.1:38423_solr ] o.a.s.c.CorePropertiesLocator Found 0 core
> definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/
> build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.
> StreamExpressionTest_1A8F22F34F79AFDB-001/tempDir-001/node4/.
> [junit4] 2> 90169 INFO (SUITE-StreamExpressionTest-
> seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.c.ZkStateReader Updated
> live nodes from ZooKeeper... (0) -> (4)
> [junit4] 2> 90170 INFO (SUITE-StreamExpressionTest-
> seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider
> Cluster at 127.0.0.1:40651/solr ready
> [junit4] 2> 90226 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr
> ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with
> params replicationFactor=1&collection.configName=conf&
> name=collection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2
> and sendToOCPQueue=true
> [junit4] 2> 90228 INFO (OverseerThreadFactory-458-
> thread-1-processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ]
> o.a.s.c.CreateCollectionCmd Create collection collection1
> [junit4] 2> 90335 INFO (OverseerStateUpdate-
> 98725440257851403-127.0.0.1:38431_solr-n_0000000000) [n:127.0.0.1:38431_solr
> ] o.a.s.c.o.SliceMutator createReplica() {
> [junit4] 2> "operation":"ADDREPLICA",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "shard":"shard1",
> [junit4] 2> "core":"collection1_shard1_replica_n1",
> [junit4] 2> "state":"down",
> [junit4] 2> "base_url":"https://127.0.0.1:41543/solr",
> [junit4] 2> "type":"NRT"}
> [junit4] 2> 90338 INFO (OverseerStateUpdate-
> 98725440257851403-127.0.0.1:38431_solr-n_0000000000) [n:127.0.0.1:38431_solr
> ] o.a.s.c.o.SliceMutator createReplica() {
> [junit4] 2> "operation":"ADDREPLICA",
> [junit4] 2> "collection":"collection1",
> [junit4] 2> "shard":"shard2",
> [junit4] 2> "core":"collection1_shard2_replica_n2",
> [junit4] 2> "state":"down",
> [junit4] 2> "base_url":"https://127.0.0.1:38431/solr",
> [junit4] 2> "type":"NRT"}
> [junit4] 2> 90558 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> ] o.a.s.h.a.CoreAdminOperation core create command
> qt=/admin/cores&coreNodeName=core_node3&collection.
> configName=conf&newCollection=true&name=collection1_shard1_
> replica_n1&action=CREATE&numShards=2&collection=
> collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
> [junit4] 2> 90559 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for
> 2147483647 transient cores
> [junit4] 2> 90618 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> ] o.a.s.h.a.CoreAdminOperation core create command
> qt=/admin/cores&coreNodeName=core_node4&collection.
> configName=conf&newCollection=true&name=collection1_shard2_
> replica_n2&action=CREATE&numShards=2&collection=
> collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
> [junit4] 2> 90618 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for
> 2147483647 transient cores
> [junit4] 2> 90722 INFO (zkCallback-229-thread-1-
> processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ]
> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json]
> for collection [collection1] has occurred - updating... (live nodes size:
> [4])
> [junit4] 2> 90722 INFO (zkCallback-227-thread-1-
> processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ]
> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json]
> for collection [collection1] has occurred - updating... (live nodes size:
> [4])
> [junit4] 2> 90722 INFO (zkCallback-229-thread-2-
> processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ]
> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json]
> for collection [collection1] has occurred - updating... (live nodes size:
> [4])
> [junit4] 2> 90723 INFO (zkCallback-227-thread-2-
> processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ]
> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json]
> for collection [collection1] has occurred - updating... (live nodes size:
> [4])
> [junit4] 2> 91569 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
> [junit4] 2> 91577 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.s.IndexSchema [collection1_shard1_replica_n1] Schema name=test
> [junit4] 2> 91614 WARN (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.s.IndexSchema Field lowerfilt1and2 is not multivalued and destination
> for multiple copyFields (2)
> [junit4] 2> 91614 WARN (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.s.IndexSchema Field text is not multivalued and destination for
> multiple copyFields (3)
> [junit4] 2> 91616 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
> [junit4] 2> 91616 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n1'
> using configuration from collection collection1, trusted=true
> [junit4] 2> 91617 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41543.solr.core.collection1.shard1.replica_n1'
> (registry 'solr.core.collection1.shard1.replica_n1') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
> [junit4] 2> 91617 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
> [junit4] 2> 91617 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.SolrCore [[collection1_shard1_replica_n1] ] Opening new SolrCore
> at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/
> build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.
> StreamExpressionTest_1A8F22F34F79AFDB-001/tempDir-
> 001/node2/collection1_shard1_replica_n1], dataDir=[/home/jenkins/
> workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/
> test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_
> 1A8F22F34F79AFDB-001/tempDir-001/node2/./collection1_
> shard1_replica_n1/data/]
> [junit4] 2> 91625 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
> [junit4] 2> 91640 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.s.IndexSchema [collection1_shard2_replica_n2] Schema name=test
> [junit4] 2> 91675 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.UpdateHandler Using UpdateLog implementation:
> org.apache.solr.update.UpdateLog
> [junit4] 2> 91675 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH
> numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
> [junit4] 2> 91675 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.CommitTracker Hard AutoCommit: disabled
> [junit4] 2> 91676 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.CommitTracker Soft AutoCommit: disabled
> [junit4] 2> 91677 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.s.SolrIndexSearcher Opening [Searcher@14407c3[collection1_shard1_replica_n1]
> main]
> [junit4] 2> 91680 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with
> znodeBase: /configs/conf
> [junit4] 2> 91680 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
> ZooKeeperStorageIO:path=/configs/conf
> [junit4] 2> 91681 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
> [junit4] 2> 91682 INFO (searcherExecutor-467-thread-1-processing-n:
> 127.0.0.1:41543_solr x:collection1_shard1_replica_n1 s:shard1
> c:collection1 r:core_node3) [n:127.0.0.1:41543_solr c:collection1
> s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore
> [collection1_shard1_replica_n1] Registered new searcher Searcher@14407c3
> [collection1_shard1_replica_n1] main{ExitableDirectoryReader(
> UninvertingDirectoryReader())}
> [junit4] 2> 91682 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.UpdateLog Could not find max version in index or recent updates,
> using new clock 1579607046377963520
> [junit4] 2> 91685 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
> [junit4] 2> 91685 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
> [junit4] 2> 91685 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:41543/solr/
> collection1_shard1_replica_n1/
> [junit4] 2> 91685 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
> [junit4] 2> 91685 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.SyncStrategy https://127.0.0.1:41543/solr/
> collection1_shard1_replica_n1/ has no replicas
> [junit4] 2> 91685 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.ShardLeaderElectionContext Found all replicas participating in
> election, clear LIR
> [junit4] 2> 91687 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.ShardLeaderElectionContext I am the new leader:
> https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/ shard1
> [junit4] 2> 91701 WARN (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.s.IndexSchema Field lowerfilt1and2 is not multivalued and destination
> for multiple copyFields (2)
> [junit4] 2> 91701 WARN (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.s.IndexSchema Field text is not multivalued and destination for
> multiple copyFields (3)
> [junit4] 2> 91701 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
> [junit4] 2> 91701 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n2'
> using configuration from collection collection1, trusted=true
> [junit4] 2> 91702 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38431.solr.core.collection1.shard2.replica_n2'
> (registry 'solr.core.collection1.shard2.replica_n2') enabled at server:
> com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
> [junit4] 2> 91702 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
> [junit4] 2> 91702 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.SolrCore [[collection1_shard2_replica_n2] ] Opening new SolrCore
> at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/
> build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.
> StreamExpressionTest_1A8F22F34F79AFDB-001/tempDir-
> 001/node3/collection1_shard2_replica_n2], dataDir=[/home/jenkins/
> workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/
> test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_
> 1A8F22F34F79AFDB-001/tempDir-001/node3/./collection1_
> shard2_replica_n2/data/]
> [junit4] 2> 91733 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.UpdateHandler Using UpdateLog implementation:
> org.apache.solr.update.UpdateLog
> [junit4] 2> 91733 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH
> numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
> [junit4] 2> 91734 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.CommitTracker Hard AutoCommit: disabled
> [junit4] 2> 91734 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.CommitTracker Soft AutoCommit: disabled
> [junit4] 2> 91735 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.s.SolrIndexSearcher Opening [Searcher@16df2e3[collection1_shard2_replica_n2]
> main]
> [junit4] 2> 91736 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with
> znodeBase: /configs/conf
> [junit4] 2> 91736 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using
> ZooKeeperStorageIO:path=/configs/conf
> [junit4] 2> 91737 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
> [junit4] 2> 91737 INFO (searcherExecutor-472-thread-1-processing-n:
> 127.0.0.1:38431_solr x:collection1_shard2_replica_n2 s:shard2
> c:collection1 r:core_node4) [n:127.0.0.1:38431_solr c:collection1
> s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.SolrCore
> [collection1_shard2_replica_n2] Registered new searcher Searcher@16df2e3
> [collection1_shard2_replica_n2] main{ExitableDirectoryReader(
> UninvertingDirectoryReader())}
> [junit4] 2> 91737 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.UpdateLog Could not find max version in index or recent updates,
> using new clock 1579607046435635200
> [junit4] 2> 91741 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
> [junit4] 2> 91741 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
> [junit4] 2> 91741 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:38431/solr/
> collection1_shard2_replica_n2/
> [junit4] 2> 91741 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
> [junit4] 2> 91741 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.SyncStrategy https://127.0.0.1:38431/solr/
> collection1_shard2_replica_n2/ has no replicas
> [junit4] 2> 91741 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.ShardLeaderElectionContext Found all replicas participating in
> election, clear LIR
> [junit4] 2> 91742 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.ShardLeaderElectionContext I am the new leader:
> https://127.0.0.1:38431/solr/collection1_shard2_replica_n2/ shard2
> [junit4] 2> 91877 INFO (zkCallback-229-thread-2-
> processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ]
> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json]
> for collection [collection1] has occurred - updating... (live nodes size:
> [4])
> [junit4] 2> 91877 INFO (zkCallback-227-thread-2-
> processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ]
> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json]
> for collection [collection1] has occurred - updating... (live nodes size:
> [4])
> [junit4] 2> 91877 INFO (zkCallback-229-thread-1-
> processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ]
> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json]
> for collection [collection1] has occurred - updating... (live nodes size:
> [4])
> [junit4] 2> 91877 INFO (zkCallback-227-thread-1-
> processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ]
> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json]
> for collection [collection1] has occurred - updating... (live nodes size:
> [4])
> [junit4] 2> 91926 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.ZkController I am the leader, no recovery necessary
> [junit4] 2> 91926 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.ZkController I am the leader, no recovery necessary
> [junit4] 2> 91928 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> params={qt=/admin/cores&coreNodeName=core_node4&
> collection.configName=conf&newCollection=true&name=
> collection1_shard2_replica_n2&action=CREATE&numShards=2&
> collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT}
> status=0 QTime=1310
> [junit4] 2> 91928 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores
> params={qt=/admin/cores&coreNodeName=core_node3&
> collection.configName=conf&newCollection=true&name=
> collection1_shard1_replica_n1&action=CREATE&numShards=2&
> collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT}
> status=0 QTime=1369
> [junit4] 2> 91933 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr
> ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for
> at most 30 seconds. Check all shard replicas
> [junit4] 2> 92034 INFO (zkCallback-227-thread-1-
> processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ]
> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json]
> for collection [collection1] has occurred - updating... (live nodes size:
> [4])
> [junit4] 2> 92034 INFO (zkCallback-229-thread-2-
> processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ]
> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json]
> for collection [collection1] has occurred - updating... (live nodes size:
> [4])
> [junit4] 2> 92034 INFO (zkCallback-227-thread-2-
> processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ]
> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json]
> for collection [collection1] has occurred - updating... (live nodes size:
> [4])
> [junit4] 2> 92034 INFO (zkCallback-229-thread-1-
> processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ]
> o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
> state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json]
> for collection [collection1] has occurred - updating... (live nodes size:
> [4])
> [junit4] 2> 92230 INFO (OverseerCollectionConfigSetPro
> cessor-98725440257851403-127.0.0.1:38431_solr-n_0000000000)
> [n:127.0.0.1:38431_solr ] o.a.s.c.OverseerTaskQueue Response ZK path:
> /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor
> may have disconnected from ZooKeeper
> [junit4] 2> 92933 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr
> ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections
> params={replicationFactor=1&collection.configName=conf&
> name=collection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2}
> status=0 QTime=2707
> [junit4] 2> 92934 INFO (SUITE-StreamExpressionTest-
> seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.AbstractDistribZkTestBase
> Wait for recoveries to finish - collection: collection1 failOnTimeout:true
> timeout (sec):90
> [junit4] 2> 92935 INFO (SUITE-StreamExpressionTest-
> seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.AbstractDistribZkTestBase
> Recoveries finished - collection: collection1
> [junit4] 2> 92969 INFO (TEST-StreamExpressionTest.
> testCartesianProductStream-seed#[1A8F22F34F79AFDB]) [ ]
> o.a.s.SolrTestCaseJ4 ###Starting testCartesianProductStream
> [junit4] 2> 92990 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2]
> webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=
> https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/
> &wt=javabin&version=2}{deleteByQuery=*:* (-1579607047747403776)} 0 1
> [junit4] 2> 92996 INFO (qtp17705314-1185) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 start commit{_version_=
> 1579607047755792384,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4] 2> 92997 INFO (qtp17705314-1185) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
> [junit4] 2> 92998 INFO (qtp17705314-1185) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 end_commit_flush
> [junit4] 2> 92998 INFO (qtp17705314-1185) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2]
> webapp=/solr path=/update params={update.distrib=
> FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&
> softCommit=false&distrib.from=https://127.0.0.1:41543/solr/
> collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=} 0 2
> [junit4] 2> 93000 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.DirectUpdateHandler2 start commit{_version_=
> 1579607047759986688,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4] 2> 93000 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
> [junit4] 2> 93001 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.DirectUpdateHandler2 end_commit_flush
> [junit4] 2> 93001 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={update.distrib=
> FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&
> softCommit=false&distrib.from=https://127.0.0.1:41543/solr/
> collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=} 0 1
> [junit4] 2> 93002 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=
> javabin&version=2}{deleteByQuery=*:* (-1579607047730626560),commit=} 0 31
> [junit4] 2> 93008 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[0
> (1579607047766278144), 1 (1579607047768375296)]} 0 3
> [junit4] 2> 93011 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.DirectUpdateHandler2 start commit{_version_=
> 1579607047771521024,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4] 2> 93011 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 start commit{_version_=
> 1579607047771521024,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4] 2> 93013 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.SolrIndexWriter Calling setCommitData with
> IW:org.apache.solr.update.SolrIndexWriter@1def314 commitCommandVersion:
> 1579607047771521024
> [junit4] 2> 93013 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
> [junit4] 2> 93013 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 end_commit_flush
> [junit4] 2> 93013 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2]
> webapp=/solr path=/update params={update.distrib=
> FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&
> softCommit=false&distrib.from=https://127.0.0.1:41543/solr/
> collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=} 0 2
> [junit4] 2> 93033 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.s.SolrIndexSearcher Opening [Searcher@10959d5[collection1_shard1_replica_n1]
> main]
> [junit4] 2> 93034 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.DirectUpdateHandler2 end_commit_flush
> [junit4] 2> 93035 INFO (searcherExecutor-467-thread-1-processing-n:
> 127.0.0.1:41543_solr x:collection1_shard1_replica_n1 s:shard1
> c:collection1 r:core_node3) [n:127.0.0.1:41543_solr c:collection1
> s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore
> [collection1_shard1_replica_n1] Registered new searcher Searcher@10959d5
> [collection1_shard1_replica_n1] main{ExitableDirectoryReader(
> UninvertingDirectoryReader(Uninverting(_0(7.1.0):c2)))}
> [junit4] 2> 93035 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={update.distrib=
> FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&
> softCommit=false&distrib.from=https://127.0.0.1:41543/solr/
> collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=} 0 24
> [junit4] 2> 93037 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=javabin&version=2}{commit=}
> 0 27
> [junit4] 2> 93044 INFO (TEST-StreamExpressionTest.
> testCartesianProductStream-seed#[1A8F22F34F79AFDB]) [ ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
> [junit4] 2> 93045 INFO (TEST-StreamExpressionTest.
> testCartesianProductStream-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider
> Cluster at 127.0.0.1:40651/solr ready
> [junit4] 2> 93052 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,a_ss&sort=id+asc&wt=json&version=2.2}
> hits=0 status=0 QTime=0
> [junit4] 2> 93052 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,a_ss&sort=id+asc&wt=json&version=2.2}
> hits=2 status=0 QTime=0
> [junit4] 2> 93059 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,a_ss&sort=id+asc&wt=json&version=2.2}
> hits=2 status=0 QTime=0
> [junit4] 2> 93059 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,a_ss&sort=id+asc&wt=json&version=2.2}
> hits=0 status=0 QTime=0
> [junit4] 2> 93068 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,a_ss,b_ls&sort=id+asc&wt=json&version=2.2}
> hits=0 status=0 QTime=0
> [junit4] 2> 93069 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,a_ss,b_ls&sort=id+asc&wt=json&version=2.2}
> hits=2 status=0 QTime=0
> [junit4] 2> 93073 INFO (qtp17705314-1185) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,a_ss,b_ls&sort=id+asc&wt=json&version=2.2}
> hits=0 status=0 QTime=0
> [junit4] 2> 93077 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,a_ss,b_ls&sort=id+asc&wt=json&version=2.2}
> hits=2 status=0 QTime=0
> [junit4] 2> 93081 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,a_ss,b_ls&sort=id+asc&wt=json&version=2.2}
> hits=0 status=0 QTime=0
> [junit4] 2> 93085 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,a_ss,b_ls&sort=id+asc&wt=json&version=2.2}
> hits=2 status=0 QTime=0
> [junit4] 2> 93087 INFO (TEST-StreamExpressionTest.
> testCartesianProductStream-seed#[1A8F22F34F79AFDB]) [ ]
> o.a.s.SolrTestCaseJ4 ###Ending testCartesianProductStream
> [junit4] 2> 93101 INFO (TEST-StreamExpressionTest.
> testComplementStream-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.SolrTestCaseJ4
> ###Starting testComplementStream
> [junit4] 2> 93104 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.s.SolrIndexSearcher Opening [Searcher@fa82be[collection1_shard1_replica_n1]
> realtime]
> [junit4] 2> 93105 INFO (qtp17705314-1200) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2]
> webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=
> https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/
> &wt=javabin&version=2}{deleteByQuery=*:* (-1579607047866941440)} 0 2
> [junit4] 2> 93106 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.DirectUpdateHandler2 start commit{_version_=
> 1579607047871135744,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4] 2> 93106 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 start commit{_version_=
> 1579607047871135744,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4] 2> 93106 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.SolrIndexWriter Calling setCommitData with
> IW:org.apache.solr.update.SolrIndexWriter@1def314 commitCommandVersion:
> 1579607047871135744
> [junit4] 2> 93106 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
> [junit4] 2> 93106 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 end_commit_flush
> [junit4] 2> 93106 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2]
> webapp=/solr path=/update params={update.distrib=
> FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&
> softCommit=false&distrib.from=https://127.0.0.1:41543/solr/
> collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=} 0 0
> [junit4] 2> 93107 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.s.SolrIndexSearcher Opening [Searcher@1e0e1ff[collection1_shard1_replica_n1]
> main]
> [junit4] 2> 93108 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.DirectUpdateHandler2 end_commit_flush
> [junit4] 2> 93108 INFO (searcherExecutor-467-thread-1-processing-n:
> 127.0.0.1:41543_solr x:collection1_shard1_replica_n1 s:shard1
> c:collection1 r:core_node3) [n:127.0.0.1:41543_solr c:collection1
> s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore
> [collection1_shard1_replica_n1] Registered new searcher Searcher@1e0e1ff
> [collection1_shard1_replica_n1] main{ExitableDirectoryReader(
> UninvertingDirectoryReader())}
> [junit4] 2> 93109 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={update.distrib=
> FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&
> softCommit=false&distrib.from=https://127.0.0.1:41543/solr/
> collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=} 0 2
> [junit4] 2> 93109 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=
> javabin&version=2}{deleteByQuery=*:* (-1579607047865892864),commit=} 0 8
> [junit4] 2> 93112 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[0
> (1579607047875330048), 4 (1579607047877427200), 8 (1579607047877427201)]} 0
> 1
> [junit4] 2> 93119 INFO (qtp17705314-1186) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2]
> webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[2
> (1579607047878475776), 3 (1579607047883718656), 5 (1579607047883718657), 6
> (1579607047883718658), 9 (1579607047884767232), 7 (1579607047884767233)]} 0
> 5
> [junit4] 2> 93121 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.DirectUpdateHandler2 start commit{_version_=
> 1579607047886864384,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4] 2> 93121 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.SolrIndexWriter Calling setCommitData with
> IW:org.apache.solr.update.SolrIndexWriter@1def314 commitCommandVersion:
> 1579607047886864384
> [junit4] 2> 93121 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 start commit{_version_=
> 1579607047886864384,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4] 2> 93121 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.SolrIndexWriter Calling setCommitData with
> IW:org.apache.solr.update.SolrIndexWriter@f43748 commitCommandVersion:
> 1579607047886864384
> [junit4] 2> 93126 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.s.SolrIndexSearcher Opening [Searcher@ccffa4[collection1_shard1_replica_n1]
> main]
> [junit4] 2> 93127 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.s.SolrIndexSearcher Opening [Searcher@179196a[collection1_shard2_replica_n2]
> main]
> [junit4] 2> 93127 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.DirectUpdateHandler2 end_commit_flush
> [junit4] 2> 93127 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 end_commit_flush
> [junit4] 2> 93128 INFO (searcherExecutor-467-thread-1-processing-n:
> 127.0.0.1:41543_solr x:collection1_shard1_replica_n1 s:shard1
> c:collection1 r:core_node3) [n:127.0.0.1:41543_solr c:collection1
> s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore
> [collection1_shard1_replica_n1] Registered new searcher Searcher@ccffa4
> [collection1_shard1_replica_n1] main{ExitableDirectoryReader(
> UninvertingDirectoryReader(Uninverting(_1(7.1.0):c3)))}
> [junit4] 2> 93128 INFO (searcherExecutor-472-thread-1-processing-n:
> 127.0.0.1:38431_solr x:collection1_shard2_replica_n2 s:shard2
> c:collection1 r:core_node4) [n:127.0.0.1:38431_solr c:collection1
> s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.SolrCore
> [collection1_shard2_replica_n2] Registered new searcher Searcher@179196a
> [collection1_shard2_replica_n2] main{ExitableDirectoryReader(
> UninvertingDirectoryReader(Uninverting(_0(7.1.0):c6)))}
> [junit4] 2> 93128 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={update.distrib=
> FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&
> softCommit=false&distrib.from=https://127.0.0.1:41543/solr/
> collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=} 0 7
> [junit4] 2> 93128 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2]
> webapp=/solr path=/update params={update.distrib=
> FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&
> softCommit=false&distrib.from=https://127.0.0.1:41543/solr/
> collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=} 0 7
> [junit4] 2> 93130 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=javabin&version=2}{commit=}
> 0 10
> [junit4] 2> 93135 INFO (TEST-StreamExpressionTest.
> testComplementStream-seed#[1A8F22F34F79AFDB]) [ ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
> [junit4] 2> 93136 INFO (TEST-StreamExpressionTest.
> testComplementStream-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider
> Cluster at 127.0.0.1:40651/solr ready
> [junit4] 2> 93144 INFO (qtp17705314-1186) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr
> path=/select params={q=a_s:(setA+||+setAB)&distrib=false&fl=id,a_s,a_i&
> sort=a_i+asc,+a_s+asc&wt=json&version=2.2} hits=3 status=0 QTime=4
> [junit4] 2> 93145 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/select params={q=a_s:(setA+||+setAB)&distrib=false&fl=id,a_s,a_i&
> sort=a_i+asc,+a_s+asc&wt=json&version=2.2} hits=3 status=0 QTime=5
> [junit4] 2> 93148 INFO (qtp26942402-1201) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/select params={q=a_s:(setB+||+setAB)&distrib=false&fl=id,a_s,a_i&
> sort=a_i+asc&wt=json&version=2.2} hits=1 status=0 QTime=0
> [junit4] 2> 93148 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr
> path=/select params={q=a_s:(setB+||+setAB)&distrib=false&fl=id,a_s,a_i&
> sort=a_i+asc&wt=json&version=2.2} hits=4 status=0 QTime=0
> [junit4] 2> 93155 INFO (TEST-StreamExpressionTest.
> testComplementStream-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.SolrTestCaseJ4
> ###Ending testComplementStream
> [junit4] 2> 93173 INFO (TEST-StreamExpressionTest.
> testAnalyzeEvaluator-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.SolrTestCaseJ4
> ###Starting testAnalyzeEvaluator
> [junit4] 2> 93176 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.s.SolrIndexSearcher Opening [Searcher@1bb0216[collection1_shard1_replica_n1]
> realtime]
> [junit4] 2> 93177 INFO (qtp17705314-1200) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.s.SolrIndexSearcher Opening [Searcher@1f8c50b[collection1_shard2_replica_n2]
> realtime]
> [junit4] 2> 93177 INFO (qtp17705314-1200) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2]
> webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=
> https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/
> &wt=javabin&version=2}{deleteByQuery=*:* (-1579607047944536064)} 0 1
> [junit4] 2> 93179 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.DirectUpdateHandler2 start commit{_version_=
> 1579607047947681792,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4] 2> 93179 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 start commit{_version_=
> 1579607047947681792,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4] 2> 93179 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.SolrIndexWriter Calling setCommitData with
> IW:org.apache.solr.update.SolrIndexWriter@1def314 commitCommandVersion:
> 1579607047947681792
> [junit4] 2> 93179 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.SolrIndexWriter Calling setCommitData with
> IW:org.apache.solr.update.SolrIndexWriter@f43748 commitCommandVersion:
> 1579607047947681792
> [junit4] 2> 93179 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.s.SolrIndexSearcher Opening [Searcher@1c70ab7[collection1_shard1_replica_n1]
> main]
> [junit4] 2> 93179 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.s.SolrIndexSearcher Opening [Searcher@e20936[collection1_shard2_replica_n2]
> main]
> [junit4] 2> 93179 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.DirectUpdateHandler2 end_commit_flush
> [junit4] 2> 93179 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 end_commit_flush
> [junit4] 2> 93180 INFO (searcherExecutor-467-thread-1-processing-n:
> 127.0.0.1:41543_solr x:collection1_shard1_replica_n1 s:shard1
> c:collection1 r:core_node3) [n:127.0.0.1:41543_solr c:collection1
> s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore
> [collection1_shard1_replica_n1] Registered new searcher Searcher@1c70ab7
> [collection1_shard1_replica_n1] main{ExitableDirectoryReader(
> UninvertingDirectoryReader())}
> [junit4] 2> 93180 INFO (searcherExecutor-472-thread-1-processing-n:
> 127.0.0.1:38431_solr x:collection1_shard2_replica_n2 s:shard2
> c:collection1 r:core_node4) [n:127.0.0.1:38431_solr c:collection1
> s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.SolrCore
> [collection1_shard2_replica_n2] Registered new searcher Searcher@e20936
> [collection1_shard2_replica_n2] main{ExitableDirectoryReader(
> UninvertingDirectoryReader())}
> [junit4] 2> 93180 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={update.distrib=
> FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&
> softCommit=false&distrib.from=https://127.0.0.1:41543/solr/
> collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=} 0 1
> [junit4] 2> 93180 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2]
> webapp=/solr path=/update params={update.distrib=
> FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&
> softCommit=false&distrib.from=https://127.0.0.1:41543/solr/
> collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=} 0 1
> [junit4] 2> 93181 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=
> javabin&version=2}{deleteByQuery=*:* (-1579607047942438912),commit=} 0 7
> [junit4] 2> 93190 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[1
> (1579607047950827520)]} 0 8
> [junit4] 2> 93193 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.DirectUpdateHandler2 start commit{_version_=
> 1579607047962361856,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4] 2> 93193 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 start commit{_version_=
> 1579607047962361856,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4] 2> 93193 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.SolrIndexWriter Calling setCommitData with
> IW:org.apache.solr.update.SolrIndexWriter@1def314 commitCommandVersion:
> 1579607047962361856
> [junit4] 2> 93193 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
> [junit4] 2> 93193 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 end_commit_flush
> [junit4] 2> 93193 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2]
> webapp=/solr path=/update params={update.distrib=
> FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&
> softCommit=false&distrib.from=https://127.0.0.1:41543/solr/
> collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=} 0 0
> [junit4] 2> 93196 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.s.SolrIndexSearcher Opening [Searcher@550423[collection1_shard1_replica_n1]
> main]
> [junit4] 2> 93196 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.DirectUpdateHandler2 end_commit_flush
> [junit4] 2> 93197 INFO (searcherExecutor-467-thread-1-processing-n:
> 127.0.0.1:41543_solr x:collection1_shard1_replica_n1 s:shard1
> c:collection1 r:core_node3) [n:127.0.0.1:41543_solr c:collection1
> s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore
> [collection1_shard1_replica_n1] Registered new searcher Searcher@550423
> [collection1_shard1_replica_n1] main{ExitableDirectoryReader(
> UninvertingDirectoryReader(Uninverting(_2(7.1.0):c1)))}
> [junit4] 2> 93197 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={update.distrib=
> FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&
> softCommit=false&distrib.from=https://127.0.0.1:41543/solr/
> collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=} 0 4
> [junit4] 2> 93198 INFO (qtp26942402-1201) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=javabin&version=2}{commit=}
> 0 7
> [junit4] 2> 93202 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/stream params={indent=off&expr=cartesianProduct(search(
> collection1,+q%3D"*:*",+fl%3D"id,+test_t",+sort%3D"id+desc")
> ,+analyze(test_t,+test_t)+as+test_t)&wt=json&version=2.2} status=0 QTime=0
> [junit4] 2> 93205 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
> [junit4] 2> 93205 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40651/solr
> ready
> [junit4] 2> 93207 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,+test_t&sort=id+desc&wt=json&version=2.2}
> hits=0 status=0 QTime=0
> [junit4] 2> 93207 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,+test_t&sort=id+desc&wt=json&version=2.2}
> hits=1 status=0 QTime=0
> [junit4] 2> 93213 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/stream params={indent=off&expr=analyze("hello+world",+test_t)&wt=json&version=2.2}
> status=0 QTime=0
> [junit4] 2> 93216 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/stream params={indent=off&expr=cartesianProduct(search(
> collection1,+q%3D"*:*",+fl%3D"id,+test_t",+sort%3D"id+desc")
> ,+analyze(test_t)+as+test_t)&wt=json&version=2.2} status=0 QTime=0
> [junit4] 2> 93218 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,+test_t&sort=id+desc&wt=json&version=2.2}
> hits=0 status=0 QTime=0
> [junit4] 2> 93218 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,+test_t&sort=id+desc&wt=json&version=2.2}
> hits=1 status=0 QTime=0
> [junit4] 2> 93223 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/stream params={indent=off&expr=cartesianProduct(search(
> collection1,+q%3D"*:*",+fl%3D"id",+sort%3D"id+desc"),+
> analyze(test_t,+test_t)+as+test_t)&wt=json&version=2.2} status=0 QTime=0
> [junit4] 2> 93226 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id&sort=id+desc&wt=json&version=2.2}
> hits=0 status=0 QTime=0
> [junit4] 2> 93226 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id&sort=id+desc&wt=json&version=2.2}
> hits=1 status=0 QTime=0
> [junit4] 2> 93233 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/stream params={indent=off&expr=select(search(collection1,+q%
> 3D"*:*",+fl%3D"id,+test_t",+sort%3D"id+desc"),+analyze(
> test_t,+test_t)+as+test1_t)&wt=json&version=2.2} status=0 QTime=0
> [junit4] 2> 93237 INFO (qtp26942402-1340) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,+test_t&sort=id+desc&wt=json&version=2.2}
> hits=1 status=0 QTime=0
> [junit4] 2> 93237 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr
> path=/select params={q=*:*&distrib=false&fl=id,+test_t&sort=id+desc&wt=json&version=2.2}
> hits=0 status=0 QTime=0
> [junit4] 2> 93239 INFO (TEST-StreamExpressionTest.
> testAnalyzeEvaluator-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.SolrTestCaseJ4
> ###Ending testAnalyzeEvaluator
> [junit4] 2> 93256 INFO (TEST-StreamExpressionTest.
> testCopyOfRange-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.SolrTestCaseJ4
> ###Starting testCopyOfRange
> [junit4] 2> 93261 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.s.SolrIndexSearcher Opening [Searcher@fc6cb7[collection1_shard1_replica_n1]
> realtime]
> [junit4] 2> 93261 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=
> https://127.0.0.1:38431/solr/collection1_shard2_replica_n2/
> &wt=javabin&version=2}{deleteByQuery=*:* (-1579607048032616448)} 0 1
> [junit4] 2> 93266 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.DirectUpdateHandler2 start commit{_version_=
> 1579607048038907904,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4] 2> 93266 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.SolrIndexWriter Calling setCommitData with
> IW:org.apache.solr.update.SolrIndexWriter@1def314 commitCommandVersion:
> 1579607048038907904
> [junit4] 2> 93266 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.s.SolrIndexSearcher Opening [Searcher@1859da1[collection1_shard1_replica_n1]
> main]
> [junit4] 2> 93267 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.DirectUpdateHandler2 end_commit_flush
> [junit4] 2> 93268 INFO (searcherExecutor-467-thread-1-processing-n:
> 127.0.0.1:41543_solr x:collection1_shard1_replica_n1 s:shard1
> c:collection1 r:core_node3) [n:127.0.0.1:41543_solr c:collection1
> s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore
> [collection1_shard1_replica_n1] Registered new searcher Searcher@1859da1
> [collection1_shard1_replica_n1] main{ExitableDirectoryReader(
> UninvertingDirectoryReader())}
> [junit4] 2> 93269 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 start commit{_version_=
> 1579607048042053632,optimize=false,openSearcher=true,waitSearcher=true,
> expungeDeletes=false,softCommit=false,prepareCommit=false}
> [junit4] 2> 93269 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
> [junit4] 2> 93269 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={update.distrib=
> FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&
> softCommit=false&distrib.from=https://127.0.0.1:38431/solr/
> collection1_shard2_replica_n2/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=} 0 3
> [junit4] 2> 93269 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.DirectUpdateHandler2 end_commit_flush
> [junit4] 2> 93269 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2]
> webapp=/solr path=/update params={update.distrib=
> FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&
> softCommit=false&distrib.from=https://127.0.0.1:38431/solr/
> collection1_shard2_replica_n2/&commit_end_point=true&wt=javabin&version=2&
> expungeDeletes=false}{commit=} 0 0
> [junit4] 2> 93270 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2]
> webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=
> javabin&version=2}{deleteByQuery=*:* (-1579607048030519296),commit=} 0 12
> [junit4] 2> 93293 INFO (qtp17705314-1186) [n:127.0.0.1:38431_solr
> c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2]
> webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[id_1
> (1579607048044150784), id_3 (1579607048051490816), id_4
> (1579607048052539392), id_5 (1579607048052539393), id_6
> (1579607048052539394), id_7 (1579607048052539395), id_8
> (1579607048052539396), id_9 (1579607048052539397), id_11
> (1579607048052539398), id_12 (1579607048053587968), ... (125 adds)]} 0 21
> [junit4] 2> 93298 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr
> c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1]
> o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1]
> webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[id_2
> (1579607048044150784), id_10 (1579607048051490816), id_13
> (1579607048052539392), id_14 (1579607048052539393), id_16
> (1579607048052539394), id_17 (1579607048052539395), id_18
> (1579607048053587968), id_21 (1579607048053587969), id_22
> (1579607048053587970), id_23 (157
>
> [...truncated too long message...]
>
> a with IW:org.apache.solr.update.SolrIndexWriter@e208ff
> commitCommandVersion:0
> [junit4] 2> 194257 INFO (coreCloseExecutor-659-thread-1)
> [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3
> x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric
> reporters for registry=solr.collection.collection1.shard1.leader,
> tag=32131367
> [junit4] 2> 194258 INFO (coreCloseExecutor-659-thread-2)
> [n:127.0.0.1:41543_solr c:destination s:shard2 r:core_node4
> x:destination_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric
> reporters for registry=solr.collection.destination.shard2.leader,
> tag=27523539
> [junit4] 2> 194266 INFO (coreCloseExecutor-660-thread-1)
> [n:127.0.0.1:38423_solr c:mainCorpus s:shard1 r:core_node3
> x:mainCorpus_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric
> reporters for registry=solr.collection.mainCorpus.shard1.leader,
> tag=21083153
> [junit4] 2> 194266 INFO (coreCloseExecutor-660-thread-2)
> [n:127.0.0.1:38423_solr c:destination s:shard1 r:core_node3
> x:destination_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric
> reporters for registry=solr.collection.destination.shard1.leader,
> tag=12541154
> [junit4] 2> 194280 INFO (jetty-closer-208-thread-3) [ ]
> o.a.s.c.Overseer Overseer (id=98725440257851403-127.0.0.1:38431_solr-n_0000000000)
> closing
> [junit4] 2> 194280 INFO (OverseerStateUpdate-
> 98725440257851403-127.0.0.1:38431_solr-n_0000000000) [n:127.0.0.1:38431_solr
> ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:38431_solr
> [junit4] 2> 194281 INFO (zkCallback-228-thread-1-
> processing-n:127.0.0.1:40441_solr) [n:127.0.0.1:40441_solr ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
> [junit4] 2> 194281 INFO (zkCallback-227-thread-1-
> processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
> [junit4] 2> 194281 INFO (zkCallback-229-thread-2-
> processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ]
> o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
> [junit4] 2> 194284 INFO (jetty-closer-208-thread-4) [ ]
> o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@
> 8a9d8b{/solr,null,UNAVAILABLE}
> [junit4] 2> 195784 WARN (zkCallback-229-thread-2-
> processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ]
> o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to
> ZK: [KeeperErrorCode = Session expired for /live_nodes]
> [junit4] 2> 195784 WARN (zkCallback-228-thread-2-
> processing-n:127.0.0.1:40441_solr) [n:127.0.0.1:40441_solr ]
> o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to
> ZK: [KeeperErrorCode = Session expired for /live_nodes]
> [junit4] 2> 195784 WARN (zkCallback-227-thread-1-
> processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ]
> o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to
> ZK: [KeeperErrorCode = Session expired for /live_nodes]
> [junit4] 2> 195785 INFO (jetty-closer-208-thread-1) [ ]
> o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@
> 72f089{/solr,null,UNAVAILABLE}
> [junit4] 2> 195785 INFO (jetty-closer-208-thread-2) [ ]
> o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@
> 53b3f5{/solr,null,UNAVAILABLE}
> [junit4] 2> 195785 INFO (jetty-closer-208-thread-3) [ ]
> o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@
> 1602ddd{/solr,null,UNAVAILABLE}
> [junit4] 2> 195785 ERROR (SUITE-StreamExpressionTest-
> seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.z.s.ZooKeeperServer
> ZKShutdownHandler is not registered, so ZooKeeper server won't take any
> action on ERROR or SHUTDOWN server state changes
> [junit4] 2> 195786 INFO (SUITE-StreamExpressionTest-
> seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.ZkTestServer connecting to
> 127.0.0.1:40651 40651
> [junit4] 2> 206582 INFO (Thread-313) [ ] o.a.s.c.ZkTestServer
> connecting to 127.0.0.1:40651 40651
> [junit4] 2> 206583 WARN (Thread-313) [ ] o.a.s.c.ZkTestServer
> Watch limit violations:
> [junit4] 2> Maximum concurrent create/delete watches above limit:
> [junit4] 2>
> [junit4] 2> 93 /solr/aliases.json
> [junit4] 2> 10 /solr/configs/conf
> [junit4] 2> 8 /solr/configs/ml
> [junit4] 2> 4 /solr/security.json
> [junit4] 2>
> [junit4] 2> Maximum concurrent data watches above limit:
> [junit4] 2>
> [junit4] 2> 93 /solr/clusterprops.json
> [junit4] 2> 93 /solr/clusterstate.json
> [junit4] 2> 38 /solr/collections/
> destinationCollection/state.json
> [junit4] 2> 26 /solr/collections/
> parallelDestinationCollection1/state.json
> [junit4] 2> 22 /solr/collections/workQueue/state.json
> [junit4] 2> 20 /solr/collections/destination/state.json
> [junit4] 2> 20 /solr/collections/
> parallelDestinationCollection/state.json
> [junit4] 2> 20 /solr/collections/mainCorpus/state.json
> [junit4] 2> 10 /solr/collections/collection1/state.json
> [junit4] 2> 10 /solr/collections/
> uknownCollection/state.json
> [junit4] 2> 10 /solr/collections/
> modelCollection/state.json
> [junit4] 2> 10 /solr/collections/
> checkpointCollection/state.json
> [junit4] 2> 2 /solr/overseer_elect/election/
> 98725440257851403-127.0.0.1:38431_solr-n_0000000000
> [junit4] 2>
> [junit4] 2> Maximum concurrent children watches above limit:
> [junit4] 2>
> [junit4] 2> 93 /solr/live_nodes
> [junit4] 2> 93 /solr/collections
> [junit4] 2>
> [junit4] 2> NOTE: leaving temporary files on disk at:
> /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/
> build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.
> StreamExpressionTest_1A8F22F34F79AFDB-001
> [junit4] 2> NOTE: test params are: codec=Asserting(Lucene70):
> {name_s=PostingsFormat(name=LuceneFixedGap), expr_s=PostingsFormat(name=Asserting),
> terms_ss=PostingsFormat(name=Direct), join2_s=PostingsFormat(name=Direct),
> field_s=FSTOrd50, multiDefault=PostingsFormat(name=Direct),
> subject=PostingsFormat(name=Direct), a1_s=FSTOrd50,
> a2_s=PostingsFormat(name=Asserting), level1_s=PostingsFormat(name=Asserting),
> body_t=PostingsFormat(name=LuceneFixedGap), s_multi=PostingsFormat(name=Asserting),
> a_ss=PostingsFormat(name=Direct), col_s=FSTOrd50, level2_s=FSTOrd50,
> a_s=PostingsFormat(name=Asserting), tv_text=FSTOrd50,
> term_s=PostingsFormat(name=Direct), a_t=PostingsFormat(name=Direct),
> text_s=PostingsFormat(name=Asserting), id=PostingsFormat(name=LuceneFixedGap),
> text=PostingsFormat(name=Asserting), featureSet_s=FSTOrd50,
> checkpoint_ss=PostingsFormat(name=LuceneFixedGap),
> test_t=PostingsFormat(name=LuceneFixedGap), ident_s=FSTOrd50,
> whitetok=PostingsFormat(name=LuceneFixedGap), side_s=PostingsFormat(name=LuceneFixedGap)},
> docValues:{name_s=DocValuesFormat(name=Lucene70),
> expr_s=DocValuesFormat(name=Lucene70), iteration_i=DocValuesFormat(name=Lucene70),
> terms_ss=DocValuesFormat(name=Direct), join2_s=DocValuesFormat(name=Direct),
> multiDefault=DocValuesFormat(name=Direct), intDefault=DocValuesFormat(name=Asserting),
> a_f=DocValuesFormat(name=Asserting), falseNegative_i=DocValuesFormat(name=Asserting),
> a2_s=DocValuesFormat(name=Lucene70), a_i=DocValuesFormat(name=Lucene70),
> level1_s=DocValuesFormat(name=Lucene70), a_s=DocValuesFormat(name=Lucene70),
> id=DocValuesFormat(name=Lucene70), alpha_d=DocValuesFormat(name=Lucene70),
> i_multi=DocValuesFormat(name=Lucene70), idf_d=DocValuesFormat(name=Asserting),
> b_ls=DocValuesFormat(name=Asserting), checkpoint_ss=DocValuesFormat(name=Lucene70),
> ident_s=DocValuesFormat(name=Asserting), order_i=DocValuesFormat(name=Direct),
> error_d=DocValuesFormat(name=Lucene70), side_s=DocValuesFormat(name=Lucene70),
> truePositive_i=DocValuesFormat(name=Lucene70),
> miles_i=DocValuesFormat(name=Asserting), field_s=DocValuesFormat(name=Asserting),
> price_f=DocValuesFormat(name=Direct), a1_s=DocValuesFormat(name=Asserting),
> join1_i=DocValuesFormat(name=Lucene70), test_i=DocValuesFormat(name=Direct),
> join3_i=DocValuesFormat(name=Lucene70), falsePositive_i=
> DocValuesFormat(name=Asserting), field_i=DocValuesFormat(name=Direct),
> s_multi=DocValuesFormat(name=Lucene70), a_ss=DocValuesFormat(name=Direct),
> col_s=DocValuesFormat(name=Asserting), level2_s=DocValuesFormat(name=Asserting),
> score_f=DocValuesFormat(name=Lucene70), term_s=DocValuesFormat(name=Direct),
> text_s=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Direct),
> idfs_ds=DocValuesFormat(name=Direct), weights_ds=DocValuesFormat(name=Lucene70),
> featureSet_s=DocValuesFormat(name=Asserting), trueNegative_i=
> DocValuesFormat(name=Lucene70), index_i=DocValuesFormat(name=Asserting),
> test_dt=DocValuesFormat(name=Lucene70), out_i=DocValuesFormat(name=Asserting),
> _version_=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1840,
> maxMBSortInHeap=6.295194766277021, sim=RandomSimilarity(queryNorm=true):
> {}, locale=und, timezone=Europe/Bratislava
> [junit4] 2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation
> 1.8.0_144 (32-bit)/cpus=8,threads=1,free=47688920,total=141258752
> [junit4] 2> NOTE: All tests run in this JVM: [GraphTest,
> ContentStreamTest, CommonAdminParamsTest, TestLBHttpSolrClient,
> CollectionAdminRequestRequiredParamsTest, ArcTangentEvaluatorTest,
> SubtractEvaluatorTest, LargeVolumeBinaryJettyTest,
> FieldAnalysisResponseTest, HttpSolrClientSSLAuthConPoolTest,
> TestNamedListCodec, TestRetryUtil, CloudSolrClientMultiConstructorTest,
> AppendEvaluatorTest, StreamExpressionToExpessionTest, TestV2Request,
> ConcatOperationTest, TestXMLEscaping, HyperbolicCosineEvaluatorTest,
> UniformDistributionEvaluatorTest, JDBCStreamTest,
> AbsoluteValueEvaluatorTest, FloorEvaluatorTest, LargeVolumeEmbeddedTest,
> SolrExampleXMLTest, TestEmbeddedSolrServer, TestJavaBinCodec, TestHash,
> TestFastInputStream, AnlysisResponseBaseTest, SolrSchemalessExampleTest,
> CloudSolrClientBuilderTest, ConcurrentUpdateSolrClientTest,
> StreamExpressionTest]
> [junit4] Completed [145/145 (1!)] on J0 in 117.09s, 113 tests, 1 error
> <<< FAILURES!
>
> [...truncated 41595 lines...]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
[JENKINS] Lucene-Solr-7.x-Linux (32bit/jdk1.8.0_144) - Build # 493
- Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/493/
Java: 32bit/jdk1.8.0_144 -server -XX:+UseConcMarkSweepGC
1 tests failed.
FAILED: org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testParallelExecutorStream
Error Message:
Error from server at https://127.0.0.1:40441/solr/workQueue_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 </title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /solr/workQueue_shard2_replica_n3/update. Reason: <pre> Can not find: /solr/workQueue_shard2_replica_n3/update</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/> </body> </html>
Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:40441/solr/workQueue_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/workQueue_shard2_replica_n3/update. Reason:
<pre> Can not find: /solr/workQueue_shard2_replica_n3/update</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>
at __randomizedtesting.SeedInfo.seed([1A8F22F34F79AFDB:A79857EA76559286]:0)
at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:539)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:993)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testParallelExecutorStream(StreamExpressionTest.java:7333)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:40441/solr/workQueue_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/workQueue_shard2_replica_n3/update. Reason:
<pre> Can not find: /solr/workQueue_shard2_replica_n3/update</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:591)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:516)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Build Log:
[...truncated 13910 lines...]
[junit4] JVM J0: stdout was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/temp/junit4-J0-20170926_120641_4815724741315114230021.sysout
[junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
[junit4] Default case invoked for:
[junit4] opcode = 0, "Node"
[junit4] Default case invoked for:
[junit4] opcode = 0, "Node"
[junit4] Default case invoked for:
[junit4] opcode = 201, "Phi"
[junit4] <<< JVM J0: EOF ----
[...truncated 887 lines...]
[junit4] Suite: org.apache.solr.client.solrj.io.stream.StreamExpressionTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_1A8F22F34F79AFDB-001/init-core-data-001
[junit4] 2> 89531 WARN (SUITE-StreamExpressionTest-seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=4 numCloses=4
[junit4] 2> 89531 INFO (SUITE-StreamExpressionTest-seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 89532 INFO (SUITE-StreamExpressionTest-seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 89533 INFO (SUITE-StreamExpressionTest-seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 89533 INFO (SUITE-StreamExpressionTest-seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_1A8F22F34F79AFDB-001/tempDir-001
[junit4] 2> 89534 INFO (SUITE-StreamExpressionTest-seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 89534 INFO (Thread-313) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 89534 INFO (Thread-313) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 89535 ERROR (Thread-313) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 89634 INFO (SUITE-StreamExpressionTest-seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:40651
[junit4] 2> 89648 INFO (jetty-launcher-207-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 89648 INFO (jetty-launcher-207-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 89650 INFO (jetty-launcher-207-thread-3) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 89651 INFO (jetty-launcher-207-thread-4) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 89662 INFO (jetty-launcher-207-thread-3) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1602ddd{/solr,null,AVAILABLE}
[junit4] 2> 89668 INFO (jetty-launcher-207-thread-4) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@8a9d8b{/solr,null,AVAILABLE}
[junit4] 2> 89668 INFO (jetty-launcher-207-thread-3) [ ] o.e.j.s.AbstractConnector Started ServerConnector@fb5d1c{SSL,[ssl, http/1.1]}{127.0.0.1:38431}
[junit4] 2> 89668 INFO (jetty-launcher-207-thread-3) [ ] o.e.j.s.Server Started @91096ms
[junit4] 2> 89668 INFO (jetty-launcher-207-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38431}
[junit4] 2> 89669 ERROR (jetty-launcher-207-thread-3) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 89669 INFO (jetty-launcher-207-thread-3) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 7.1.0
[junit4] 2> 89669 INFO (jetty-launcher-207-thread-3) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 89669 INFO (jetty-launcher-207-thread-3) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 89669 INFO (jetty-launcher-207-thread-3) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T12:57:48.132Z
[junit4] 2> 89671 INFO (jetty-launcher-207-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@53b3f5{/solr,null,AVAILABLE}
[junit4] 2> 89671 INFO (jetty-launcher-207-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@72f089{/solr,null,AVAILABLE}
[junit4] 2> 89671 INFO (jetty-launcher-207-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@db9c64{SSL,[ssl, http/1.1]}{127.0.0.1:41543}
[junit4] 2> 89671 INFO (jetty-launcher-207-thread-2) [ ] o.e.j.s.Server Started @91099ms
[junit4] 2> 89671 INFO (jetty-launcher-207-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41543}
[junit4] 2> 89672 INFO (jetty-launcher-207-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@127ade3{SSL,[ssl, http/1.1]}{127.0.0.1:40441}
[junit4] 2> 89672 INFO (jetty-launcher-207-thread-1) [ ] o.e.j.s.Server Started @91099ms
[junit4] 2> 89672 INFO (jetty-launcher-207-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=40441}
[junit4] 2> 89672 ERROR (jetty-launcher-207-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 89672 ERROR (jetty-launcher-207-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 89672 INFO (jetty-launcher-207-thread-4) [ ] o.e.j.s.AbstractConnector Started ServerConnector@f4f4e7{SSL,[ssl, http/1.1]}{127.0.0.1:38423}
[junit4] 2> 89672 INFO (jetty-launcher-207-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 7.1.0
[junit4] 2> 89672 INFO (jetty-launcher-207-thread-4) [ ] o.e.j.s.Server Started @91099ms
[junit4] 2> 89672 INFO (jetty-launcher-207-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 89672 INFO (jetty-launcher-207-thread-4) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38423}
[junit4] 2> 89672 INFO (jetty-launcher-207-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 89676 INFO (jetty-launcher-207-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T12:57:48.139Z
[junit4] 2> 89676 ERROR (jetty-launcher-207-thread-4) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 89676 INFO (jetty-launcher-207-thread-4) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 7.1.0
[junit4] 2> 89676 INFO (jetty-launcher-207-thread-4) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 89676 INFO (jetty-launcher-207-thread-4) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 89676 INFO (jetty-launcher-207-thread-4) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T12:57:48.139Z
[junit4] 2> 89672 INFO (jetty-launcher-207-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 7.1.0
[junit4] 2> 89679 INFO (jetty-launcher-207-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 89679 INFO (jetty-launcher-207-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 89679 INFO (jetty-launcher-207-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T12:57:48.142Z
[junit4] 2> 89696 INFO (jetty-launcher-207-thread-3) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 89699 INFO (jetty-launcher-207-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 89699 INFO (jetty-launcher-207-thread-4) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 89735 INFO (jetty-launcher-207-thread-3) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40651/solr
[junit4] 2> 89736 INFO (jetty-launcher-207-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 89738 INFO (jetty-launcher-207-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40651/solr
[junit4] 2> 89744 INFO (jetty-launcher-207-thread-4) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40651/solr
[junit4] 2> 89770 INFO (jetty-launcher-207-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40651/solr
[junit4] 2> 89807 WARN (jetty-launcher-207-thread-4) [n:127.0.0.1:38423_solr ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
[junit4] 2> 89807 WARN (jetty-launcher-207-thread-3) [n:127.0.0.1:38431_solr ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
[junit4] 2> 89807 WARN (jetty-launcher-207-thread-2) [n:127.0.0.1:41543_solr ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
[junit4] 2> 89808 WARN (jetty-launcher-207-thread-1) [n:127.0.0.1:40441_solr ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
[junit4] 2> 89814 INFO (jetty-launcher-207-thread-3) [n:127.0.0.1:38431_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 89814 INFO (jetty-launcher-207-thread-4) [n:127.0.0.1:38423_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 89814 INFO (jetty-launcher-207-thread-1) [n:127.0.0.1:40441_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 89814 INFO (jetty-launcher-207-thread-2) [n:127.0.0.1:41543_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 89814 INFO (jetty-launcher-207-thread-3) [n:127.0.0.1:38431_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38431_solr
[junit4] 2> 89815 INFO (jetty-launcher-207-thread-4) [n:127.0.0.1:38423_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38423_solr
[junit4] 2> 89815 INFO (jetty-launcher-207-thread-2) [n:127.0.0.1:41543_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41543_solr
[junit4] 2> 89815 INFO (jetty-launcher-207-thread-1) [n:127.0.0.1:40441_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40441_solr
[junit4] 2> 89815 INFO (jetty-launcher-207-thread-3) [n:127.0.0.1:38431_solr ] o.a.s.c.Overseer Overseer (id=98725440257851403-127.0.0.1:38431_solr-n_0000000000) starting
[junit4] 2> 89822 INFO (zkCallback-229-thread-1-processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 89822 INFO (zkCallback-227-thread-1-processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 89834 INFO (zkCallback-226-thread-1-processing-n:127.0.0.1:38423_solr) [n:127.0.0.1:38423_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 89835 INFO (zkCallback-228-thread-1-processing-n:127.0.0.1:40441_solr) [n:127.0.0.1:40441_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 89861 INFO (jetty-launcher-207-thread-3) [n:127.0.0.1:38431_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38431_solr
[junit4] 2> 89861 INFO (zkCallback-229-thread-1-processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 89861 INFO (zkCallback-228-thread-1-processing-n:127.0.0.1:40441_solr) [n:127.0.0.1:40441_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 89861 INFO (zkCallback-227-thread-1-processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 89863 INFO (zkCallback-226-thread-1-processing-n:127.0.0.1:38423_solr) [n:127.0.0.1:38423_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 89924 INFO (jetty-launcher-207-thread-2) [n:127.0.0.1:41543_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41543.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
[junit4] 2> 89935 INFO (jetty-launcher-207-thread-2) [n:127.0.0.1:41543_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41543.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
[junit4] 2> 89935 INFO (jetty-launcher-207-thread-2) [n:127.0.0.1:41543_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41543.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
[junit4] 2> 89936 INFO (jetty-launcher-207-thread-2) [n:127.0.0.1:41543_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_1A8F22F34F79AFDB-001/tempDir-001/node2/.
[junit4] 2> 89939 INFO (jetty-launcher-207-thread-1) [n:127.0.0.1:40441_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40441.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
[junit4] 2> 89947 INFO (jetty-launcher-207-thread-1) [n:127.0.0.1:40441_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40441.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
[junit4] 2> 89947 INFO (jetty-launcher-207-thread-1) [n:127.0.0.1:40441_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_40441.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
[junit4] 2> 89957 INFO (jetty-launcher-207-thread-1) [n:127.0.0.1:40441_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_1A8F22F34F79AFDB-001/tempDir-001/node1/.
[junit4] 2> 90056 INFO (jetty-launcher-207-thread-3) [n:127.0.0.1:38431_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38431.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
[junit4] 2> 90072 INFO (jetty-launcher-207-thread-3) [n:127.0.0.1:38431_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38431.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
[junit4] 2> 90072 INFO (jetty-launcher-207-thread-3) [n:127.0.0.1:38431_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38431.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
[junit4] 2> 90075 INFO (jetty-launcher-207-thread-3) [n:127.0.0.1:38431_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_1A8F22F34F79AFDB-001/tempDir-001/node3/.
[junit4] 2> 90077 INFO (jetty-launcher-207-thread-4) [n:127.0.0.1:38423_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38423.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
[junit4] 2> 90082 INFO (jetty-launcher-207-thread-4) [n:127.0.0.1:38423_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38423.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
[junit4] 2> 90082 INFO (jetty-launcher-207-thread-4) [n:127.0.0.1:38423_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38423.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
[junit4] 2> 90083 INFO (jetty-launcher-207-thread-4) [n:127.0.0.1:38423_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_1A8F22F34F79AFDB-001/tempDir-001/node4/.
[junit4] 2> 90169 INFO (SUITE-StreamExpressionTest-seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 90170 INFO (SUITE-StreamExpressionTest-seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40651/solr ready
[junit4] 2> 90226 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 90228 INFO (OverseerThreadFactory-458-thread-1-processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ] o.a.s.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 90335 INFO (OverseerStateUpdate-98725440257851403-127.0.0.1:38431_solr-n_0000000000) [n:127.0.0.1:38431_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:41543/solr",
[junit4] 2> "type":"NRT"}
[junit4] 2> 90338 INFO (OverseerStateUpdate-98725440257851403-127.0.0.1:38431_solr-n_0000000000) [n:127.0.0.1:38431_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"collection1_shard2_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:38431/solr",
[junit4] 2> "type":"NRT"}
[junit4] 2> 90558 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=collection1_shard1_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 90559 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 90618 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=collection1_shard2_replica_n2&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 90618 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 90722 INFO (zkCallback-229-thread-1-processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 90722 INFO (zkCallback-227-thread-1-processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 90722 INFO (zkCallback-229-thread-2-processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 90723 INFO (zkCallback-227-thread-2-processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 91569 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 91577 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema [collection1_shard1_replica_n1] Schema name=test
[junit4] 2> 91614 WARN (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4] 2> 91614 WARN (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (3)
[junit4] 2> 91616 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 91616 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n1' using configuration from collection collection1, trusted=true
[junit4] 2> 91617 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41543.solr.core.collection1.shard1.replica_n1' (registry 'solr.core.collection1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
[junit4] 2> 91617 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 91617 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [[collection1_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_1A8F22F34F79AFDB-001/tempDir-001/node2/collection1_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_1A8F22F34F79AFDB-001/tempDir-001/node2/./collection1_shard1_replica_n1/data/]
[junit4] 2> 91625 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 91640 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.s.IndexSchema [collection1_shard2_replica_n2] Schema name=test
[junit4] 2> 91675 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 91675 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 91675 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 91676 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 91677 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@14407c3[collection1_shard1_replica_n1] main]
[junit4] 2> 91680 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 91680 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 91681 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 91682 INFO (searcherExecutor-467-thread-1-processing-n:127.0.0.1:41543_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@14407c3[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 91682 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579607046377963520
[junit4] 2> 91685 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 91685 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 91685 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/
[junit4] 2> 91685 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 91685 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SyncStrategy https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/ has no replicas
[junit4] 2> 91685 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 91687 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/ shard1
[junit4] 2> 91701 WARN (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.s.IndexSchema Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4] 2> 91701 WARN (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (3)
[junit4] 2> 91701 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 91701 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n2' using configuration from collection collection1, trusted=true
[junit4] 2> 91702 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38431.solr.core.collection1.shard2.replica_n2' (registry 'solr.core.collection1.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@17fa502
[junit4] 2> 91702 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 91702 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.SolrCore [[collection1_shard2_replica_n2] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_1A8F22F34F79AFDB-001/tempDir-001/node3/collection1_shard2_replica_n2], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_1A8F22F34F79AFDB-001/tempDir-001/node3/./collection1_shard2_replica_n2/data/]
[junit4] 2> 91733 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 91733 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 91734 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 91734 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 91735 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@16df2e3[collection1_shard2_replica_n2] main]
[junit4] 2> 91736 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 91736 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 91737 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 91737 INFO (searcherExecutor-472-thread-1-processing-n:127.0.0.1:38431_solr x:collection1_shard2_replica_n2 s:shard2 c:collection1 r:core_node4) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.SolrCore [collection1_shard2_replica_n2] Registered new searcher Searcher@16df2e3[collection1_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 91737 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579607046435635200
[junit4] 2> 91741 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 91741 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 91741 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:38431/solr/collection1_shard2_replica_n2/
[junit4] 2> 91741 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 91741 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.SyncStrategy https://127.0.0.1:38431/solr/collection1_shard2_replica_n2/ has no replicas
[junit4] 2> 91741 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 91742 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:38431/solr/collection1_shard2_replica_n2/ shard2
[junit4] 2> 91877 INFO (zkCallback-229-thread-2-processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 91877 INFO (zkCallback-227-thread-2-processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 91877 INFO (zkCallback-229-thread-1-processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 91877 INFO (zkCallback-227-thread-1-processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 91926 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 91926 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 91928 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=collection1_shard2_replica_n2&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1310
[junit4] 2> 91928 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=collection1_shard1_replica_n1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1369
[junit4] 2> 91933 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 92034 INFO (zkCallback-227-thread-1-processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 92034 INFO (zkCallback-229-thread-2-processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 92034 INFO (zkCallback-227-thread-2-processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 92034 INFO (zkCallback-229-thread-1-processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 92230 INFO (OverseerCollectionConfigSetProcessor-98725440257851403-127.0.0.1:38431_solr-n_0000000000) [n:127.0.0.1:38431_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 92933 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2707
[junit4] 2> 92934 INFO (SUITE-StreamExpressionTest-seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):90
[junit4] 2> 92935 INFO (SUITE-StreamExpressionTest-seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
[junit4] 2> 92969 INFO (TEST-StreamExpressionTest.testCartesianProductStream-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testCartesianProductStream
[junit4] 2> 92990 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/&wt=javabin&version=2}{deleteByQuery=*:* (-1579607047747403776)} 0 1
[junit4] 2> 92996 INFO (qtp17705314-1185) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579607047755792384,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 92997 INFO (qtp17705314-1185) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 92998 INFO (qtp17705314-1185) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 92998 INFO (qtp17705314-1185) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 2
[junit4] 2> 93000 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579607047759986688,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 93000 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 93001 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 93001 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 1
[junit4] 2> 93002 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=javabin&version=2}{deleteByQuery=*:* (-1579607047730626560),commit=} 0 31
[junit4] 2> 93008 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[0 (1579607047766278144), 1 (1579607047768375296)]} 0 3
[junit4] 2> 93011 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579607047771521024,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 93011 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579607047771521024,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 93013 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@1def314 commitCommandVersion:1579607047771521024
[junit4] 2> 93013 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 93013 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 93013 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 2
[junit4] 2> 93033 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@10959d5[collection1_shard1_replica_n1] main]
[junit4] 2> 93034 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 93035 INFO (searcherExecutor-467-thread-1-processing-n:127.0.0.1:41543_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@10959d5[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):c2)))}
[junit4] 2> 93035 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 24
[junit4] 2> 93037 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=javabin&version=2}{commit=} 0 27
[junit4] 2> 93044 INFO (TEST-StreamExpressionTest.testCartesianProductStream-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 93045 INFO (TEST-StreamExpressionTest.testCartesianProductStream-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40651/solr ready
[junit4] 2> 93052 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_ss&sort=id+asc&wt=json&version=2.2} hits=0 status=0 QTime=0
[junit4] 2> 93052 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_ss&sort=id+asc&wt=json&version=2.2} hits=2 status=0 QTime=0
[junit4] 2> 93059 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_ss&sort=id+asc&wt=json&version=2.2} hits=2 status=0 QTime=0
[junit4] 2> 93059 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_ss&sort=id+asc&wt=json&version=2.2} hits=0 status=0 QTime=0
[junit4] 2> 93068 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_ss,b_ls&sort=id+asc&wt=json&version=2.2} hits=0 status=0 QTime=0
[junit4] 2> 93069 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_ss,b_ls&sort=id+asc&wt=json&version=2.2} hits=2 status=0 QTime=0
[junit4] 2> 93073 INFO (qtp17705314-1185) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_ss,b_ls&sort=id+asc&wt=json&version=2.2} hits=0 status=0 QTime=0
[junit4] 2> 93077 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_ss,b_ls&sort=id+asc&wt=json&version=2.2} hits=2 status=0 QTime=0
[junit4] 2> 93081 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_ss,b_ls&sort=id+asc&wt=json&version=2.2} hits=0 status=0 QTime=0
[junit4] 2> 93085 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_ss,b_ls&sort=id+asc&wt=json&version=2.2} hits=2 status=0 QTime=0
[junit4] 2> 93087 INFO (TEST-StreamExpressionTest.testCartesianProductStream-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testCartesianProductStream
[junit4] 2> 93101 INFO (TEST-StreamExpressionTest.testComplementStream-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testComplementStream
[junit4] 2> 93104 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@fa82be[collection1_shard1_replica_n1] realtime]
[junit4] 2> 93105 INFO (qtp17705314-1200) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/&wt=javabin&version=2}{deleteByQuery=*:* (-1579607047866941440)} 0 2
[junit4] 2> 93106 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579607047871135744,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 93106 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579607047871135744,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 93106 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@1def314 commitCommandVersion:1579607047871135744
[junit4] 2> 93106 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 93106 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 93106 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 93107 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1e0e1ff[collection1_shard1_replica_n1] main]
[junit4] 2> 93108 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 93108 INFO (searcherExecutor-467-thread-1-processing-n:127.0.0.1:41543_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@1e0e1ff[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 93109 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 2
[junit4] 2> 93109 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=javabin&version=2}{deleteByQuery=*:* (-1579607047865892864),commit=} 0 8
[junit4] 2> 93112 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[0 (1579607047875330048), 4 (1579607047877427200), 8 (1579607047877427201)]} 0 1
[junit4] 2> 93119 INFO (qtp17705314-1186) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2] webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[2 (1579607047878475776), 3 (1579607047883718656), 5 (1579607047883718657), 6 (1579607047883718658), 9 (1579607047884767232), 7 (1579607047884767233)]} 0 5
[junit4] 2> 93121 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579607047886864384,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 93121 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@1def314 commitCommandVersion:1579607047886864384
[junit4] 2> 93121 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579607047886864384,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 93121 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@f43748 commitCommandVersion:1579607047886864384
[junit4] 2> 93126 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@ccffa4[collection1_shard1_replica_n1] main]
[junit4] 2> 93127 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@179196a[collection1_shard2_replica_n2] main]
[junit4] 2> 93127 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 93127 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 93128 INFO (searcherExecutor-467-thread-1-processing-n:127.0.0.1:41543_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@ccffa4[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.1.0):c3)))}
[junit4] 2> 93128 INFO (searcherExecutor-472-thread-1-processing-n:127.0.0.1:38431_solr x:collection1_shard2_replica_n2 s:shard2 c:collection1 r:core_node4) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.SolrCore [collection1_shard2_replica_n2] Registered new searcher Searcher@179196a[collection1_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):c6)))}
[junit4] 2> 93128 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 7
[junit4] 2> 93128 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 7
[junit4] 2> 93130 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=javabin&version=2}{commit=} 0 10
[junit4] 2> 93135 INFO (TEST-StreamExpressionTest.testComplementStream-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 93136 INFO (TEST-StreamExpressionTest.testComplementStream-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40651/solr ready
[junit4] 2> 93144 INFO (qtp17705314-1186) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr path=/select params={q=a_s:(setA+||+setAB)&distrib=false&fl=id,a_s,a_i&sort=a_i+asc,+a_s+asc&wt=json&version=2.2} hits=3 status=0 QTime=4
[junit4] 2> 93145 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/select params={q=a_s:(setA+||+setAB)&distrib=false&fl=id,a_s,a_i&sort=a_i+asc,+a_s+asc&wt=json&version=2.2} hits=3 status=0 QTime=5
[junit4] 2> 93148 INFO (qtp26942402-1201) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/select params={q=a_s:(setB+||+setAB)&distrib=false&fl=id,a_s,a_i&sort=a_i+asc&wt=json&version=2.2} hits=1 status=0 QTime=0
[junit4] 2> 93148 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr path=/select params={q=a_s:(setB+||+setAB)&distrib=false&fl=id,a_s,a_i&sort=a_i+asc&wt=json&version=2.2} hits=4 status=0 QTime=0
[junit4] 2> 93155 INFO (TEST-StreamExpressionTest.testComplementStream-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testComplementStream
[junit4] 2> 93173 INFO (TEST-StreamExpressionTest.testAnalyzeEvaluator-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testAnalyzeEvaluator
[junit4] 2> 93176 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1bb0216[collection1_shard1_replica_n1] realtime]
[junit4] 2> 93177 INFO (qtp17705314-1200) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@1f8c50b[collection1_shard2_replica_n2] realtime]
[junit4] 2> 93177 INFO (qtp17705314-1200) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/&wt=javabin&version=2}{deleteByQuery=*:* (-1579607047944536064)} 0 1
[junit4] 2> 93179 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579607047947681792,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 93179 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579607047947681792,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 93179 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@1def314 commitCommandVersion:1579607047947681792
[junit4] 2> 93179 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@f43748 commitCommandVersion:1579607047947681792
[junit4] 2> 93179 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1c70ab7[collection1_shard1_replica_n1] main]
[junit4] 2> 93179 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@e20936[collection1_shard2_replica_n2] main]
[junit4] 2> 93179 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 93179 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 93180 INFO (searcherExecutor-467-thread-1-processing-n:127.0.0.1:41543_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@1c70ab7[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 93180 INFO (searcherExecutor-472-thread-1-processing-n:127.0.0.1:38431_solr x:collection1_shard2_replica_n2 s:shard2 c:collection1 r:core_node4) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.SolrCore [collection1_shard2_replica_n2] Registered new searcher Searcher@e20936[collection1_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 93180 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 1
[junit4] 2> 93180 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 1
[junit4] 2> 93181 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=javabin&version=2}{deleteByQuery=*:* (-1579607047942438912),commit=} 0 7
[junit4] 2> 93190 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[1 (1579607047950827520)]} 0 8
[junit4] 2> 93193 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579607047962361856,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 93193 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579607047962361856,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 93193 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@1def314 commitCommandVersion:1579607047962361856
[junit4] 2> 93193 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 93193 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 93193 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 93196 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@550423[collection1_shard1_replica_n1] main]
[junit4] 2> 93196 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 93197 INFO (searcherExecutor-467-thread-1-processing-n:127.0.0.1:41543_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@550423[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_2(7.1.0):c1)))}
[junit4] 2> 93197 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:41543/solr/collection1_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 4
[junit4] 2> 93198 INFO (qtp26942402-1201) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=javabin&version=2}{commit=} 0 7
[junit4] 2> 93202 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/stream params={indent=off&expr=cartesianProduct(search(collection1,+q%3D"*:*",+fl%3D"id,+test_t",+sort%3D"id+desc"),+analyze(test_t,+test_t)+as+test_t)&wt=json&version=2.2} status=0 QTime=0
[junit4] 2> 93205 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 93205 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:40651/solr ready
[junit4] 2> 93207 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,+test_t&sort=id+desc&wt=json&version=2.2} hits=0 status=0 QTime=0
[junit4] 2> 93207 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,+test_t&sort=id+desc&wt=json&version=2.2} hits=1 status=0 QTime=0
[junit4] 2> 93213 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/stream params={indent=off&expr=analyze("hello+world",+test_t)&wt=json&version=2.2} status=0 QTime=0
[junit4] 2> 93216 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/stream params={indent=off&expr=cartesianProduct(search(collection1,+q%3D"*:*",+fl%3D"id,+test_t",+sort%3D"id+desc"),+analyze(test_t)+as+test_t)&wt=json&version=2.2} status=0 QTime=0
[junit4] 2> 93218 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,+test_t&sort=id+desc&wt=json&version=2.2} hits=0 status=0 QTime=0
[junit4] 2> 93218 INFO (qtp26942402-1191) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,+test_t&sort=id+desc&wt=json&version=2.2} hits=1 status=0 QTime=0
[junit4] 2> 93223 INFO (qtp26942402-1213) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/stream params={indent=off&expr=cartesianProduct(search(collection1,+q%3D"*:*",+fl%3D"id",+sort%3D"id+desc"),+analyze(test_t,+test_t)+as+test_t)&wt=json&version=2.2} status=0 QTime=0
[junit4] 2> 93226 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id&sort=id+desc&wt=json&version=2.2} hits=0 status=0 QTime=0
[junit4] 2> 93226 INFO (qtp26942402-1183) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id&sort=id+desc&wt=json&version=2.2} hits=1 status=0 QTime=0
[junit4] 2> 93233 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/stream params={indent=off&expr=select(search(collection1,+q%3D"*:*",+fl%3D"id,+test_t",+sort%3D"id+desc"),+analyze(test_t,+test_t)+as+test1_t)&wt=json&version=2.2} status=0 QTime=0
[junit4] 2> 93237 INFO (qtp26942402-1340) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.S.Request [collection1_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,+test_t&sort=id+desc&wt=json&version=2.2} hits=1 status=0 QTime=0
[junit4] 2> 93237 INFO (qtp17705314-1196) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.c.S.Request [collection1_shard2_replica_n2] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,+test_t&sort=id+desc&wt=json&version=2.2} hits=0 status=0 QTime=0
[junit4] 2> 93239 INFO (TEST-StreamExpressionTest.testAnalyzeEvaluator-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testAnalyzeEvaluator
[junit4] 2> 93256 INFO (TEST-StreamExpressionTest.testCopyOfRange-seed#[1A8F22F34F79AFDB]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testCopyOfRange
[junit4] 2> 93261 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@fc6cb7[collection1_shard1_replica_n1] realtime]
[junit4] 2> 93261 INFO (qtp26942402-1210) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:38431/solr/collection1_shard2_replica_n2/&wt=javabin&version=2}{deleteByQuery=*:* (-1579607048032616448)} 0 1
[junit4] 2> 93266 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579607048038907904,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 93266 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@1def314 commitCommandVersion:1579607048038907904
[junit4] 2> 93266 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1859da1[collection1_shard1_replica_n1] main]
[junit4] 2> 93267 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 93268 INFO (searcherExecutor-467-thread-1-processing-n:127.0.0.1:41543_solr x:collection1_shard1_replica_n1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.c.SolrCore [collection1_shard1_replica_n1] Registered new searcher Searcher@1859da1[collection1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 93269 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579607048042053632,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 93269 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 93269 INFO (qtp26942402-1211) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38431/solr/collection1_shard2_replica_n2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 3
[junit4] 2> 93269 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 93269 INFO (qtp17705314-1187) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38431/solr/collection1_shard2_replica_n2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 93270 INFO (qtp17705314-1182) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2] webapp=/solr path=/update params={_stateVer_=collection1:4&commit=true&wt=javabin&version=2}{deleteByQuery=*:* (-1579607048030519296),commit=} 0 12
[junit4] 2> 93293 INFO (qtp17705314-1186) [n:127.0.0.1:38431_solr c:collection1 s:shard2 r:core_node4 x:collection1_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_n2] webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[id_1 (1579607048044150784), id_3 (1579607048051490816), id_4 (1579607048052539392), id_5 (1579607048052539393), id_6 (1579607048052539394), id_7 (1579607048052539395), id_8 (1579607048052539396), id_9 (1579607048052539397), id_11 (1579607048052539398), id_12 (1579607048053587968), ... (125 adds)]} 0 21
[junit4] 2> 93298 INFO (qtp26942402-1197) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=collection1:4&wt=javabin&version=2}{add=[id_2 (1579607048044150784), id_10 (1579607048051490816), id_13 (1579607048052539392), id_14 (1579607048052539393), id_16 (1579607048052539394), id_17 (1579607048052539395), id_18 (1579607048053587968), id_21 (1579607048053587969), id_22 (1579607048053587970), id_23 (157
[...truncated too long message...]
a with IW:org.apache.solr.update.SolrIndexWriter@e208ff commitCommandVersion:0
[junit4] 2> 194257 INFO (coreCloseExecutor-659-thread-1) [n:127.0.0.1:41543_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=32131367
[junit4] 2> 194258 INFO (coreCloseExecutor-659-thread-2) [n:127.0.0.1:41543_solr c:destination s:shard2 r:core_node4 x:destination_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.destination.shard2.leader, tag=27523539
[junit4] 2> 194266 INFO (coreCloseExecutor-660-thread-1) [n:127.0.0.1:38423_solr c:mainCorpus s:shard1 r:core_node3 x:mainCorpus_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.mainCorpus.shard1.leader, tag=21083153
[junit4] 2> 194266 INFO (coreCloseExecutor-660-thread-2) [n:127.0.0.1:38423_solr c:destination s:shard1 r:core_node3 x:destination_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.destination.shard1.leader, tag=12541154
[junit4] 2> 194280 INFO (jetty-closer-208-thread-3) [ ] o.a.s.c.Overseer Overseer (id=98725440257851403-127.0.0.1:38431_solr-n_0000000000) closing
[junit4] 2> 194280 INFO (OverseerStateUpdate-98725440257851403-127.0.0.1:38431_solr-n_0000000000) [n:127.0.0.1:38431_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:38431_solr
[junit4] 2> 194281 INFO (zkCallback-228-thread-1-processing-n:127.0.0.1:40441_solr) [n:127.0.0.1:40441_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 194281 INFO (zkCallback-227-thread-1-processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 194281 INFO (zkCallback-229-thread-2-processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 194284 INFO (jetty-closer-208-thread-4) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@8a9d8b{/solr,null,UNAVAILABLE}
[junit4] 2> 195784 WARN (zkCallback-229-thread-2-processing-n:127.0.0.1:41543_solr) [n:127.0.0.1:41543_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 195784 WARN (zkCallback-228-thread-2-processing-n:127.0.0.1:40441_solr) [n:127.0.0.1:40441_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 195784 WARN (zkCallback-227-thread-1-processing-n:127.0.0.1:38431_solr) [n:127.0.0.1:38431_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 195785 INFO (jetty-closer-208-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@72f089{/solr,null,UNAVAILABLE}
[junit4] 2> 195785 INFO (jetty-closer-208-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@53b3f5{/solr,null,UNAVAILABLE}
[junit4] 2> 195785 INFO (jetty-closer-208-thread-3) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1602ddd{/solr,null,UNAVAILABLE}
[junit4] 2> 195785 ERROR (SUITE-StreamExpressionTest-seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 195786 INFO (SUITE-StreamExpressionTest-seed#[1A8F22F34F79AFDB]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40651 40651
[junit4] 2> 206582 INFO (Thread-313) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40651 40651
[junit4] 2> 206583 WARN (Thread-313) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 93 /solr/aliases.json
[junit4] 2> 10 /solr/configs/conf
[junit4] 2> 8 /solr/configs/ml
[junit4] 2> 4 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 93 /solr/clusterprops.json
[junit4] 2> 93 /solr/clusterstate.json
[junit4] 2> 38 /solr/collections/destinationCollection/state.json
[junit4] 2> 26 /solr/collections/parallelDestinationCollection1/state.json
[junit4] 2> 22 /solr/collections/workQueue/state.json
[junit4] 2> 20 /solr/collections/destination/state.json
[junit4] 2> 20 /solr/collections/parallelDestinationCollection/state.json
[junit4] 2> 20 /solr/collections/mainCorpus/state.json
[junit4] 2> 10 /solr/collections/collection1/state.json
[junit4] 2> 10 /solr/collections/uknownCollection/state.json
[junit4] 2> 10 /solr/collections/modelCollection/state.json
[junit4] 2> 10 /solr/collections/checkpointCollection/state.json
[junit4] 2> 2 /solr/overseer_elect/election/98725440257851403-127.0.0.1:38431_solr-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 93 /solr/live_nodes
[junit4] 2> 93 /solr/collections
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J0/temp/solr.client.solrj.io.stream.StreamExpressionTest_1A8F22F34F79AFDB-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {name_s=PostingsFormat(name=LuceneFixedGap), expr_s=PostingsFormat(name=Asserting), terms_ss=PostingsFormat(name=Direct), join2_s=PostingsFormat(name=Direct), field_s=FSTOrd50, multiDefault=PostingsFormat(name=Direct), subject=PostingsFormat(name=Direct), a1_s=FSTOrd50, a2_s=PostingsFormat(name=Asserting), level1_s=PostingsFormat(name=Asserting), body_t=PostingsFormat(name=LuceneFixedGap), s_multi=PostingsFormat(name=Asserting), a_ss=PostingsFormat(name=Direct), col_s=FSTOrd50, level2_s=FSTOrd50, a_s=PostingsFormat(name=Asserting), tv_text=FSTOrd50, term_s=PostingsFormat(name=Direct), a_t=PostingsFormat(name=Direct), text_s=PostingsFormat(name=Asserting), id=PostingsFormat(name=LuceneFixedGap), text=PostingsFormat(name=Asserting), featureSet_s=FSTOrd50, checkpoint_ss=PostingsFormat(name=LuceneFixedGap), test_t=PostingsFormat(name=LuceneFixedGap), ident_s=FSTOrd50, whitetok=PostingsFormat(name=LuceneFixedGap), side_s=PostingsFormat(name=LuceneFixedGap)}, docValues:{name_s=DocValuesFormat(name=Lucene70), expr_s=DocValuesFormat(name=Lucene70), iteration_i=DocValuesFormat(name=Lucene70), terms_ss=DocValuesFormat(name=Direct), join2_s=DocValuesFormat(name=Direct), multiDefault=DocValuesFormat(name=Direct), intDefault=DocValuesFormat(name=Asserting), a_f=DocValuesFormat(name=Asserting), falseNegative_i=DocValuesFormat(name=Asserting), a2_s=DocValuesFormat(name=Lucene70), a_i=DocValuesFormat(name=Lucene70), level1_s=DocValuesFormat(name=Lucene70), a_s=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Lucene70), alpha_d=DocValuesFormat(name=Lucene70), i_multi=DocValuesFormat(name=Lucene70), idf_d=DocValuesFormat(name=Asserting), b_ls=DocValuesFormat(name=Asserting), checkpoint_ss=DocValuesFormat(name=Lucene70), ident_s=DocValuesFormat(name=Asserting), order_i=DocValuesFormat(name=Direct), error_d=DocValuesFormat(name=Lucene70), side_s=DocValuesFormat(name=Lucene70), truePositive_i=DocValuesFormat(name=Lucene70), miles_i=DocValuesFormat(name=Asserting), field_s=DocValuesFormat(name=Asserting), price_f=DocValuesFormat(name=Direct), a1_s=DocValuesFormat(name=Asserting), join1_i=DocValuesFormat(name=Lucene70), test_i=DocValuesFormat(name=Direct), join3_i=DocValuesFormat(name=Lucene70), falsePositive_i=DocValuesFormat(name=Asserting), field_i=DocValuesFormat(name=Direct), s_multi=DocValuesFormat(name=Lucene70), a_ss=DocValuesFormat(name=Direct), col_s=DocValuesFormat(name=Asserting), level2_s=DocValuesFormat(name=Asserting), score_f=DocValuesFormat(name=Lucene70), term_s=DocValuesFormat(name=Direct), text_s=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Direct), idfs_ds=DocValuesFormat(name=Direct), weights_ds=DocValuesFormat(name=Lucene70), featureSet_s=DocValuesFormat(name=Asserting), trueNegative_i=DocValuesFormat(name=Lucene70), index_i=DocValuesFormat(name=Asserting), test_dt=DocValuesFormat(name=Lucene70), out_i=DocValuesFormat(name=Asserting), _version_=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=1840, maxMBSortInHeap=6.295194766277021, sim=RandomSimilarity(queryNorm=true): {}, locale=und, timezone=Europe/Bratislava
[junit4] 2> NOTE: Linux 4.10.0-33-generic i386/Oracle Corporation 1.8.0_144 (32-bit)/cpus=8,threads=1,free=47688920,total=141258752
[junit4] 2> NOTE: All tests run in this JVM: [GraphTest, ContentStreamTest, CommonAdminParamsTest, TestLBHttpSolrClient, CollectionAdminRequestRequiredParamsTest, ArcTangentEvaluatorTest, SubtractEvaluatorTest, LargeVolumeBinaryJettyTest, FieldAnalysisResponseTest, HttpSolrClientSSLAuthConPoolTest, TestNamedListCodec, TestRetryUtil, CloudSolrClientMultiConstructorTest, AppendEvaluatorTest, StreamExpressionToExpessionTest, TestV2Request, ConcatOperationTest, TestXMLEscaping, HyperbolicCosineEvaluatorTest, UniformDistributionEvaluatorTest, JDBCStreamTest, AbsoluteValueEvaluatorTest, FloorEvaluatorTest, LargeVolumeEmbeddedTest, SolrExampleXMLTest, TestEmbeddedSolrServer, TestJavaBinCodec, TestHash, TestFastInputStream, AnlysisResponseBaseTest, SolrSchemalessExampleTest, CloudSolrClientBuilderTest, ConcurrentUpdateSolrClientTest, StreamExpressionTest]
[junit4] Completed [145/145 (1!)] on J0 in 117.09s, 113 tests, 1 error <<< FAILURES!
[...truncated 41595 lines...]
[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk1.8.0_144) - Build # 492
- Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/492/
Java: 64bit/jdk1.8.0_144 -XX:+UseCompressedOops -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.TestTlogReplica.testRecovery
Error Message:
Can not find doc 3 in https://127.0.0.1:38399/solr
Stack Trace:
java.lang.AssertionError: Can not find doc 3 in https://127.0.0.1:38399/solr
at __randomizedtesting.SeedInfo.seed([47D3774E87C6C015:86230EE2AA960AB2]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:868)
at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:559)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 11496 lines...]
[junit4] Suite: org.apache.solr.cloud.TestTlogReplica
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_47D3774E87C6C015-001/init-core-data-001
[junit4] 2> 56399 INFO (SUITE-TestTlogReplica-seed#[47D3774E87C6C015]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 56401 INFO (SUITE-TestTlogReplica-seed#[47D3774E87C6C015]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 56401 INFO (SUITE-TestTlogReplica-seed#[47D3774E87C6C015]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 56401 INFO (SUITE-TestTlogReplica-seed#[47D3774E87C6C015]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_47D3774E87C6C015-001/tempDir-001
[junit4] 2> 56401 INFO (SUITE-TestTlogReplica-seed#[47D3774E87C6C015]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 56402 INFO (Thread-147) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 56402 INFO (Thread-147) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 56420 ERROR (Thread-147) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 56502 INFO (SUITE-TestTlogReplica-seed#[47D3774E87C6C015]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:38845
[junit4] 2> 56514 INFO (jetty-launcher-168-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 56518 INFO (jetty-launcher-168-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 56533 INFO (jetty-launcher-168-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3a5d989{/solr,null,AVAILABLE}
[junit4] 2> 56533 INFO (jetty-launcher-168-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@745b50e7{/solr,null,AVAILABLE}
[junit4] 2> 56534 INFO (jetty-launcher-168-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@5401f964{SSL,[ssl, http/1.1]}{127.0.0.1:34031}
[junit4] 2> 56534 INFO (jetty-launcher-168-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@6339165e{SSL,[ssl, http/1.1]}{127.0.0.1:38399}
[junit4] 2> 56534 INFO (jetty-launcher-168-thread-1) [ ] o.e.j.s.Server Started @57945ms
[junit4] 2> 56534 INFO (jetty-launcher-168-thread-2) [ ] o.e.j.s.Server Started @57945ms
[junit4] 2> 56534 INFO (jetty-launcher-168-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38399}
[junit4] 2> 56534 INFO (jetty-launcher-168-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34031}
[junit4] 2> 56534 ERROR (jetty-launcher-168-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 56535 INFO (jetty-launcher-168-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 56535 INFO (jetty-launcher-168-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 56535 INFO (jetty-launcher-168-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 56535 INFO (jetty-launcher-168-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T09:04:21.849Z
[junit4] 2> 56534 ERROR (jetty-launcher-168-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 56536 INFO (jetty-launcher-168-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 56536 INFO (jetty-launcher-168-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 56536 INFO (jetty-launcher-168-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 56536 INFO (jetty-launcher-168-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T09:04:21.850Z
[junit4] 2> 56541 INFO (jetty-launcher-168-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 56541 INFO (jetty-launcher-168-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 56546 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ebd6e14d60002, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 56549 INFO (jetty-launcher-168-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38845/solr
[junit4] 2> 56561 INFO (jetty-launcher-168-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38845/solr
[junit4] 2> 56572 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ebd6e14d60003, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 56684 INFO (jetty-launcher-168-thread-1) [n:127.0.0.1:34031_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 56684 INFO (jetty-launcher-168-thread-2) [n:127.0.0.1:38399_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 56684 INFO (jetty-launcher-168-thread-1) [n:127.0.0.1:34031_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34031_solr
[junit4] 2> 56685 INFO (jetty-launcher-168-thread-2) [n:127.0.0.1:38399_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38399_solr
[junit4] 2> 56685 INFO (jetty-launcher-168-thread-1) [n:127.0.0.1:34031_solr ] o.a.s.c.Overseer Overseer (id=98724522342350854-127.0.0.1:34031_solr-n_0000000000) starting
[junit4] 2> 56686 INFO (zkCallback-179-thread-1-processing-n:127.0.0.1:38399_solr) [n:127.0.0.1:38399_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 56686 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 56691 INFO (jetty-launcher-168-thread-1) [n:127.0.0.1:34031_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34031_solr
[junit4] 2> 56691 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 56691 INFO (zkCallback-179-thread-1-processing-n:127.0.0.1:38399_solr) [n:127.0.0.1:38399_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 56752 INFO (jetty-launcher-168-thread-2) [n:127.0.0.1:38399_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38399.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c6134a8
[junit4] 2> 56757 INFO (jetty-launcher-168-thread-2) [n:127.0.0.1:38399_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38399.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c6134a8
[junit4] 2> 56757 INFO (jetty-launcher-168-thread-2) [n:127.0.0.1:38399_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38399.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c6134a8
[junit4] 2> 56758 INFO (jetty-launcher-168-thread-2) [n:127.0.0.1:38399_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_47D3774E87C6C015-001/tempDir-001/node2/.
[junit4] 2> 56762 INFO (jetty-launcher-168-thread-1) [n:127.0.0.1:34031_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34031.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c6134a8
[junit4] 2> 56771 INFO (jetty-launcher-168-thread-1) [n:127.0.0.1:34031_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34031.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c6134a8
[junit4] 2> 56772 INFO (jetty-launcher-168-thread-1) [n:127.0.0.1:34031_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34031.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c6134a8
[junit4] 2> 56773 INFO (jetty-launcher-168-thread-1) [n:127.0.0.1:34031_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_47D3774E87C6C015-001/tempDir-001/node1/.
[junit4] 2> 56869 INFO (SUITE-TestTlogReplica-seed#[47D3774E87C6C015]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 56870 INFO (SUITE-TestTlogReplica-seed#[47D3774E87C6C015]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38845/solr ready
[junit4] 2> 56871 INFO (SUITE-TestTlogReplica-seed#[47D3774E87C6C015]-worker) [ ] o.a.s.c.TestTlogReplica Using legacyCloud?: false
[junit4] 2> 56889 INFO (qtp375948239-716) [n:127.0.0.1:38399_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterprop with params val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 56891 INFO (qtp375948239-716) [n:127.0.0.1:38399_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 56900 INFO (TEST-TestTlogReplica.testRealTimeGet-seed#[47D3774E87C6C015]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testRealTimeGet
[junit4] 2> 56903 INFO (qtp375948239-706) [n:127.0.0.1:38399_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=2&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_real_time_get&nrtReplicas=2&action=CREATE&numShards=1&tlogReplicas=1&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 56905 INFO (OverseerThreadFactory-184-thread-1-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.CreateCollectionCmd Create collection tlog_replica_test_real_time_get
[junit4] 2> 56905 WARN (OverseerThreadFactory-184-thread-1-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.CreateCollectionCmd Specified number of replicas of 3 on collection tlog_replica_test_real_time_get is higher than the number of Solr instances currently live or live and part of your createNodeSet(2). It's unusual to run two replica of the same slice on the same Solr-instance.
[junit4] 2> 57014 INFO (OverseerStateUpdate-98724522342350854-127.0.0.1:34031_solr-n_0000000000) [n:127.0.0.1:34031_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_real_time_get",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"tlog_replica_test_real_time_get_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:38399/solr",
[junit4] 2> "type":"NRT"}
[junit4] 2> 57017 INFO (OverseerStateUpdate-98724522342350854-127.0.0.1:34031_solr-n_0000000000) [n:127.0.0.1:34031_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_real_time_get",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"tlog_replica_test_real_time_get_shard1_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:34031/solr",
[junit4] 2> "type":"NRT"}
[junit4] 2> 57022 INFO (OverseerStateUpdate-98724522342350854-127.0.0.1:34031_solr-n_0000000000) [n:127.0.0.1:34031_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_real_time_get",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"tlog_replica_test_real_time_get_shard1_replica_t4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:38399/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 57228 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_t4&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 57228 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 57241 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_n1&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 57258 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_n2&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 57259 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 57365 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 57365 INFO (zkCallback-180-thread-2-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 57365 INFO (zkCallback-179-thread-1-processing-n:127.0.0.1:38399_solr) [n:127.0.0.1:38399_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 57365 INFO (zkCallback-179-thread-2-processing-n:127.0.0.1:38399_solr) [n:127.0.0.1:38399_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 58254 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 58258 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 58273 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.s.IndexSchema [tlog_replica_test_real_time_get_shard1_replica_n1] Schema name=minimal
[junit4] 2> 58275 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.s.IndexSchema [tlog_replica_test_real_time_get_shard1_replica_t4] Schema name=minimal
[junit4] 2> 58277 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 58277 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_real_time_get_shard1_replica_t4' using configuration from collection tlog_replica_test_real_time_get, trusted=true
[junit4] 2> 58278 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38399.solr.core.tlog_replica_test_real_time_get.shard1.replica_t4' (registry 'solr.core.tlog_replica_test_real_time_get.shard1.replica_t4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c6134a8
[junit4] 2> 58278 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 58279 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.SolrCore [[tlog_replica_test_real_time_get_shard1_replica_t4] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_47D3774E87C6C015-001/tempDir-001/node2/tlog_replica_test_real_time_get_shard1_replica_t4], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_47D3774E87C6C015-001/tempDir-001/node2/./tlog_replica_test_real_time_get_shard1_replica_t4/data/]
[junit4] 2> 58279 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 58279 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_real_time_get_shard1_replica_n1' using configuration from collection tlog_replica_test_real_time_get, trusted=true
[junit4] 2> 58280 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38399.solr.core.tlog_replica_test_real_time_get.shard1.replica_n1' (registry 'solr.core.tlog_replica_test_real_time_get.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c6134a8
[junit4] 2> 58280 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 58280 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.SolrCore [[tlog_replica_test_real_time_get_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_47D3774E87C6C015-001/tempDir-001/node2/tlog_replica_test_real_time_get_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_47D3774E87C6C015-001/tempDir-001/node2/./tlog_replica_test_real_time_get_shard1_replica_n1/data/]
[junit4] 2> 58290 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 58311 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.s.IndexSchema [tlog_replica_test_real_time_get_shard1_replica_n2] Schema name=minimal
[junit4] 2> 58315 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 58315 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_real_time_get_shard1_replica_n2' using configuration from collection tlog_replica_test_real_time_get, trusted=true
[junit4] 2> 58317 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34031.solr.core.tlog_replica_test_real_time_get.shard1.replica_n2' (registry 'solr.core.tlog_replica_test_real_time_get.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c6134a8
[junit4] 2> 58317 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 58317 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.SolrCore [[tlog_replica_test_real_time_get_shard1_replica_n2] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_47D3774E87C6C015-001/tempDir-001/node1/tlog_replica_test_real_time_get_shard1_replica_n2], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_47D3774E87C6C015-001/tempDir-001/node1/./tlog_replica_test_real_time_get_shard1_replica_n2/data/]
[junit4] 2> 58349 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 58349 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 58350 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 58350 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 58352 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.s.SolrIndexSearcher Opening [Searcher@33bd124a[tlog_replica_test_real_time_get_shard1_replica_t4] main]
[junit4] 2> 58353 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 58354 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 58354 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 58356 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579592359524433920
[junit4] 2> 58359 INFO (searcherExecutor-189-thread-1-processing-n:127.0.0.1:38399_solr x:tlog_replica_test_real_time_get_shard1_replica_t4 s:shard1 c:tlog_replica_test_real_time_get r:core_node6) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_t4] Registered new searcher Searcher@33bd124a[tlog_replica_test_real_time_get_shard1_replica_t4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 58363 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=3 found=1 timeoutin=9999ms
[junit4] 2> 58370 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 58370 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 58372 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 58373 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 58375 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2d1cf0b6[tlog_replica_test_real_time_get_shard1_replica_n1] main]
[junit4] 2> 58376 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 58376 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 58377 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 58377 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579592359546454016
[junit4] 2> 58380 INFO (searcherExecutor-190-thread-1-processing-n:127.0.0.1:38399_solr x:tlog_replica_test_real_time_get_shard1_replica_n1 s:shard1 c:tlog_replica_test_real_time_get r:core_node3) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_n1] Registered new searcher Searcher@2d1cf0b6[tlog_replica_test_real_time_get_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 58408 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 58408 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 58410 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 58410 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 58411 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@50f44477[tlog_replica_test_real_time_get_shard1_replica_n2] main]
[junit4] 2> 58413 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 58414 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 58415 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 58421 INFO (searcherExecutor-191-thread-1-processing-n:127.0.0.1:34031_solr x:tlog_replica_test_real_time_get_shard1_replica_n2 s:shard1 c:tlog_replica_test_real_time_get r:core_node5) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_n2] Registered new searcher Searcher@50f44477[tlog_replica_test_real_time_get_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 58421 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579592359592591360
[junit4] 2> 58465 INFO (zkCallback-179-thread-2-processing-n:127.0.0.1:38399_solr) [n:127.0.0.1:38399_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 58465 INFO (zkCallback-179-thread-1-processing-n:127.0.0.1:38399_solr) [n:127.0.0.1:38399_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 58465 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 58465 INFO (zkCallback-180-thread-2-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 58863 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 58863 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 58863 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/
[junit4] 2> 58864 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_real_time_get_shard1_replica_t4 url=https://127.0.0.1:38399/solr START replicas=[https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_n1/, https://127.0.0.1:34031/solr/tlog_replica_test_real_time_get_shard1_replica_n2/] nUpdates=100
[junit4] 2> 58879 INFO (qtp375948239-768) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 58880 INFO (qtp1355711058-719) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 59165 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_real_time_get_shard1_replica_t4 url=https://127.0.0.1:38399/solr DONE. We have no versions. sync failed.
[junit4] 2> 59165 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 59165 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 59165 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 59166 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ZkController tlog_replica_test_real_time_get_shard1_replica_t4 stopping background replication from leader
[junit4] 2> 59168 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/ shard1
[junit4] 2> 59270 INFO (zkCallback-179-thread-2-processing-n:127.0.0.1:38399_solr) [n:127.0.0.1:38399_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 59270 INFO (zkCallback-179-thread-1-processing-n:127.0.0.1:38399_solr) [n:127.0.0.1:38399_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 59270 INFO (zkCallback-180-thread-2-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 59270 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 59319 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 59320 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_t4&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2093
[junit4] 2> 59387 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_n1&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2145
[junit4] 2> 59428 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_n2&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2170
[junit4] 2> 59432 INFO (qtp375948239-706) [n:127.0.0.1:38399_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 59530 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 59530 INFO (zkCallback-179-thread-2-processing-n:127.0.0.1:38399_solr) [n:127.0.0.1:38399_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 59530 INFO (zkCallback-180-thread-2-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 59530 INFO (zkCallback-179-thread-1-processing-n:127.0.0.1:38399_solr) [n:127.0.0.1:38399_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 60433 INFO (qtp375948239-706) [n:127.0.0.1:38399_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=2&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_real_time_get&nrtReplicas=2&action=CREATE&numShards=1&tlogReplicas=1&wt=javabin&version=2} status=0 QTime=3530
[junit4] 2> 60522 INFO (qtp375948239-707) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/&wt=javabin&version=2}{add=[0 (1579592361743220736)]} 0 25
[junit4] 2> 60523 INFO (qtp1355711058-711) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/&wt=javabin&version=2}{add=[0 (1579592361743220736)]} 0 37
[junit4] 2> 60524 INFO (qtp375948239-782) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_n1/&wt=javabin&version=2}{add=[0 (1579592361743220736)]} 0 52
[junit4] 2> 60524 INFO (qtp375948239-712) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 72
[junit4] 2> 60535 INFO (qtp1355711058-719) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={qt=/get&_stateVer_=tlog_replica_test_real_time_get:5&ids=0&wt=javabin&version=2} status=0 QTime=3
[junit4] 2> 60538 INFO (qtp375948239-709) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&ids=0&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 60540 INFO (qtp1355711058-711) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={qt=/get&ids=0&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 60544 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/|https://127.0.0.1:34031/solr/tlog_replica_test_real_time_get_shard1_replica_n2/|https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_n1/
[junit4] 2> 60549 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1506416665856&ids=0&isShard=true&shard.url=https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/|https://127.0.0.1:34031/solr/tlog_replica_test_real_time_get_shard1_replica_n2/|https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_n1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 60550 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/get params={qt=/get&ids=0&wt=javabin&version=2} status=0 QTime=7
[junit4] 2> 60559 INFO (qtp375948239-768) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/&wt=javabin&version=2}{add=[1 (1579592361830252544)]} 0 1
[junit4] 2> 60560 INFO (qtp1355711058-718) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/&wt=javabin&version=2}{add=[1 (1579592361830252544)]} 0 2
[junit4] 2> 60561 INFO (qtp375948239-716) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:34031/solr/tlog_replica_test_real_time_get_shard1_replica_n2/&wt=javabin&version=2}{add=[1 (1579592361830252544)]} 0 5
[junit4] 2> 60562 INFO (qtp1355711058-711) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 10
[junit4] 2> 60564 INFO (qtp375948239-706) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&_stateVer_=tlog_replica_test_real_time_get:5&ids=1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 60566 INFO (qtp375948239-707) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&ids=1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 60568 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={qt=/get&ids=1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 60570 INFO (qtp375948239-782) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_n1/|https://127.0.0.1:34031/solr/tlog_replica_test_real_time_get_shard1_replica_n2/|https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/
[junit4] 2> 60572 INFO (qtp375948239-712) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1506416665883&ids=1&isShard=true&shard.url=https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_n1/|https://127.0.0.1:34031/solr/tlog_replica_test_real_time_get_shard1_replica_n2/|https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 60573 INFO (qtp375948239-782) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/get params={qt=/get&ids=1&wt=javabin&version=2} status=0 QTime=3
[junit4] 2> 60578 INFO (qtp375948239-709) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/&wt=javabin&version=2}{add=[2 (1579592361850175488)]} 0 1
[junit4] 2> 60579 INFO (qtp1355711058-713) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/&wt=javabin&version=2}{add=[2 (1579592361850175488)]} 0 2
[junit4] 2> 60580 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/update params={wt=javabin&version=2}{add=[2 (1579592361850175488)]} 0 5
[junit4] 2> 60582 INFO (qtp1355711058-719) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={qt=/get&_stateVer_=tlog_replica_test_real_time_get:5&ids=2&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 60584 INFO (qtp375948239-716) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&ids=2&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 60586 INFO (qtp1355711058-711) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={qt=/get&ids=2&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 60588 INFO (qtp375948239-707) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_n1/|https://127.0.0.1:34031/solr/tlog_replica_test_real_time_get_shard1_replica_n2/|https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/
[junit4] 2> 60590 INFO (qtp375948239-768) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1506416665902&ids=2&isShard=true&shard.url=https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_n1/|https://127.0.0.1:34031/solr/tlog_replica_test_real_time_get_shard1_replica_n2/|https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 60591 INFO (qtp375948239-707) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/get params={qt=/get&ids=2&wt=javabin&version=2} status=0 QTime=3
[junit4] 2> 60593 INFO (qtp375948239-712) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node3 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&ids=0&ids=1&ids=2&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 60596 INFO (qtp1355711058-721) [n:127.0.0.1:34031_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n2] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n2] webapp=/solr path=/get params={qt=/get&ids=0&ids=1&ids=2&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 60598 INFO (qtp375948239-709) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/|https://127.0.0.1:34031/solr/tlog_replica_test_real_time_get_shard1_replica_n2/|https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_n1/
[junit4] 2> 60600 INFO (qtp375948239-768) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1506416665911&ids=0,1,2&isShard=true&shard.url=https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_t4/|https://127.0.0.1:34031/solr/tlog_replica_test_real_time_get_shard1_replica_n2/|https://127.0.0.1:38399/solr/tlog_replica_test_real_time_get_shard1_replica_n1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 60600 INFO (qtp375948239-709) [n:127.0.0.1:38399_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/get params={qt=/get&ids=0&ids=1&ids=2&wt=javabin&version=2} status=0 QTime=3
[junit4] 2> 60601 INFO (TEST-TestTlogReplica.testRealTimeGet-seed#[47D3774E87C6C015]) [ ] o.a.s.c.TestTlogReplica tearDown deleting collection
[junit4] 2> 60604 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=tlog_replica_test_real_time_get&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 60609 INFO (OverseerCollectionConfigSetProcessor-98724522342350854-127.0.0.1:34031_solr-n_0000000000) [n:127.0.0.1:34031_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 60612 INFO (OverseerThreadFactory-184-thread-2-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
[junit4] 2> 60615 INFO (qtp375948239-768) [n:127.0.0.1:38399_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_real_time_get.shard1.replica_n1, tag=null
[junit4] 2> 60615 INFO (qtp1355711058-705) [n:127.0.0.1:34031_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_real_time_get.shard1.replica_n2, tag=null
[junit4] 2> 60615 INFO (qtp1355711058-705) [n:127.0.0.1:34031_solr ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6e11e7b7: rootName = solr_34031, domain = solr.core.tlog_replica_test_real_time_get.shard1.replica_n2, service url = null, agent id = null] for registry solr.core.tlog_replica_test_real_time_get.shard1.replica_n2 / com.codahale.metrics.MetricRegistry@7f72985d
[junit4] 2> 60615 INFO (qtp375948239-768) [n:127.0.0.1:38399_solr ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@43a1f9f7: rootName = solr_38399, domain = solr.core.tlog_replica_test_real_time_get.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.tlog_replica_test_real_time_get.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@34ac0359
[junit4] 2> 60636 INFO (qtp1355711058-705) [n:127.0.0.1:34031_solr ] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_n2] CLOSING SolrCore org.apache.solr.core.SolrCore@551fe8e8
[junit4] 2> 60636 INFO (qtp1355711058-705) [n:127.0.0.1:34031_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_real_time_get.shard1.replica_n2, tag=1428154600
[junit4] 2> 60636 INFO (qtp1355711058-705) [n:127.0.0.1:34031_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.tlog_replica_test_real_time_get.shard1.leader, tag=1428154600
[junit4] 2> 60636 INFO (qtp1355711058-705) [n:127.0.0.1:34031_solr ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
[junit4] 2> 60636 INFO (qtp1355711058-705) [n:127.0.0.1:34031_solr ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@4e47e2a9 commitCommandVersion:0
[junit4] 2> 60640 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_real_time_get.shard1.replica_t4, tag=null
[junit4] 2> 60641 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@78437fe6: rootName = solr_38399, domain = solr.core.tlog_replica_test_real_time_get.shard1.replica_t4, service url = null, agent id = null] for registry solr.core.tlog_replica_test_real_time_get.shard1.replica_t4 / com.codahale.metrics.MetricRegistry@f7e00
[junit4] 2> 60657 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr ] o.a.s.c.ZkController tlog_replica_test_real_time_get_shard1_replica_t4 stopping background replication from leader
[junit4] 2> 60658 INFO (qtp375948239-768) [n:127.0.0.1:38399_solr ] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_n1] CLOSING SolrCore org.apache.solr.core.SolrCore@3ed19837
[junit4] 2> 60658 INFO (qtp375948239-768) [n:127.0.0.1:38399_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_real_time_get.shard1.replica_n1, tag=1053923383
[junit4] 2> 60658 INFO (qtp375948239-768) [n:127.0.0.1:38399_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.tlog_replica_test_real_time_get.shard1.leader, tag=1053923383
[junit4] 2> 60659 INFO (qtp375948239-768) [n:127.0.0.1:38399_solr ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
[junit4] 2> 60659 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr ] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_t4] CLOSING SolrCore org.apache.solr.core.SolrCore@6b280502
[junit4] 2> 60659 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_real_time_get.shard1.replica_t4, tag=1797784834
[junit4] 2> 60660 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.tlog_replica_test_real_time_get.shard1.leader, tag=1797784834
[junit4] 2> 60659 INFO (qtp375948239-768) [n:127.0.0.1:38399_solr ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@75910962 commitCommandVersion:0
[junit4] 2> 60660 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
[junit4] 2> 60660 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@72314ac commitCommandVersion:0
[junit4] 2> 60690 INFO (qtp375948239-768) [n:127.0.0.1:38399_solr ] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
[junit4] 2> 60692 INFO (qtp375948239-768) [n:127.0.0.1:38399_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=tlog_replica_test_real_time_get_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=77
[junit4] 2> 60693 WARN (zkCallback-179-thread-1-processing-n:127.0.0.1:38399_solr) [n:127.0.0.1:38399_solr ] o.a.s.c.LeaderElector Our node is no longer in line to be leader
[junit4] 2> 60693 WARN (zkCallback-180-thread-2-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.LeaderElector
[junit4] 2> org.apache.solr.common.SolrException: SolrCore not found:tlog_replica_test_real_time_get_shard1_replica_n2 in []
[junit4] 2> at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:297)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
[junit4] 2> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)
[junit4] 2> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:269)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 60697 INFO (qtp1355711058-705) [n:127.0.0.1:34031_solr ] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
[junit4] 2> 60700 INFO (qtp1355711058-705) [n:127.0.0.1:34031_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=tlog_replica_test_real_time_get_shard1_replica_n2&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=85
[junit4] 2> 60702 INFO (qtp375948239-714) [n:127.0.0.1:38399_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=tlog_replica_test_real_time_get_shard1_replica_t4&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=87
[junit4] 2> 61411 INFO (qtp375948239-708) [n:127.0.0.1:38399_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=tlog_replica_test_real_time_get&action=DELETE&wt=javabin&version=2} status=0 QTime=807
[junit4] 2> 61512 INFO (TEST-TestTlogReplica.testRealTimeGet-seed#[47D3774E87C6C015]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testRealTimeGet
[junit4] 2> 61524 INFO (TEST-TestTlogReplica.testCreateDelete-seed#[47D3774E87C6C015]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testCreateDelete {seed=[47D3774E87C6C015:5CC3860271363825]}
[junit4] 2> 61526 INFO (qtp375948239-782) [n:127.0.0.1:38399_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_create_delete&nrtReplicas=0&action=CREATE&numShards=2&tlogReplicas=4&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 61533 INFO (OverseerThreadFactory-184-thread-3-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.CreateCollectionCmd Create collection tlog_replica_test_create_delete
[junit4] 2> 61533 INFO (OverseerCollectionConfigSetProcessor-98724522342350854-127.0.0.1:34031_solr-n_0000000000) [n:127.0.0.1:34031_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 61534 WARN (OverseerThreadFactory-184-thread-3-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.CreateCollectionCmd Specified number of replicas of 4 on collection tlog_replica_test_create_delete is higher than the number of Solr instances currently live or live and part of your createNodeSet(2). It's unusual to run two replica of the same slice on the same Solr-instance.
[junit4] 2> 61740 INFO (OverseerStateUpdate-98724522342350854-127.0.0.1:34031_solr-n_0000000000) [n:127.0.0.1:34031_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_create_delete",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"tlog_replica_test_create_delete_shard1_replica_t1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:38399/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 61743 INFO (OverseerStateUpdate-98724522342350854-127.0.0.1:34031_solr-n_0000000000) [n:127.0.0.1:34031_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_create_delete",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"tlog_replica_test_create_delete_shard1_replica_t2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:34031/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 61745 INFO (OverseerStateUpdate-98724522342350854-127.0.0.1:34031_solr-n_0000000000) [n:127.0.0.1:34031_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_create_delete",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"tlog_replica_test_create_delete_shard1_replica_t4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:38399/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 61746 INFO (OverseerStateUpdate-98724522342350854-127.0.0.1:34031_solr-n_0000000000) [n:127.0.0.1:34031_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_create_delete",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"tlog_replica_test_create_delete_shard1_replica_t6",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:34031/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 61750 INFO (OverseerStateUpdate-98724522342350854-127.0.0.1:34031_solr-n_0000000000) [n:127.0.0.1:34031_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_create_delete",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"tlog_replica_test_create_delete_shard2_replica_t8",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:38399/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 61761 INFO (OverseerStateUpdate-98724522342350854-127.0.0.1:34031_solr-n_0000000000) [n:127.0.0.1:34031_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_create_delete",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"tlog_replica_test_create_delete_shard2_replica_t9",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:34031/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 61768 INFO (OverseerStateUpdate-98724522342350854-127.0.0.1:34031_solr-n_0000000000) [n:127.0.0.1:34031_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_create_delete",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"tlog_replica_test_create_delete_shard2_replica_t11",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:38399/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 61769 INFO (OverseerStateUpdate-98724522342350854-127.0.0.1:34031_solr-n_0000000000) [n:127.0.0.1:34031_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_create_delete",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"tlog_replica_test_create_delete_shard2_replica_t12",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:34031/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 61954 INFO (qtp1355711058-719) [n:127.0.0.1:34031_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=tlog_replica_test_create_delete_shard1_replica_t2&action=CREATE&numShards=2&collection=tlog_replica_test_create_delete&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 61954 INFO (qtp375948239-712) [n:127.0.0.1:38399_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_create_delete_shard1_replica_t1&action=CREATE&numShards=2&collection=tlog_replica_test_create_delete&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 61958 INFO (qtp375948239-707) [n:127.0.0.1:38399_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=tlog_replica_test_create_delete_shard1_replica_t4&action=CREATE&numShards=2&collection=tlog_replica_test_create_delete&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 61967 INFO (qtp375948239-716) [n:127.0.0.1:38399_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node13&collection.configName=conf&newCollection=true&name=tlog_replica_test_create_delete_shard2_replica_t8&action=CREATE&numShards=2&collection=tlog_replica_test_create_delete&shard=shard2&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 61972 INFO (qtp1355711058-718) [n:127.0.0.1:34031_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node10&collection.configName=conf&newCollection=true&name=tlog_replica_test_create_delete_shard1_replica_t6&action=CREATE&numShards=2&collection=tlog_replica_test_create_delete&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 61974 INFO (qtp1355711058-798) [n:127.0.0.1:34031_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node16&collection.configName=conf&newCollection=true&name=tlog_replica_test_create_delete_shard2_replica_t12&action=CREATE&numShards=2&collection=tlog_replica_test_create_delete&shard=shard2&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 61988 INFO (qtp375948239-800) [n:127.0.0.1:38399_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node15&collection.configName=conf&newCollection=true&name=tlog_replica_test_create_delete_shard2_replica_t11&action=CREATE&numShards=2&collection=tlog_replica_test_create_delete&shard=shard2&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 61995 INFO (qtp1355711058-705) [n:127.0.0.1:34031_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node14&collection.configName=conf&newCollection=true&name=tlog_replica_test_create_delete_shard2_replica_t9&action=CREATE&numShards=2&collection=tlog_replica_test_create_delete&shard=shard2&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 62098 INFO (zkCallback-179-thread-1-processing-n:127.0.0.1:38399_solr) [n:127.0.0.1:38399_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_create_delete/state.json] for collection [tlog_replica_test_create_delete] has occurred - updating... (live nodes size: [2])
[junit4] 2> 62098 INFO (zkCallback-179-thread-2-processing-n:127.0.0.1:38399_solr) [n:127.0.0.1:38399_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_create_delete/state.json] for collection [tlog_replica_test_create_delete] has occurred - updating... (live nodes size: [2])
[junit4] 2> 62098 INFO (zkCallback-180-thread-2-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_create_delete/state.json] for collection [tlog_replica_test_create_delete] has occurred - updating... (live nodes size: [2])
[junit4] 2> 62098 INFO (zkCallback-180-thread-1-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_create_delete/state.json] for collection [tlog_replica_test_create_delete] has occurred - updating... (live nodes size: [2])
[junit4] 2> 63017 INFO (qtp375948239-707) [n:127.0.0.1:38399_solr c:tlog_replica_test_create_delete s:shard1 r:core_node7 x:tlog_replica_test_create_delete_shard1_replica_t4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 63017 INFO (qtp1355711058-719) [n:127.0.0.1:34031_solr c:tlog_replica_test_create_delete s:shard1 r:core_node5 x:tlog_replica_test_create_delete_shard1_replica_t2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 63030 INFO (qtp1355711058-719) [n:127.0.0.1:34031_solr c:tlog_replica_test_create_delete s:shard1 r:core_node5 x:tlog_replica_test_create_delete_shard1_replica_t2] o.a.s.s.IndexSchema [tlog_replica_test_create_delete_shard1_replica_t2] Schema name=minimal
[junit4] 2> 63034 INFO (qtp1355711058-719) [n:127.0.0.1:34031_solr c:tlog_replica_test_create_delete s:shard1 r:core_node5 x:tlog_replica_test_create_delete_shard1_replica_t2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 63034 INFO (qtp1355711058-719) [n:127.0.0.1:34031_solr c:tlog_replica_test_create_delete s:shard1 r:core_node5 x:tlog_replica_test_create_delete_shard1_replica_t2] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_create_delete_shard1_replica_t2' using configuration from collection tlog_replica_test_create_delete, trusted=true
[junit4] 2> 63036 INFO (qtp375948239-716) [n:127.0.0.1:38399_solr c:tlog_replica_test_create_delete s:shard2 r:core_node13 x:tlog_replica_test_create_delete_shard2_replica_t8] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 63036 INFO (qtp375948239-707) [n:127.0.0.1:38399_solr c:tlog_replica_test_create_delete s:shard1 r:core_node7 x:tlog_replica_test_create_delete_shard1_replica_t4] o.a.s.s.IndexSchema [tlog_replica_test_create_delete_shard1_replica_t4] Schema name=minimal
[junit4] 2> 63038 INFO (qtp375948239-800) [n:127.0.0.1:38399_solr c:tlog_replica_test_create_delete s:shard2 r:core_node15 x:tlog_replica_test_create_delete_shard2_replica_t11] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 63040 INFO (qtp1355711058-718) [n:127.0.0.1:34031_solr c:tlog_replica_test_create_delete s:shard1 r:core_node10 x:tlog_replica_test_create_delete_shard1_replica_t6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 63041 INFO (qtp1355711058-719) [n:127.0.0.1:34031_solr c:tlog_replica_test_create_delete s:shard1 r:core_node5 x:tlog_replica_test_create_delete_shard1_replica_t2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34031.solr.core.tlog_replica_test_create_delete.shard1.replica_t2' (registry 'solr.core.tlog_replica_test_create_delete.shard1.replica_t2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c6134a8
[junit4] 2> 63041 INFO (qtp1355711058-719) [n:127.0.0.1:34031_solr c:tlog_replica_test_create_delete s:shard1 r:core_node5 x:tlog_replica_test_create_delete_shard1_replica_t2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 63042 INFO (qtp1355711058-719) [n:127.0.0.1:34031_solr c:tlog_replica_test_create_delete s:shard1 r:core_node5 x:tlog_replica_test_create_delete_shard1_replica_t2] o.a.s.c.SolrCore [[tlog_replica_test_create_delete_shard1_replica_t2] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_47D3774E87C6C015-001/tempDir-001/node1/tlog_replica_test_create_delete_shard1_replica_t2], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_47D3774E87C6C015-001/tempDir-001/node1/./tlog_replica_test_create_delete_shard1_replica_t2/data/]
[junit4] 2> 63043 INFO (qtp1355711058-798) [n:127.0.0.1:34031_solr c:tlog_replica_test_create_delete s:shard2 r:core_node16 x:tlog_replica_test_create_delete_shard2_replica_t12] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 63044 INFO (qtp375948239-712) [n:127.0.0.1:38399_solr c:tlog_replica_test_create_delete s:shard1 r:core_node3 x:tlog_replica_test_create_delete_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 63044 INFO (qtp1355711058-705) [n:127.0.0.1:34031_solr c:tlog_replica_test_create_delete s:shard2 r:core_node14 x:tlog_replica_test_create_delete_shard2_replica_t9] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 63046 INFO (qtp375948239-707) [n:127.0.0.1:38399_solr c:tlog_replica_test_create_delete s:shard1 r:core_node7 x:tlog_replica_test_create_delete_shard1_replica_t4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 63046 INFO (qtp375948239-707) [n:127.0.0.1:38399_solr c:tlog_replica_test_create_delete s:shard1 r:core_node7 x:tlog_replica_test_create_delete_shard1_replica_t4] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_create_delete_shard1_replica_t4' using configuration from collection tlog_replica_test_create_delete, trusted=true
[junit4] 2> 63046 INFO (qtp375948239-707) [n:127.0.0.1:38399_solr c:tlog_replica_test_create_delete s:shard1 r:core_node7 x:tlog_replica_test_create_delete_shard1_replica_t4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38399.solr.core.tlog_replica_test_create_delete.shard1.replica_t4' (registry 'solr.core.tlog_replica_test_create_delete.shard1.replica_t4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3c6134a8
[junit4] 2> 63047 INFO (qtp375948239-707) [n:127.0.0.1:38399_solr c:tlog_replica_test_create_delete s:shard1 r:core_node7 x:tlog_replica_test_create_delete_shard1_replica_t4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 63047 INFO (qtp375948239-707) [n:127.0.0.1:38399_solr c:tlog_replica_test_create_delete s:shard1 r:core_node7 x:tlog_replica_test_create_delete_shard1_replica_t4] o.a.s.c.SolrCore [[tlog_replica_test_create_delete_shard1_replica_t4] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_47D3774E87C6C015-001/tempDir-001/node2/tlog_replica_test_create_delete_shard1_replica_t4], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_47D3774E87C6C015-001/tempDir-001/node2/./tlog_replica_test_create_delete_shard1_replica_t4/data/]
[junit4] 2> 63049 INFO (qtp1355711058-718) [n:127.0.0.1:34031_solr c:tlog_replica_test_create_delete s:shard1 r:core_node10 x:tlog_replica_test_create_delete_shard1_replica_t6] o.
[...truncated too long message...]
ger Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 157788 INFO (jetty-closer-169-thread-2) [ ] o.a.s.c.Overseer Overseer (id=98724522342350862-127.0.0.1:34031_solr-n_0000000003) closing
[junit4] 2> 157788 INFO (OverseerStateUpdate-98724522342350862-127.0.0.1:34031_solr-n_0000000003) [n:127.0.0.1:34031_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:34031_solr
[junit4] 2> 157788 WARN (zkCallback-198-thread-1-processing-n:127.0.0.1:34031_solr) [n:127.0.0.1:34031_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 157788 INFO (jetty-closer-169-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@51dd5aff{/solr,null,UNAVAILABLE}
[junit4] 2> 158673 ERROR (qtp850057765-1004) [n:127.0.0.1:34031_solr ] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: Solr is shutting down
[junit4] 2> at org.apache.solr.handler.admin.PrepRecoveryOp.execute(PrepRecoveryOp.java:177)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:384)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:389)
[junit4] 2> at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:174)
[junit4] 2> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:177)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:745)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:726)
[junit4] 2> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:507)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:382)
[junit4] 2> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:326)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:139)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
[junit4] 2> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
[junit4] 2> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
[junit4] 2> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
[junit4] 2> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
[junit4] 2> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)
[junit4] 2> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
[junit4] 2> at org.eclipse.jetty.server.Server.handle(Server.java:534)
[junit4] 2> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
[junit4] 2> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:251)
[junit4] 2> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
[junit4] 2> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
[junit4] 2> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] 2> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2>
[junit4] 2> 158674 INFO (qtp850057765-1004) [n:127.0.0.1:34031_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=127.0.0.1:38399_solr&onlyIfLeaderActive=true&core=tlog_replica_test_recovery_shard1_replica_t2&coreNodeName=core_node3&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=400 QTime=13002
[junit4] 2> 158675 ERROR (SUITE-TestTlogReplica-seed#[47D3774E87C6C015]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 158676 INFO (SUITE-TestTlogReplica-seed#[47D3774E87C6C015]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38845 38845
[junit4] 2> 164490 INFO (Thread-147) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:38845 38845
[junit4] 2> 164491 WARN (Thread-147) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 33 /solr/configs/conf
[junit4] 2> 9 /solr/aliases.json
[junit4] 2> 8 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 28 /solr/collections/tlog_replica_test_out_of_order_db_qwith_in_place_updates/state.json
[junit4] 2> 25 /solr/collections/tlog_replica_test_kill_leader/state.json
[junit4] 2> 25 /solr/collections/tlog_replica_test_recovery/state.json
[junit4] 2> 25 /solr/collections/tlog_replica_test_basic_leader_election/state.json
[junit4] 2> 24 /solr/collections/tlog_replica_test_remove_leader/state.json
[junit4] 2> 23 /solr/collections/tlog_replica_test_add_remove_tlog_replica/state.json
[junit4] 2> 22 /solr/collections/tlog_replica_test_kill_tlog_replica/state.json
[junit4] 2> 19 /solr/collections/tlog_replica_test_create_delete/state.json
[junit4] 2> 15 /solr/collections/tlog_replica_test_only_leader_indexes/state.json
[junit4] 2> 15 /solr/collections/tlog_replica_test_delete_by_id/state.json
[junit4] 2> 13 /solr/collections/tlog_replica_test_add_docs/state.json
[junit4] 2> 13 /solr/collections/tlog_replica_test_real_time_get/state.json
[junit4] 2> 9 /solr/clusterprops.json
[junit4] 2> 9 /solr/clusterstate.json
[junit4] 2> 4 /solr/overseer_elect/election/98724522342350862-127.0.0.1:34031_solr-n_0000000003
[junit4] 2> 3 /solr/collections/tlog_replica_test_recovery/leader_elect/shard1/election/98724522342350862-core_node4-n_0000000000
[junit4] 2> 2 /solr/collections/tlog_replica_test_kill_tlog_replica/leader_elect/shard1/election/98724522342350859-core_node4-n_0000000000
[junit4] 2> 2 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard2/election/98724522342350854-core_node16-n_0000000000
[junit4] 2> 2 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard2/election/98724522342350854-core_node16-n_0000000001
[junit4] 2> 2 /solr/overseer_elect/election/98724522342350854-127.0.0.1:34031_solr-n_0000000000
[junit4] 2> 2 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard1/election/98724522342350853-core_node7-n_0000000001
[junit4] 2> 2 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard1/election/98724522342350854-core_node5-n_0000000000
[junit4] 2> 2 /solr/collections/tlog_replica_test_real_time_get/leader_elect/shard1/election/98724522342350853-core_node6-n_0000000000
[junit4] 2> 2 /solr/collections/tlog_replica_test_create_delete/leader_elect/shard1/election/98724522342350854-core_node9-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 9 /solr/collections
[junit4] 2> 3 /solr/overseer/queue
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/overseer/collection-queue-work
[junit4] 2> 2 /solr/overseer/queue-work
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestTlogReplica_47D3774E87C6C015-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {foo=BlockTreeOrds(blocksize=128), title_s=BlockTreeOrds(blocksize=128), foo_s=BlockTreeOrds(blocksize=128), id=PostingsFormat(name=Direct)}, docValues:{_version_=DocValuesFormat(name=Direct), id=DocValuesFormat(name=Lucene70), inplace_updatable_int=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=1258, maxMBSortInHeap=5.663316749774738, sim=RandomSimilarity(queryNorm=false): {}, locale=es, timezone=America/Regina
[junit4] 2> NOTE: Linux 4.10.0-33-generic amd64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=8,threads=1,free=237059800,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [BasicZkTest, TestSolrConfigHandlerConcurrent, TestFieldCollectionResource, DefaultValueUpdateProcessorTest, TestImpersonationWithHadoopAuth, V2StandaloneTest, TestTlogReplica]
[junit4] Completed [30/732 (1!)] on J0 in 108.22s, 13 tests, 1 failure <<< FAILURES!
[...truncated 49028 lines...]
[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-9) - Build # 491 - Still
Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/491/
Java: 64bit/jdk-9 -XX:-UseCompressedOops -XX:+UseG1GC --illegal-access=deny
1 tests failed.
FAILED: org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testParallelExecutorStream
Error Message:
Error from server at https://127.0.0.1:35689/solr/workQueue_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 404 </title> </head> <body> <h2>HTTP ERROR: 404</h2> <p>Problem accessing /solr/workQueue_shard2_replica_n3/update. Reason: <pre> Can not find: /solr/workQueue_shard2_replica_n3/update</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/> </body> </html>
Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://127.0.0.1:35689/solr/workQueue_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/workQueue_shard2_replica_n3/update. Reason:
<pre> Can not find: /solr/workQueue_shard2_replica_n3/update</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>
at __randomizedtesting.SeedInfo.seed([5D8C555080DAB489:E09B2049B9F689D4]:0)
at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:539)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:993)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
at org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testParallelExecutorStream(StreamExpressionTest.java:7333)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:35689/solr/workQueue_shard2_replica_n3: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 404 </title>
</head>
<body>
<h2>HTTP ERROR: 404</h2>
<p>Problem accessing /solr/workQueue_shard2_replica_n3/update. Reason:
<pre> Can not find: /solr/workQueue_shard2_replica_n3/update</pre></p>
<hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/>
</body>
</html>
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:591)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$directUpdate$0(CloudSolrClient.java:516)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
... 1 more
Build Log:
[...truncated 14754 lines...]
[junit4] Suite: org.apache.solr.client.solrj.io.stream.StreamExpressionTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_5D8C555080DAB489-001/init-core-data-001
[junit4] 2> 21842 INFO (SUITE-StreamExpressionTest-seed#[5D8C555080DAB489]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 21843 INFO (SUITE-StreamExpressionTest-seed#[5D8C555080DAB489]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0)
[junit4] 2> 21844 INFO (SUITE-StreamExpressionTest-seed#[5D8C555080DAB489]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 21845 INFO (SUITE-StreamExpressionTest-seed#[5D8C555080DAB489]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_5D8C555080DAB489-001/tempDir-001
[junit4] 2> 21848 INFO (SUITE-StreamExpressionTest-seed#[5D8C555080DAB489]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 21849 INFO (Thread-383) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 21849 INFO (Thread-383) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 21927 ERROR (Thread-383) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 21949 INFO (SUITE-StreamExpressionTest-seed#[5D8C555080DAB489]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:44751
[junit4] 2> 21995 INFO (jetty-launcher-97-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 21999 INFO (jetty-launcher-97-thread-3) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 21999 INFO (jetty-launcher-97-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 22000 INFO (jetty-launcher-97-thread-4) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 22006 INFO (jetty-launcher-97-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4f41f70e{/solr,null,AVAILABLE}
[junit4] 2> 22013 INFO (jetty-launcher-97-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@7fb457e{SSL,[ssl, http/1.1]}{127.0.0.1:35689}
[junit4] 2> 22013 INFO (jetty-launcher-97-thread-2) [ ] o.e.j.s.Server Started @24015ms
[junit4] 2> 22013 INFO (jetty-launcher-97-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35689}
[junit4] 2> 22013 ERROR (jetty-launcher-97-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 22013 INFO (jetty-launcher-97-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 7.1.0
[junit4] 2> 22013 INFO (jetty-launcher-97-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 22013 INFO (jetty-launcher-97-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 22013 INFO (jetty-launcher-97-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T06:47:11.186907Z
[junit4] 2> 22014 INFO (jetty-launcher-97-thread-2) [ ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
[junit4] 2> 22021 INFO (jetty-launcher-97-thread-3) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1ea1b7da{/solr,null,AVAILABLE}
[junit4] 2> 22023 INFO (jetty-launcher-97-thread-3) [ ] o.e.j.s.AbstractConnector Started ServerConnector@58fd8ccc{SSL,[ssl, http/1.1]}{127.0.0.1:38439}
[junit4] 2> 22025 INFO (jetty-launcher-97-thread-3) [ ] o.e.j.s.Server Started @24027ms
[junit4] 2> 22025 INFO (jetty-launcher-97-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38439}
[junit4] 2> 22025 ERROR (jetty-launcher-97-thread-3) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 22025 INFO (jetty-launcher-97-thread-3) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 7.1.0
[junit4] 2> 22025 INFO (jetty-launcher-97-thread-3) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 22026 INFO (jetty-launcher-97-thread-3) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 22026 INFO (jetty-launcher-97-thread-3) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T06:47:11.199029Z
[junit4] 2> 22029 INFO (jetty-launcher-97-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@588b7e99{/solr,null,AVAILABLE}
[junit4] 2> 22029 INFO (jetty-launcher-97-thread-4) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@46654fc9{/solr,null,AVAILABLE}
[junit4] 2> 22033 INFO (jetty-launcher-97-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@52d0fbfe{SSL,[ssl, http/1.1]}{127.0.0.1:36083}
[junit4] 2> 22033 INFO (jetty-launcher-97-thread-1) [ ] o.e.j.s.Server Started @24035ms
[junit4] 2> 22033 INFO (jetty-launcher-97-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36083}
[junit4] 2> 22033 INFO (jetty-launcher-97-thread-4) [ ] o.e.j.s.AbstractConnector Started ServerConnector@6ccc7b86{SSL,[ssl, http/1.1]}{127.0.0.1:38491}
[junit4] 2> 22033 INFO (jetty-launcher-97-thread-4) [ ] o.e.j.s.Server Started @24035ms
[junit4] 2> 22033 INFO (jetty-launcher-97-thread-4) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38491}
[junit4] 2> 22034 ERROR (jetty-launcher-97-thread-4) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 22034 ERROR (jetty-launcher-97-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 22034 INFO (jetty-launcher-97-thread-4) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 7.1.0
[junit4] 2> 22034 INFO (jetty-launcher-97-thread-4) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 22034 INFO (jetty-launcher-97-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solrâ„¢ version 7.1.0
[junit4] 2> 22034 INFO (jetty-launcher-97-thread-4) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 22034 INFO (jetty-launcher-97-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 22034 INFO (jetty-launcher-97-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 22034 INFO (jetty-launcher-97-thread-4) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T06:47:11.207321Z
[junit4] 2> 22034 INFO (jetty-launcher-97-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-26T06:47:11.207354Z
[junit4] 2> 22038 INFO (jetty-launcher-97-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 22041 INFO (jetty-launcher-97-thread-4) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 22041 INFO (jetty-launcher-97-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 22041 INFO (jetty-launcher-97-thread-3) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 22058 INFO (jetty-launcher-97-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44751/solr
[junit4] 2> 22065 INFO (jetty-launcher-97-thread-3) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44751/solr
[junit4] 2> 22069 INFO (jetty-launcher-97-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44751/solr
[junit4] 2> 22069 INFO (jetty-launcher-97-thread-4) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:44751/solr
[junit4] 2> 22116 WARN (jetty-launcher-97-thread-2) [ ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
[junit4] 2> 22116 WARN (jetty-launcher-97-thread-1) [ ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
[junit4] 2> 22116 WARN (jetty-launcher-97-thread-4) [ ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
[junit4] 2> 22116 WARN (jetty-launcher-97-thread-3) [ ] o.a.s.c.ZkController The _default configset could not be uploaded. Please provide 'solr.default.confdir' parameter that points to a configset intended to be the default. Current 'solr.default.confdir' value: null
[junit4] 2> 22130 INFO (jetty-launcher-97-thread-1) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 22130 INFO (jetty-launcher-97-thread-3) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 22131 INFO (jetty-launcher-97-thread-2) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 22131 INFO (jetty-launcher-97-thread-4) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 22132 INFO (jetty-launcher-97-thread-1) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36083_solr
[junit4] 2> 22133 INFO (jetty-launcher-97-thread-3) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38439_solr
[junit4] 2> 22134 INFO (jetty-launcher-97-thread-1) [ ] o.a.s.c.Overseer Overseer (id=98723982936768523-127.0.0.1:36083_solr-n_0000000000) starting
[junit4] 2> 22136 INFO (zkCallback-118-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 22136 INFO (zkCallback-116-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 22136 INFO (zkCallback-117-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 22137 INFO (zkCallback-119-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 22133 INFO (jetty-launcher-97-thread-4) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38491_solr
[junit4] 2> 22133 INFO (jetty-launcher-97-thread-2) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35689_solr
[junit4] 2> 22139 INFO (zkCallback-119-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
[junit4] 2> 22139 INFO (zkCallback-117-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
[junit4] 2> 22140 INFO (zkCallback-118-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
[junit4] 2> 22140 INFO (zkCallback-116-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (3)
[junit4] 2> 22187 INFO (jetty-launcher-97-thread-1) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36083_solr
[junit4] 2> 22188 INFO (zkCallback-119-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 22188 INFO (zkCallback-117-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 22188 INFO (zkCallback-116-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 22188 INFO (zkCallback-118-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 22257 INFO (jetty-launcher-97-thread-1) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36083.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e176a83
[junit4] 2> 22260 INFO (jetty-launcher-97-thread-4) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38491.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e176a83
[junit4] 2> 22261 INFO (jetty-launcher-97-thread-2) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35689.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e176a83
[junit4] 2> 22272 INFO (jetty-launcher-97-thread-3) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38439.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e176a83
[junit4] 2> 22284 INFO (jetty-launcher-97-thread-1) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36083.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e176a83
[junit4] 2> 22284 INFO (jetty-launcher-97-thread-1) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36083.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e176a83
[junit4] 2> 22285 INFO (jetty-launcher-97-thread-4) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38491.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e176a83
[junit4] 2> 22285 INFO (jetty-launcher-97-thread-4) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38491.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e176a83
[junit4] 2> 22286 INFO (jetty-launcher-97-thread-2) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35689.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e176a83
[junit4] 2> 22286 INFO (jetty-launcher-97-thread-2) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35689.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e176a83
[junit4] 2> 22287 INFO (jetty-launcher-97-thread-1) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_5D8C555080DAB489-001/tempDir-001/node1/.
[junit4] 2> 22288 INFO (jetty-launcher-97-thread-2) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_5D8C555080DAB489-001/tempDir-001/node2/.
[junit4] 2> 22290 INFO (jetty-launcher-97-thread-3) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38439.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e176a83
[junit4] 2> 22290 INFO (jetty-launcher-97-thread-3) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38439.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e176a83
[junit4] 2> 22291 INFO (jetty-launcher-97-thread-3) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_5D8C555080DAB489-001/tempDir-001/node3/.
[junit4] 2> 22292 INFO (jetty-launcher-97-thread-4) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_5D8C555080DAB489-001/tempDir-001/node4/.
[junit4] 2> 22522 INFO (SUITE-StreamExpressionTest-seed#[5D8C555080DAB489]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 22526 INFO (SUITE-StreamExpressionTest-seed#[5D8C555080DAB489]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44751/solr ready
[junit4] 2> 22572 INFO (qtp1884836168-1141) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf&name=collection1_collection&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 22591 INFO (OverseerThreadFactory-738-thread-1) [ ] o.a.s.c.CreateCollectionCmd Create collection collection1_collection
[junit4] 2> 22719 INFO (OverseerStateUpdate-98723982936768523-127.0.0.1:36083_solr-n_0000000000) [ ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1_collection",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1_collection_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:38439/solr",
[junit4] 2> "type":"NRT"}
[junit4] 2> 22725 INFO (OverseerStateUpdate-98723982936768523-127.0.0.1:36083_solr-n_0000000000) [ ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1_collection",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"collection1_collection_shard2_replica_n3",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:38491/solr",
[junit4] 2> "type":"NRT"}
[junit4] 2> 22957 INFO (qtp1884836168-1138) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=collection1_collection_shard1_replica_n1&action=CREATE&numShards=2&collection=collection1_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 22958 INFO (qtp1884836168-1138) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 23009 INFO (qtp950758985-1149) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=collection1_collection_shard2_replica_n3&action=CREATE&numShards=2&collection=collection1_collection&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 23009 INFO (qtp950758985-1149) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 23135 INFO (zkCallback-116-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1_collection/state.json] for collection [collection1_collection] has occurred - updating... (live nodes size: [4])
[junit4] 2> 23135 INFO (zkCallback-119-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1_collection/state.json] for collection [collection1_collection] has occurred - updating... (live nodes size: [4])
[junit4] 2> 23143 INFO (zkCallback-119-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1_collection/state.json] for collection [collection1_collection] has occurred - updating... (live nodes size: [4])
[junit4] 2> 23145 INFO (zkCallback-116-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1_collection/state.json] for collection [collection1_collection] has occurred - updating... (live nodes size: [4])
[junit4] 2> 23999 INFO (qtp1884836168-1138) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 24007 INFO (qtp1884836168-1138) [ ] o.a.s.s.IndexSchema [collection1_collection_shard1_replica_n1] Schema name=test
[junit4] 2> 24031 INFO (qtp950758985-1149) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 24041 INFO (qtp950758985-1149) [ ] o.a.s.s.IndexSchema [collection1_collection_shard2_replica_n3] Schema name=test
[junit4] 2> 24068 WARN (qtp1884836168-1138) [ ] o.a.s.s.IndexSchema Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4] 2> 24068 WARN (qtp1884836168-1138) [ ] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (3)
[junit4] 2> 24068 INFO (qtp1884836168-1138) [ ] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 24068 INFO (qtp1884836168-1138) [ ] o.a.s.c.CoreContainer Creating SolrCore 'collection1_collection_shard1_replica_n1' using configuration from collection collection1_collection, trusted=true
[junit4] 2> 24069 INFO (qtp1884836168-1138) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38439.solr.core.collection1_collection.shard1.replica_n1' (registry 'solr.core.collection1_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e176a83
[junit4] 2> 24070 INFO (qtp1884836168-1138) [ ] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 24070 INFO (qtp1884836168-1138) [ ] o.a.s.c.SolrCore [[collection1_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_5D8C555080DAB489-001/tempDir-001/node3/collection1_collection_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_5D8C555080DAB489-001/tempDir-001/node3/./collection1_collection_shard1_replica_n1/data/]
[junit4] 2> 24104 WARN (qtp950758985-1149) [ ] o.a.s.s.IndexSchema Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
[junit4] 2> 24104 WARN (qtp950758985-1149) [ ] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (3)
[junit4] 2> 24104 INFO (qtp950758985-1149) [ ] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 24104 INFO (qtp950758985-1149) [ ] o.a.s.c.CoreContainer Creating SolrCore 'collection1_collection_shard2_replica_n3' using configuration from collection collection1_collection, trusted=true
[junit4] 2> 24105 INFO (qtp950758985-1149) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38491.solr.core.collection1_collection.shard2.replica_n3' (registry 'solr.core.collection1_collection.shard2.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3e176a83
[junit4] 2> 24105 INFO (qtp950758985-1149) [ ] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 24105 INFO (qtp950758985-1149) [ ] o.a.s.c.SolrCore [[collection1_collection_shard2_replica_n3] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_5D8C555080DAB489-001/tempDir-001/node4/collection1_collection_shard2_replica_n3], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_5D8C555080DAB489-001/tempDir-001/node4/./collection1_collection_shard2_replica_n3/data/]
[junit4] 2> 24133 INFO (qtp1884836168-1138) [ ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 24133 INFO (qtp1884836168-1138) [ ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 24135 INFO (qtp1884836168-1138) [ ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 24135 INFO (qtp1884836168-1138) [ ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 24136 INFO (qtp1884836168-1138) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@81f6e04[collection1_collection_shard1_replica_n1] main]
[junit4] 2> 24138 INFO (qtp1884836168-1138) [ ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 24139 INFO (qtp1884836168-1138) [ ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 24139 INFO (qtp1884836168-1138) [ ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 24141 INFO (qtp1884836168-1138) [ ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579583729370660864
[junit4] 2> 24145 INFO (searcherExecutor-747-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard1_replica_n1] Registered new searcher Searcher@81f6e04[collection1_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 24153 INFO (qtp1884836168-1138) [ ] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 24153 INFO (qtp1884836168-1138) [ ] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 24153 INFO (qtp1884836168-1138) [ ] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:38439/solr/collection1_collection_shard1_replica_n1/
[junit4] 2> 24153 INFO (qtp1884836168-1138) [ ] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 24153 INFO (qtp1884836168-1138) [ ] o.a.s.c.SyncStrategy https://127.0.0.1:38439/solr/collection1_collection_shard1_replica_n1/ has no replicas
[junit4] 2> 24153 INFO (qtp1884836168-1138) [ ] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 24159 INFO (qtp950758985-1149) [ ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 24159 INFO (qtp950758985-1149) [ ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 24161 INFO (qtp950758985-1149) [ ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 24161 INFO (qtp950758985-1149) [ ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 24163 INFO (qtp950758985-1149) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@39e3a0de[collection1_collection_shard2_replica_n3] main]
[junit4] 2> 24165 INFO (qtp950758985-1149) [ ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 24166 INFO (qtp1884836168-1138) [ ] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:38439/solr/collection1_collection_shard1_replica_n1/ shard1
[junit4] 2> 24166 INFO (qtp950758985-1149) [ ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 24167 INFO (qtp950758985-1149) [ ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 24170 INFO (searcherExecutor-748-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard2_replica_n3] Registered new searcher Searcher@39e3a0de[collection1_collection_shard2_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 24170 INFO (qtp950758985-1149) [ ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1579583729401069568
[junit4] 2> 24175 INFO (qtp950758985-1149) [ ] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 24175 INFO (qtp950758985-1149) [ ] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 24175 INFO (qtp950758985-1149) [ ] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/
[junit4] 2> 24175 INFO (qtp950758985-1149) [ ] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 24175 INFO (qtp950758985-1149) [ ] o.a.s.c.SyncStrategy https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/ has no replicas
[junit4] 2> 24175 INFO (qtp950758985-1149) [ ] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 24178 INFO (qtp950758985-1149) [ ] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/ shard2
[junit4] 2> 24281 INFO (zkCallback-119-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1_collection/state.json] for collection [collection1_collection] has occurred - updating... (live nodes size: [4])
[junit4] 2> 24281 INFO (zkCallback-116-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1_collection/state.json] for collection [collection1_collection] has occurred - updating... (live nodes size: [4])
[junit4] 2> 24281 INFO (zkCallback-116-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1_collection/state.json] for collection [collection1_collection] has occurred - updating... (live nodes size: [4])
[junit4] 2> 24281 INFO (zkCallback-119-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1_collection/state.json] for collection [collection1_collection] has occurred - updating... (live nodes size: [4])
[junit4] 2> 24317 INFO (qtp1884836168-1138) [ ] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 24319 INFO (qtp1884836168-1138) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=collection1_collection_shard1_replica_n1&action=CREATE&numShards=2&collection=collection1_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1362
[junit4] 2> 24329 INFO (qtp950758985-1149) [ ] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 24330 INFO (qtp950758985-1149) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=collection1_collection_shard2_replica_n3&action=CREATE&numShards=2&collection=collection1_collection&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1321
[junit4] 2> 24342 INFO (qtp1884836168-1141) [ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 24432 INFO (zkCallback-116-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1_collection/state.json] for collection [collection1_collection] has occurred - updating... (live nodes size: [4])
[junit4] 2> 24432 INFO (zkCallback-119-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1_collection/state.json] for collection [collection1_collection] has occurred - updating... (live nodes size: [4])
[junit4] 2> 24432 INFO (zkCallback-116-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1_collection/state.json] for collection [collection1_collection] has occurred - updating... (live nodes size: [4])
[junit4] 2> 24432 INFO (zkCallback-119-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1_collection/state.json] for collection [collection1_collection] has occurred - updating... (live nodes size: [4])
[junit4] 2> 24592 INFO (OverseerCollectionConfigSetProcessor-98723982936768523-127.0.0.1:36083_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 25343 INFO (qtp1884836168-1141) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf&name=collection1_collection&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2770
[junit4] 2> 25346 INFO (SUITE-StreamExpressionTest-seed#[5D8C555080DAB489]-worker) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1_collection failOnTimeout:true timeout (sec):90
[junit4] 2> 25347 INFO (SUITE-StreamExpressionTest-seed#[5D8C555080DAB489]-worker) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1_collection
[junit4] 2> 25348 INFO (qtp1884836168-1144) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :createalias with params collections=collection1_collection&name=collection1&action=CREATEALIAS&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 25452 INFO (qtp1884836168-1144) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collections=collection1_collection&name=collection1&action=CREATEALIAS&wt=javabin&version=2} status=0 QTime=104
[junit4] 2> 25465 INFO (TEST-StreamExpressionTest.testLetStream-seed#[5D8C555080DAB489]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testLetStream
[junit4] 2> 25481 INFO (qtp950758985-1149) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:38439/solr/collection1_collection_shard1_replica_n1/&wt=javabin&version=2}{deleteByQuery=*:* (-1579583730774704128)} 0 1
[junit4] 2> 25490 INFO (qtp950758985-1148) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583730785189888,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25490 INFO (qtp950758985-1148) [ ] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 25491 INFO (qtp950758985-1148) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25491 INFO (qtp950758985-1148) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38439/solr/collection1_collection_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 25491 INFO (qtp1884836168-1144) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583730786238464,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25491 INFO (qtp1884836168-1144) [ ] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 25491 INFO (qtp1884836168-1144) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25491 INFO (qtp1884836168-1144) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38439/solr/collection1_collection_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 25492 INFO (qtp1884836168-1127) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&commit=true&wt=javabin&version=2}{deleteByQuery=*:* (-1579583730768412672),commit=} 0 24
[junit4] 2> 25503 INFO (qtp950758985-1148) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&wt=javabin&version=2}{add=[hello (1579583730793578496), hello1 (1579583730798821376)]} 0 5
[junit4] 2> 25519 INFO (qtp1884836168-1127) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583730815598592,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25525 INFO (qtp1884836168-1127) [ ] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 25525 INFO (qtp1884836168-1127) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25525 INFO (qtp1884836168-1127) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 6
[junit4] 2> 25529 INFO (qtp950758985-1149) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583730826084352,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25530 INFO (qtp950758985-1149) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@15d68926 commitCommandVersion:1579583730826084352
[junit4] 2> 25538 INFO (qtp950758985-1149) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@7378f61e[collection1_collection_shard2_replica_n3] main]
[junit4] 2> 25539 INFO (qtp950758985-1149) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25541 INFO (searcherExecutor-748-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard2_replica_n3] Registered new searcher Searcher@7378f61e[collection1_collection_shard2_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):c2)))}
[junit4] 2> 25541 INFO (qtp950758985-1149) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 12
[junit4] 2> 25543 INFO (qtp950758985-1143) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&commit=true&wt=javabin&version=2}{commit=} 0 37
[junit4] 2> 25556 INFO (qtp1884836168-1130) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/stream params={indent=off&expr=let(d+%3Dsearch(collection1,+q%3D"*:*",+fl%3D"id,test_t,+test_i",+sort%3D"id+desc"),+b+%3D+add(1,3),+c%3Dcol(d,+test_i),+tuple(test+%3D+add(1,1),+test1%3Db,+results%3Dd,+test2%3Dadd(c)))&wt=json&version=2.2} status=0 QTime=4
[junit4] 2> 25568 INFO (qtp1884836168-1130) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 25569 INFO (qtp1884836168-1130) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44751/solr ready
[junit4] 2> 25575 INFO (qtp1884836168-1144) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,test_t,+test_i&sort=id+desc&wt=json&version=2.2} hits=0 status=0 QTime=0
[junit4] 2> 25576 INFO (qtp950758985-1149) [ ] o.a.s.c.S.Request [collection1_collection_shard2_replica_n3] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,test_t,+test_i&sort=id+desc&wt=json&version=2.2} hits=2 status=0 QTime=1
[junit4] 2> 25581 INFO (TEST-StreamExpressionTest.testLetStream-seed#[5D8C555080DAB489]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testLetStream
[junit4] 2> 25608 INFO (TEST-StreamExpressionTest.testCloudSolrStreamWithZkHost-seed#[5D8C555080DAB489]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testCloudSolrStreamWithZkHost
[junit4] 2> 25612 INFO (qtp950758985-1156) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@390f2604[collection1_collection_shard2_replica_n3] realtime]
[junit4] 2> 25612 INFO (qtp1884836168-1141) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&wt=javabin&version=2}{deleteByQuery=*:* (-1579583730912067584)} 0 2
[junit4] 2> 25617 INFO (qtp1884836168-1130) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583730918359040,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25617 INFO (qtp1884836168-1130) [ ] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 25617 INFO (qtp950758985-1145) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583730918359040,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25617 INFO (qtp950758985-1145) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@15d68926 commitCommandVersion:1579583730918359040
[junit4] 2> 25617 INFO (qtp1884836168-1130) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25618 INFO (qtp1884836168-1130) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 25620 INFO (qtp950758985-1145) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@31e4cf7b[collection1_collection_shard2_replica_n3] main]
[junit4] 2> 25620 INFO (qtp950758985-1145) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25622 INFO (searcherExecutor-748-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard2_replica_n3] Registered new searcher Searcher@31e4cf7b[collection1_collection_shard2_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 25622 INFO (qtp950758985-1145) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 5
[junit4] 2> 25623 INFO (qtp950758985-1156) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&commit=true&wt=javabin&version=2}{deleteByQuery=*:* (-1579583730909970432),commit=} 0 14
[junit4] 2> 25626 INFO (qtp950758985-1148) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&wt=javabin&version=2}{add=[2 (1579583730926747648), 3 (1579583730927796224)]} 0 1
[junit4] 2> 25630 INFO (qtp1884836168-1138) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&wt=javabin&version=2}{add=[0 (1579583730929893376), 4 (1579583730930941952), 1 (1579583730931990528)]} 0 1
[junit4] 2> 25632 INFO (qtp950758985-1149) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583730934087680,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25632 INFO (qtp950758985-1149) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@15d68926 commitCommandVersion:1579583730934087680
[junit4] 2> 25633 INFO (qtp1884836168-1126) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583730935136256,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25633 INFO (qtp1884836168-1126) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@467dfac4 commitCommandVersion:1579583730935136256
[junit4] 2> 25638 INFO (qtp950758985-1149) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@1412a855[collection1_collection_shard2_replica_n3] main]
[junit4] 2> 25638 INFO (qtp950758985-1149) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25640 INFO (searcherExecutor-748-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard2_replica_n3] Registered new searcher Searcher@1412a855[collection1_collection_shard2_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.1.0):c2)))}
[junit4] 2> 25640 INFO (qtp950758985-1149) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 7
[junit4] 2> 25641 INFO (qtp1884836168-1126) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@44d7cd3d[collection1_collection_shard1_replica_n1] main]
[junit4] 2> 25642 INFO (qtp1884836168-1126) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25643 INFO (searcherExecutor-747-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard1_replica_n1] Registered new searcher Searcher@44d7cd3d[collection1_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.1.0):c3)))}
[junit4] 2> 25644 INFO (qtp1884836168-1126) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 10
[junit4] 2> 25645 INFO (qtp950758985-1143) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&commit=true&wt=javabin&version=2}{commit=} 0 14
[junit4] 2> 25653 INFO (TEST-StreamExpressionTest.testCloudSolrStreamWithZkHost-seed#[5D8C555080DAB489]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 25654 INFO (TEST-StreamExpressionTest.testCloudSolrStreamWithZkHost-seed#[5D8C555080DAB489]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44751/solr ready
[junit4] 2> 25665 INFO (qtp950758985-1149) [ ] o.a.s.c.S.Request [collection1_collection_shard2_replica_n3] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_s,a_i,a_f&sort=a_f+asc,+a_i+asc&wt=json&version=2.2} hits=2 status=0 QTime=1
[junit4] 2> 25666 INFO (qtp1884836168-1130) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_s,a_i,a_f&sort=a_f+asc,+a_i+asc&wt=json&version=2.2} hits=3 status=0 QTime=0
[junit4] 2> 25673 INFO (qtp950758985-1143) [ ] o.a.s.c.S.Request [collection1_collection_shard2_replica_n3] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_s,a_i,a_f&sort=a_f+asc,+a_i+asc&wt=json&version=2.2} hits=2 status=0 QTime=0
[junit4] 2> 25675 INFO (qtp1884836168-1138) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_s,a_i,a_f&sort=a_f+asc,+a_i+asc&wt=json&version=2.2} hits=3 status=0 QTime=0
[junit4] 2> 25693 INFO (qtp950758985-1148) [ ] o.a.s.c.S.Request [collection1_collection_shard2_replica_n3] webapp=/solr path=/select params={q=id:(0+3+4)&distrib=false&fl=id,a_s,a_i,a_f&sort=a_f+asc,+a_i+asc&wt=json&version=2.2} hits=1 status=0 QTime=0
[junit4] 2> 25693 INFO (qtp1884836168-1127) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/select params={q=id:(0+3+4)&distrib=false&fl=id,a_s,a_i,a_f&sort=a_f+asc,+a_i+asc&wt=json&version=2.2} hits=2 status=0 QTime=0
[junit4] 2> 25708 INFO (qtp950758985-1149) [ ] o.a.s.c.S.Request [collection1_collection_shard2_replica_n3] webapp=/solr path=/select params={q=id:(*)&distrib=false&fl=id,a_s,a_i,a_f&fq=a_s:hello0&fq=a_s:hello1&sort=a_f+asc,+a_i+asc&wt=json&version=2.2} hits=0 status=0 QTime=9
[junit4] 2> 25708 INFO (qtp1884836168-1126) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/select params={q=id:(*)&distrib=false&fl=id,a_s,a_i,a_f&fq=a_s:hello0&fq=a_s:hello1&sort=a_f+asc,+a_i+asc&wt=json&version=2.2} hits=0 status=0 QTime=9
[junit4] 2> 25718 INFO (qtp1884836168-1130) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/select params={q=id:(*)&distrib=false&fl=id,a_s,a_i,a_f&fq=a_s:(hello0+OR+hello1)&sort=a_f+asc,+a_i+asc&wt=json&version=2.2} hits=2 status=0 QTime=1
[junit4] 2> 25718 INFO (qtp950758985-1143) [ ] o.a.s.c.S.Request [collection1_collection_shard2_replica_n3] webapp=/solr path=/select params={q=id:(*)&distrib=false&fl=id,a_s,a_i,a_f&fq=a_s:(hello0+OR+hello1)&sort=a_f+asc,+a_i+asc&wt=json&version=2.2} hits=0 status=0 QTime=0
[junit4] 2> 25721 INFO (TEST-StreamExpressionTest.testCloudSolrStreamWithZkHost-seed#[5D8C555080DAB489]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testCloudSolrStreamWithZkHost
[junit4] 2> 25721 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ebcf07dc20010, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> 25742 INFO (TEST-StreamExpressionTest.testParallelNullStream-seed#[5D8C555080DAB489]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testParallelNullStream
[junit4] 2> 25746 INFO (qtp950758985-1145) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@36c4093c[collection1_collection_shard2_replica_n3] realtime]
[junit4] 2> 25746 INFO (qtp1884836168-1141) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@183de6d4[collection1_collection_shard1_replica_n1] realtime]
[junit4] 2> 25747 INFO (qtp1884836168-1141) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&wt=javabin&version=2}{deleteByQuery=*:* (-1579583731052576768)} 0 2
[junit4] 2> 25748 INFO (qtp1884836168-1127) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583731055722496,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25748 INFO (qtp950758985-1156) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583731055722496,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25749 INFO (qtp1884836168-1127) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@467dfac4 commitCommandVersion:1579583731055722496
[junit4] 2> 25749 INFO (qtp950758985-1156) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@15d68926 commitCommandVersion:1579583731055722496
[junit4] 2> 25749 INFO (qtp1884836168-1127) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@5d35fb67[collection1_collection_shard1_replica_n1] main]
[junit4] 2> 25749 INFO (qtp950758985-1156) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@6cacefd9[collection1_collection_shard2_replica_n3] main]
[junit4] 2> 25749 INFO (qtp1884836168-1127) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25749 INFO (qtp950758985-1156) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25750 INFO (searcherExecutor-748-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard2_replica_n3] Registered new searcher Searcher@6cacefd9[collection1_collection_shard2_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 25751 INFO (qtp950758985-1156) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 2
[junit4] 2> 25751 INFO (searcherExecutor-747-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard1_replica_n1] Registered new searcher Searcher@5d35fb67[collection1_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 25751 INFO (qtp1884836168-1127) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 3
[junit4] 2> 25752 INFO (qtp950758985-1145) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&commit=true&wt=javabin&version=2}{deleteByQuery=*:* (-1579583731050479616),commit=} 0 9
[junit4] 2> 25755 INFO (qtp1884836168-1144) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&wt=javabin&version=2}{add=[0 (1579583731060965376), 4 (1579583731063062528), 1 (1579583731063062529)]} 0 2
[junit4] 2> 25756 INFO (qtp950758985-1149) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&wt=javabin&version=2}{add=[2 (1579583731062013952), 3 (1579583731063062528), 5 (1579583731063062529)]} 0 2
[junit4] 2> 25758 INFO (qtp950758985-1139) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583731066208256,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25759 INFO (qtp950758985-1139) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@15d68926 commitCommandVersion:1579583731066208256
[junit4] 2> 25759 INFO (qtp1884836168-1128) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583731067256832,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25759 INFO (qtp1884836168-1128) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@467dfac4 commitCommandVersion:1579583731067256832
[junit4] 2> 25762 INFO (qtp950758985-1139) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@8f46c74[collection1_collection_shard2_replica_n3] main]
[junit4] 2> 25762 INFO (qtp1884836168-1128) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@21e2f32c[collection1_collection_shard1_replica_n1] main]
[junit4] 2> 25763 INFO (qtp950758985-1139) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25763 INFO (qtp1884836168-1128) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25765 INFO (searcherExecutor-747-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard1_replica_n1] Registered new searcher Searcher@21e2f32c[collection1_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_1(7.1.0):c3)))}
[junit4] 2> 25765 INFO (searcherExecutor-748-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard2_replica_n3] Registered new searcher Searcher@8f46c74[collection1_collection_shard2_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_2(7.1.0):c3)))}
[junit4] 2> 25765 INFO (qtp1884836168-1128) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38439/solr/collection1_collection_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 6
[junit4] 2> 25765 INFO (qtp950758985-1139) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38439/solr/collection1_collection_shard1_replica_n1/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 6
[junit4] 2> 25767 INFO (qtp1884836168-1126) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&commit=true&wt=javabin&version=2}{commit=} 0 10
[junit4] 2> 25775 INFO (TEST-StreamExpressionTest.testParallelNullStream-seed#[5D8C555080DAB489]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 25776 INFO (TEST-StreamExpressionTest.testParallelNullStream-seed#[5D8C555080DAB489]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:44751/solr ready
[junit4] 2> 25780 INFO (qtp950758985-1149) [ ] o.a.s.c.S.Request [collection1_collection_shard2_replica_n3] webapp=/solr path=/stream params={distrib=false&workerID=0&indent=off&numWorkers=2&expr=null(search(collection1,q%3D"*:*",fl%3D"id,a_s,a_i,a_f",sort%3D"a_f+asc",partitionKeys%3Did,zkHost%3D"127.0.0.1:44751/solr"))&wt=json&version=2.2} status=0 QTime=0
[junit4] 2> 25780 INFO (qtp1884836168-1144) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/stream params={distrib=false&workerID=1&indent=off&numWorkers=2&expr=null(search(collection1,q%3D"*:*",fl%3D"id,a_s,a_i,a_f",sort%3D"a_f+asc",partitionKeys%3Did,zkHost%3D"127.0.0.1:44751/solr"))&wt=json&version=2.2} status=0 QTime=0
[junit4] 2> 25791 INFO (qtp1884836168-1138) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_s,a_i,a_f&sort=a_f+asc&partitionKeys=id&fq={!hash+workers%3D2+worker%3D0}&wt=json&version=2.2} hits=2 status=0 QTime=1
[junit4] 2> 25791 INFO (qtp950758985-1145) [ ] o.a.s.c.S.Request [collection1_collection_shard2_replica_n3] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_s,a_i,a_f&sort=a_f+asc&partitionKeys=id&fq={!hash+workers%3D2+worker%3D0}&wt=json&version=2.2} hits=1 status=0 QTime=2
[junit4] 2> 25791 INFO (qtp1884836168-1141) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_s,a_i,a_f&sort=a_f+asc&partitionKeys=id&fq={!hash+workers%3D2+worker%3D1}&wt=json&version=2.2} hits=1 status=0 QTime=2
[junit4] 2> 25799 INFO (qtp950758985-1143) [ ] o.a.s.c.S.Request [collection1_collection_shard2_replica_n3] webapp=/solr path=/select params={q=*:*&distrib=false&fl=id,a_s,a_i,a_f&sort=a_f+asc&partitionKeys=id&fq={!hash+workers%3D2+worker%3D1}&wt=json&version=2.2} hits=2 status=0 QTime=0
[junit4] 2> 25802 INFO (TEST-StreamExpressionTest.testParallelNullStream-seed#[5D8C555080DAB489]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testParallelNullStream
[junit4] 2> 25815 INFO (TEST-StreamExpressionTest.testRankTransform-seed#[5D8C555080DAB489]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testRankTransform
[junit4] 2> 25818 INFO (qtp950758985-1148) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@a2bf2f6[collection1_collection_shard2_replica_n3] realtime]
[junit4] 2> 25818 INFO (qtp1884836168-1130) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@28a1d951[collection1_collection_shard1_replica_n1] realtime]
[junit4] 2> 25818 INFO (qtp1884836168-1130) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&wt=javabin&version=2}{deleteByQuery=*:* (-1579583731129122816)} 0 1
[junit4] 2> 25820 INFO (qtp950758985-1280) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583731131219968,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25820 INFO (qtp1884836168-1144) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583731131219968,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25820 INFO (qtp950758985-1280) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@15d68926 commitCommandVersion:1579583731131219968
[junit4] 2> 25820 INFO (qtp1884836168-1144) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@467dfac4 commitCommandVersion:1579583731131219968
[junit4] 2> 25821 INFO (qtp1884836168-1144) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@665d85a8[collection1_collection_shard1_replica_n1] main]
[junit4] 2> 25821 INFO (qtp950758985-1280) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@7542e240[collection1_collection_shard2_replica_n3] main]
[junit4] 2> 25821 INFO (qtp950758985-1280) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25821 INFO (qtp1884836168-1144) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25823 INFO (searcherExecutor-748-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard2_replica_n3] Registered new searcher Searcher@7542e240[collection1_collection_shard2_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 25823 INFO (searcherExecutor-747-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard1_replica_n1] Registered new searcher Searcher@665d85a8[collection1_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 25823 INFO (qtp950758985-1280) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 3
[junit4] 2> 25823 INFO (qtp1884836168-1144) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 3
[junit4] 2> 25824 INFO (qtp950758985-1148) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&commit=true&wt=javabin&version=2}{deleteByQuery=*:* (-1579583731128074240),commit=} 0 7
[junit4] 2> 25848 INFO (qtp950758985-1145) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&wt=javabin&version=2}{add=[id_1 (1579583731138560000), id_3 (1579583731139608576), id_4 (1579583731139608577), id_5 (1579583731140657152), id_6 (1579583731140657153), id_7 (1579583731140657154), id_8 (1579583731140657155), id_9 (1579583731140657156), id_11 (1579583731140657157), id_12 (1579583731141705728), ... (125 adds)]} 0 21
[junit4] 2> 25856 INFO (qtp1884836168-1128) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&wt=javabin&version=2}{add=[id_2 (1579583731138560000), id_10 (1579583731139608576), id_13 (1579583731139608577), id_14 (1579583731140657152), id_16 (1579583731140657153), id_17 (1579583731140657154), id_18 (1579583731140657155), id_21 (1579583731141705728), id_22 (1579583731141705729), id_23 (1579583731141705730), ... (125 adds)]} 0 28
[junit4] 2> 25858 INFO (qtp1884836168-1281) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583731171065856,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25858 INFO (qtp950758985-1279) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583731171065856,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25858 INFO (qtp1884836168-1281) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@467dfac4 commitCommandVersion:1579583731171065856
[junit4] 2> 25858 INFO (qtp950758985-1279) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@15d68926 commitCommandVersion:1579583731171065856
[junit4] 2> 25863 INFO (qtp950758985-1279) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@3eb02e4[collection1_collection_shard2_replica_n3] main]
[junit4] 2> 25864 INFO (qtp1884836168-1281) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@67590296[collection1_collection_shard1_replica_n1] main]
[junit4] 2> 25864 INFO (qtp950758985-1279) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25864 INFO (qtp1884836168-1281) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25865 INFO (searcherExecutor-748-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard2_replica_n3] Registered new searcher Searcher@3eb02e4[collection1_collection_shard2_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_3(7.1.0):c125)))}
[junit4] 2> 25865 INFO (qtp950758985-1279) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 7
[junit4] 2> 25865 INFO (searcherExecutor-747-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard1_replica_n1] Registered new searcher Searcher@67590296[collection1_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_2(7.1.0):c125)))}
[junit4] 2> 25866 INFO (qtp1884836168-1281) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 7
[junit4] 2> 25867 INFO (qtp950758985-1139) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&commit=true&wt=javabin&version=2}{commit=} 0 9
[junit4] 2> 25871 INFO (qtp1884836168-1128) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/stream params={indent=off&expr=let(a%3Dtimeseries(collection1,+q%3D"*:*",+start%3D"2013-01-01T01:00:00.000Z",+end%3D"2016-12-01T01:00:00.000Z",+gap%3D"%2B1YEAR",+field%3D"test_dt",+count(*),+sum(price_f),+max(price_f),+min(price_f)),+c%3Dcol(a,+max(price_f)),+tuple(reverse%3Drev(c),+ranked%3Drank(c)))&wt=json&version=2.2} status=0 QTime=1
[junit4] 2> 25893 INFO (qtp950758985-1279) [ ] o.a.s.c.S.Request [collection1_collection_shard2_replica_n3] webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1_collection:4&_facet_={}&fl=id&fl=score&shards.purpose=1048580&start=0&fsv=true&collection=collection1_collection&shard.url=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&rows=0&version=2&q=*:*&json.facet={"timeseries":{"type":"range","field":"test_dt","start":"2013-01-01T01:00:00.000Z","end":"2016-12-01T01:00:00.000Z","gap":"%2B1YEAR","facet":{"facet_0":"sum(price_f)","facet_1":"max(price_f)","facet_2":"min(price_f)"}}}&field=test_dt&NOW=1506408435047&isShard=true&wt=javabin} hits=125 status=0 QTime=9
[junit4] 2> 25894 INFO (qtp1884836168-1144) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/select params={df=text&distrib=false&_stateVer_=collection1_collection:4&_facet_={}&fl=id&fl=score&shards.purpose=1048580&start=0&fsv=true&collection=collection1_collection&shard.url=https://127.0.0.1:38439/solr/collection1_collection_shard1_replica_n1/&rows=0&version=2&q=*:*&json.facet={"timeseries":{"type":"range","field":"test_dt","start":"2013-01-01T01:00:00.000Z","end":"2016-12-01T01:00:00.000Z","gap":"%2B1YEAR","facet":{"facet_0":"sum(price_f)","facet_1":"max(price_f)","facet_2":"min(price_f)"}}}&field=test_dt&NOW=1506408435047&isShard=true&wt=javabin} hits=125 status=0 QTime=9
[junit4] 2> 25897 INFO (qtp1884836168-1141) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&json.facet={"timeseries":{"type":"range","field":"test_dt","start":"2013-01-01T01:00:00.000Z","end":"2016-12-01T01:00:00.000Z","gap":"%2B1YEAR","facet":{"facet_0":"sum(price_f)","facet_1":"max(price_f)","facet_2":"min(price_f)"}}}&_stateVer_=collection1_collection:4&field=test_dt&rows=0&wt=javabin&version=2} hits=250 status=0 QTime=23
[junit4] 2> 25902 INFO (TEST-StreamExpressionTest.testRankTransform-seed#[5D8C555080DAB489]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testRankTransform
[junit4] 2> 25914 INFO (TEST-StreamExpressionTest.testListStream-seed#[5D8C555080DAB489]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testListStream
[junit4] 2> 25916 INFO (qtp950758985-1149) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@5e9cb910[collection1_collection_shard2_replica_n3] realtime]
[junit4] 2> 25917 INFO (qtp1884836168-1126) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@5e52fc0f[collection1_collection_shard1_replica_n1] realtime]
[junit4] 2> 25917 INFO (qtp1884836168-1126) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&wt=javabin&version=2}{deleteByQuery=*:* (-1579583731231883264)} 0 1
[junit4] 2> 25919 INFO (qtp950758985-1280) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583731235028992,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25919 INFO (qtp950758985-1280) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@15d68926 commitCommandVersion:1579583731235028992
[junit4] 2> 25919 INFO (qtp1884836168-1141) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583731235028992,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25919 INFO (qtp1884836168-1141) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@467dfac4 commitCommandVersion:1579583731235028992
[junit4] 2> 25919 INFO (qtp950758985-1280) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@6308d72c[collection1_collection_shard2_replica_n3] main]
[junit4] 2> 25919 INFO (qtp1884836168-1141) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@7059bcff[collection1_collection_shard1_replica_n1] main]
[junit4] 2> 25920 INFO (qtp950758985-1280) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25921 INFO (searcherExecutor-747-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard1_replica_n1] Registered new searcher Searcher@7059bcff[collection1_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 25921 INFO (searcherExecutor-748-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard2_replica_n3] Registered new searcher Searcher@6308d72c[collection1_collection_shard2_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 25922 INFO (qtp1884836168-1141) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25922 INFO (qtp950758985-1280) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 2
[junit4] 2> 25922 INFO (qtp1884836168-1141) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 2
[junit4] 2> 25923 INFO (qtp950758985-1149) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&commit=true&wt=javabin&version=2}{deleteByQuery=*:* (-1579583731230834688),commit=} 0 8
[junit4] 2> 25926 INFO (qtp950758985-1143) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&wt=javabin&version=2}{add=[hello (1579583731240271872), hello1 (1579583731242369024), hello2 (1579583731242369025)]} 0 2
[junit4] 2> 25929 INFO (qtp950758985-1148) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583731245514752,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25929 INFO (qtp950758985-1148) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@15d68926 commitCommandVersion:1579583731245514752
[junit4] 2> 25929 INFO (qtp1884836168-1128) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1579583731245514752,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 25929 INFO (qtp1884836168-1128) [ ] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 25930 INFO (qtp1884836168-1128) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25930 INFO (qtp1884836168-1128) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 25932 INFO (qtp950758985-1148) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@7a236664[collection1_collection_shard2_replica_n3] main]
[junit4] 2> 25932 INFO (qtp950758985-1148) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 25933 INFO (searcherExecutor-748-thread-1) [ ] o.a.s.c.SolrCore [collection1_collection_shard2_replica_n3] Registered new searcher Searcher@7a236664[collection1_collection_shard2_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_4(7.1.0):c3)))}
[junit4] 2> 25934 INFO (qtp950758985-1148) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=https://127.0.0.1:38491/solr/collection1_collection_shard2_replica_n3/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 4
[junit4] 2> 25935 INFO (qtp950758985-1145) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection_shard2_replica_n3] webapp=/solr path=/update params={_stateVer_=collection1_collection:4&commit=true&wt=javabin&version=2}{commit=} 0 7
[junit4] 2> 25939 INFO (qtp1884836168-1126) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/stream params={indent=off&expr=list(search(collection1,+q%3D"id:hello",++fl%3Did,+sort%3D"id+desc"),search(collection1,+q%3D"id:hello1",+fl%3Did,+sort%3D"id+desc"),search(collection1,+q%3D"id:hello2",+fl%3Did,+sort%3D"id+desc"))&wt=json&version=2.2} status=0 QTime=0
[junit4] 2> 25943 INFO (qtp1884836168-1130) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/select params={q=id:hello&distrib=false&fl=id&sort=id+desc&wt=json&version=2.2} hits=0 status=0 QTime=0
[junit4] 2> 25943 INFO (qtp950758985-1148) [ ] o.a.s.c.S.Request [collection1_collection_shard2_replica_n3] webapp=/solr path=/select params={q=id:hello&distrib=false&fl=id&sort=id+desc&wt=json&version=2.2} hits=1 status=0 QTime=0
[junit4] 2> 25952 INFO (qtp1884836168-1130) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/select params={q=id:hello1&distrib=false&fl=id&sort=id+desc&wt=json&version=2.2} hits=0 status=0 QTime=0
[junit4] 2> 25952 INFO (qtp950758985-1143) [ ] o.a.s.c.S.Request [collection1_collection_shard2_replica_n3] webapp=/solr path=/select params={q=id:hello1&distrib=false&fl=id&sort=id+desc&wt=json&version=2.2} hits=1 status=0 QTime=0
[junit4] 2> 25960 INFO (qtp1884836168-1141) [ ] o.a.s.c.S.Request [collection1_collection_shard1_replica_n1] webapp=/solr path=/select params={q=id:hello2&distrib=false&fl=id&sort=id+desc&wt=json&version=2.2} hits=0 status=0 QTime=0
[junit4] 2> 25961 INFO (qtp950758985-1149) [ ] o.a.s.c.S.Request [collection1_collection_shard2_replica_n3] webapp=/solr path=/select params={q=id:hello2&distrib=false&fl=id&sort=id+desc&wt=json&version=2.2} hits=1 status=0 QTime=0
[junit4] 2> 25964 INFO (TEST-StreamExpressionTest.testListStream-seed#[5D8C555080DAB489]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testListStream
[junit4] 2> 25982 INFO (TEST-StreamExpressionTest.testCopyOf-seed#[5D8C555080DAB489]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testCopyOf
[junit4] 2> 25984 INFO (qtp950758985-1148) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@5c88f5b6[collection1_collection_shard2_replica_n3] realtime]
[junit4] 2> 25986 INFO (qtp1884836168-1281) [ ] o.a.s.u.p.LogUpdateProcessorFactory [collection1_collection
[...truncated too long message...]
trics.reporters.SolrJmxReporter@63ad4acf: rootName = solr_38439, domain = solr.core.collection1_collection.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.collection1_collection.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@280b3df2
[junit4] 2> 96737 INFO (coreCloseExecutor-878-thread-2) [ ] o.a.s.c.SolrCore [destination_shard2_replica_n2] CLOSING SolrCore org.apache.solr.core.SolrCore@3489007b
[junit4] 2> 96738 INFO (coreCloseExecutor-880-thread-2) [ ] o.a.s.c.SolrCore [mainCorpus_shard2_replica_n2] CLOSING SolrCore org.apache.solr.core.SolrCore@44fb421c
[junit4] 2> 96742 INFO (coreCloseExecutor-879-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.workQueue.shard2.leader, tag=950475717
[junit4] 2> 96749 INFO (coreCloseExecutor-878-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.mainCorpus.shard1.leader, tag=705306894
[junit4] 2> 96757 INFO (coreCloseExecutor-880-thread-3) [ ] o.a.s.c.SolrCore [destination_shard1_replica_n1] CLOSING SolrCore org.apache.solr.core.SolrCore@3c0de757
[junit4] 2> 96757 INFO (coreCloseExecutor-877-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.workQueue.shard1.leader, tag=380488660
[junit4] 2> 96749 INFO (coreCloseExecutor-880-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1_collection.shard1.leader, tag=311366389
[junit4] 2> 96757 INFO (coreCloseExecutor-877-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1_collection.shard2.leader, tag=399345601
[junit4] 2> 96757 INFO (coreCloseExecutor-877-thread-2) [ ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
[junit4] 2> 96757 INFO (coreCloseExecutor-877-thread-2) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@83784bc commitCommandVersion:0
[junit4] 2> 96757 INFO (coreCloseExecutor-880-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.mainCorpus.shard2.replica_n2, tag=1157317148
[junit4] 2> 96758 INFO (coreCloseExecutor-880-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@44589dc8: rootName = solr_38439, domain = solr.core.mainCorpus.shard2.replica_n2, service url = null, agent id = null] for registry solr.core.mainCorpus.shard2.replica_n2 / com.codahale.metrics.MetricRegistry@48a129bc
[junit4] 2> 96757 INFO (coreCloseExecutor-878-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.destination.shard2.replica_n2, tag=881393787
[junit4] 2> 96758 INFO (coreCloseExecutor-878-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@553fc38c: rootName = solr_36083, domain = solr.core.destination.shard2.replica_n2, service url = null, agent id = null] for registry solr.core.destination.shard2.replica_n2 / com.codahale.metrics.MetricRegistry@569672bd
[junit4] 2> 96767 INFO (coreCloseExecutor-878-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.destination.shard2.leader, tag=881393787
[junit4] 2> 96773 INFO (coreCloseExecutor-880-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.mainCorpus.shard2.leader, tag=1157317148
[junit4] 2> 96773 INFO (coreCloseExecutor-880-thread-3) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.destination.shard1.replica_n1, tag=1007544151
[junit4] 2> 96773 INFO (coreCloseExecutor-880-thread-3) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@368e0901: rootName = solr_38439, domain = solr.core.destination.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.destination.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@2c4a2fe9
[junit4] 2> 96775 INFO (jetty-closer-98-thread-3) [ ] o.a.s.c.Overseer Overseer (id=98723982936768523-127.0.0.1:36083_solr-n_0000000000) closing
[junit4] 2> 96775 INFO (zkCallback-118-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 96775 INFO (zkCallback-116-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 96775 INFO (zkCallback-117-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 96777 INFO (OverseerStateUpdate-98723982936768523-127.0.0.1:36083_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:36083_solr
[junit4] 2> 96778 WARN (zkCallback-117-thread-2) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 96778 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15ebcf07dc20009, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> 96778 INFO (zkCallback-118-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
[junit4] 2> 96778 INFO (zkCallback-116-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
[junit4] 2> 96779 INFO (jetty-closer-98-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4f41f70e{/solr,null,UNAVAILABLE}
[junit4] 2> 96779 INFO (zkCallback-116-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 96779 INFO (jetty-closer-98-thread-3) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@588b7e99{/solr,null,UNAVAILABLE}
[junit4] 2> 96779 INFO (zkCallback-116-thread-2) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38439_solr
[junit4] 2> 96783 INFO (coreCloseExecutor-880-thread-3) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.destination.shard1.leader, tag=1007544151
[junit4] 2> 96785 WARN (zkCallback-116-thread-2) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 96786 INFO (jetty-closer-98-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1ea1b7da{/solr,null,UNAVAILABLE}
[junit4] 2> 98276 WARN (zkCallback-119-thread-1) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 98276 INFO (jetty-closer-98-thread-4) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@46654fc9{/solr,null,UNAVAILABLE}
[junit4] 2> 98277 ERROR (SUITE-StreamExpressionTest-seed#[5D8C555080DAB489]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 98278 INFO (SUITE-StreamExpressionTest-seed#[5D8C555080DAB489]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44751 44751
[junit4] 2> 104694 INFO (Thread-383) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44751 44751
[junit4] 2> 104695 WARN (Thread-383) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 73 /solr/aliases.json
[junit4] 2> 11 /solr/configs/conf
[junit4] 2> 4 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 73 /solr/clusterprops.json
[junit4] 2> 73 /solr/clusterstate.json
[junit4] 2> 20 /solr/collections/destination/state.json
[junit4] 2> 20 /solr/collections/parallelDestinationCollection/state.json
[junit4] 2> 20 /solr/collections/destinationCollection/state.json
[junit4] 2> 20 /solr/collections/mainCorpus/state.json
[junit4] 2> 18 /solr/collections/workQueue/state.json
[junit4] 2> 18 /solr/collections/parallelDestinationCollection1/state.json
[junit4] 2> 12 /solr/collections/collection1_collection/state.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 73 /solr/live_nodes
[junit4] 2> 73 /solr/collections
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.io.stream.StreamExpressionTest_5D8C555080DAB489-001
[junit4] 2> NOTE: test params are: codec=Lucene70, sim=RandomSimilarity(queryNorm=true): {}, locale=af, timezone=America/Cordoba
[junit4] 2> NOTE: Linux 4.10.0-33-generic amd64/Oracle Corporation 9 (64-bit)/cpus=8,threads=1,free=243618184,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [CubedRootEvaluatorTest, AndEvaluatorTest, ArcSineEvaluatorTest, LargeVolumeJettyTest, SchemaTest, TestCoreAdmin, DocumentAnalysisResponseTest, SolrExampleXMLTest, EmpiricalDistributionEvaluatorTest, HyperbolicTangentEvaluatorTest, TestPathTrie, StreamExpressionTest]
[junit4] Completed [124/145 (1!)] on J2 in 82.87s, 113 tests, 1 error, 9 skipped <<< FAILURES!
[...truncated 38714 lines...]