You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2017/08/24 21:52:19 UTC
[JENKINS-EA] Lucene-Solr-7.0-Linux (64bit/jdk-9-ea+181) - Build #
254 - Still Failing!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.0-Linux/254/
Java: 64bit/jdk-9-ea+181 -XX:+UseCompressedOops -XX:+UseG1GC --illegal-access=deny
4 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest
Error Message:
3 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest: 1) Thread[id=15009, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[8DC1F601C865E878]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest] at java.base@9/jdk.internal.misc.Unsafe.park(Native Method) at java.base@9/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2062) at java.base@9/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435) at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) 2) Thread[id=15007, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest] at java.base@9/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@9/java.lang.Thread.run(Thread.java:844) 3) Thread[id=15008, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[8DC1F601C865E878]-SendThread(127.0.0.1:39369), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest] at java.base@9/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest:
1) Thread[id=15009, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[8DC1F601C865E878]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@9/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2062)
at java.base@9/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
2) Thread[id=15007, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
at java.base@9/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@9/java.lang.Thread.run(Thread.java:844)
3) Thread[id=15008, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[8DC1F601C865E878]-SendThread(127.0.0.1:39369), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
at java.base@9/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
at __randomizedtesting.SeedInfo.seed([8DC1F601C865E878]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=15008, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[8DC1F601C865E878]-SendThread(127.0.0.1:39369), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest] at java.base@9/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=15008, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[8DC1F601C865E878]-SendThread(127.0.0.1:39369), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
at java.base@9/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
at __randomizedtesting.SeedInfo.seed([8DC1F601C865E878]:0)
FAILED: org.apache.solr.cloud.CdcrBootstrapTest.testBootstrapWithContinousIndexingOnSourceCluster
Error Message:
Document mismatch on target after sync expected:<2000> but was:<1901>
Stack Trace:
java.lang.AssertionError: Document mismatch on target after sync expected:<2000> but was:<1901>
at __randomizedtesting.SeedInfo.seed([8DC1F601C865E878:5984BD582F335B83]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.apache.solr.cloud.CdcrBootstrapTest.testBootstrapWithContinousIndexingOnSourceCluster(CdcrBootstrapTest.java:309)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test
Error Message:
Timed out waiting for replica core_node2 (1503610888661) to replicate from leader core_node1 (1503610914938)
Stack Trace:
java.lang.AssertionError: Timed out waiting for replica core_node2 (1503610888661) to replicate from leader core_node1 (1503610914938)
at __randomizedtesting.SeedInfo.seed([8DC1F601C865E878:595C9DB66998580]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForReplicationFromReplicas(AbstractFullDistribZkTestBase.java:2137)
at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test(ChaosMonkeyNothingIsSafeWithPullReplicasTest.java:268)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
Build Log:
[...truncated 11938 lines...]
[junit4] Suite: org.apache.solr.cloud.CdcrBootstrapTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-7.0-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_8DC1F601C865E878-001/init-core-data-001
[junit4] 2> 965181 INFO (SUITE-CdcrBootstrapTest-seed#[8DC1F601C865E878]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 965182 INFO (SUITE-CdcrBootstrapTest-seed#[8DC1F601C865E878]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0)
[junit4] 2> 965183 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testConvertClusterToCdcrAndBootstrap
[junit4] 2> 965183 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in /home/jenkins/workspace/Lucene-Solr-7.0-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_8DC1F601C865E878-001/cdcr-target-001
[junit4] 2> 965183 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 965183 INFO (Thread-6394) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 965183 INFO (Thread-6394) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 965185 ERROR (Thread-6394) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 965283 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.ZkTestServer start zk server on port:37551
[junit4] 2> 965287 INFO (jetty-launcher-1089-thread-1) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 965289 INFO (jetty-launcher-1089-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2a2e55e0{/solr,null,AVAILABLE}
[junit4] 2> 965290 INFO (jetty-launcher-1089-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@6c6502d3{HTTP/1.1,[http/1.1]}{127.0.0.1:41615}
[junit4] 2> 965290 INFO (jetty-launcher-1089-thread-1) [ ] o.e.j.s.Server Started @967173ms
[junit4] 2> 965290 INFO (jetty-launcher-1089-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41615}
[junit4] 2> 965290 ERROR (jetty-launcher-1089-thread-1) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 965290 INFO (jetty-launcher-1089-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.0.0
[junit4] 2> 965290 INFO (jetty-launcher-1089-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 965290 INFO (jetty-launcher-1089-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 965290 INFO (jetty-launcher-1089-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-08-24T21:33:34.675907Z
[junit4] 2> 965292 INFO (jetty-launcher-1089-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 965297 INFO (jetty-launcher-1089-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37551/solr
[junit4] 2> 965346 INFO (jetty-launcher-1089-thread-1) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 965347 INFO (jetty-launcher-1089-thread-1) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41615_solr
[junit4] 2> 965347 INFO (jetty-launcher-1089-thread-1) [ ] o.a.s.c.Overseer Overseer (id=98540612129259523-127.0.0.1:41615_solr-n_0000000000) starting
[junit4] 2> 965352 INFO (jetty-launcher-1089-thread-1) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41615_solr
[junit4] 2> 965353 INFO (zkCallback-1096-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 965443 INFO (jetty-launcher-1089-thread-1) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41615.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47b7d3bc
[junit4] 2> 965450 INFO (jetty-launcher-1089-thread-1) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41615.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47b7d3bc
[junit4] 2> 965450 INFO (jetty-launcher-1089-thread-1) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41615.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47b7d3bc
[junit4] 2> 965451 INFO (jetty-launcher-1089-thread-1) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.0-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_8DC1F601C865E878-001/cdcr-target-001/node1/.
[junit4] 1> Target zkHost = 127.0.0.1:37551/solr
[junit4] 2> 965468 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in /home/jenkins/workspace/Lucene-Solr-7.0-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_8DC1F601C865E878-001/cdcr-source-001
[junit4] 2> 965468 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 965468 INFO (Thread-6404) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 965468 INFO (Thread-6404) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 965469 ERROR (Thread-6404) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 965568 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.ZkTestServer start zk server on port:38715
[junit4] 2> 965573 INFO (jetty-launcher-1101-thread-1) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 965573 INFO (jetty-launcher-1101-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@96d8bb6{/solr,null,AVAILABLE}
[junit4] 2> 965573 INFO (jetty-launcher-1101-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@20b8490e{HTTP/1.1,[http/1.1]}{127.0.0.1:34639}
[junit4] 2> 965573 INFO (jetty-launcher-1101-thread-1) [ ] o.e.j.s.Server Started @967457ms
[junit4] 2> 965574 INFO (jetty-launcher-1101-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34639}
[junit4] 2> 965574 ERROR (jetty-launcher-1101-thread-1) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 965574 INFO (jetty-launcher-1101-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.0.0
[junit4] 2> 965574 INFO (jetty-launcher-1101-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 965574 INFO (jetty-launcher-1101-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 965574 INFO (jetty-launcher-1101-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-08-24T21:33:34.959331Z
[junit4] 2> 965575 INFO (jetty-launcher-1101-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 965579 INFO (jetty-launcher-1101-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38715/solr
[junit4] 2> 965622 INFO (jetty-launcher-1101-thread-1) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 965622 INFO (jetty-launcher-1101-thread-1) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34639_solr
[junit4] 2> 965623 INFO (jetty-launcher-1101-thread-1) [ ] o.a.s.c.Overseer Overseer (id=98540612147871747-127.0.0.1:34639_solr-n_0000000000) starting
[junit4] 2> 965625 INFO (jetty-launcher-1101-thread-1) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34639_solr
[junit4] 2> 965626 INFO (zkCallback-1108-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 965665 INFO (jetty-launcher-1101-thread-1) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34639.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47b7d3bc
[junit4] 2> 965670 INFO (jetty-launcher-1101-thread-1) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34639.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47b7d3bc
[junit4] 2> 965670 INFO (jetty-launcher-1101-thread-1) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34639.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47b7d3bc
[junit4] 2> 965671 INFO (jetty-launcher-1101-thread-1) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.0-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_8DC1F601C865E878-001/cdcr-source-001/node1/.
[junit4] 2> 965688 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 965689 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:38715/solr ready
[junit4] 2> 965690 INFO (qtp1562951153-12206) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=cdcr-source&name=cdcr-source&nrtReplicas=1&action=CREATE&numShards=1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 965691 INFO (OverseerThreadFactory-3596-thread-1) [ ] o.a.s.c.CreateCollectionCmd Create collection cdcr-source
[junit4] 2> 965794 INFO (OverseerStateUpdate-98540612147871747-127.0.0.1:34639_solr-n_0000000000) [ ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"cdcr-source",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"cdcr-source_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:34639/solr",
[junit4] 2> "type":"NRT"}
[junit4] 2> 965996 INFO (qtp1562951153-12208) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=cdcr-source&newCollection=true&collection=cdcr-source&version=2&replicaType=NRT&coreNodeName=core_node1&name=cdcr-source_shard1_replica_n1&action=CREATE&numShards=1&shard=shard1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin
[junit4] 2> 965996 INFO (qtp1562951153-12208) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 966101 INFO (zkCallback-1108-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/state.json] for collection [cdcr-source] has occurred - updating... (live nodes size: [1])
[junit4] 2> 966102 INFO (zkCallback-1108-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/state.json] for collection [cdcr-source] has occurred - updating... (live nodes size: [1])
[junit4] 2> 967008 INFO (qtp1562951153-12208) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 967013 INFO (qtp1562951153-12208) [ ] o.a.s.s.IndexSchema [cdcr-source_shard1_replica_n1] Schema name=minimal
[junit4] 2> 967015 INFO (qtp1562951153-12208) [ ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 967015 INFO (qtp1562951153-12208) [ ] o.a.s.c.CoreContainer Creating SolrCore 'cdcr-source_shard1_replica_n1' using configuration from collection cdcr-source, trusted=true
[junit4] 2> 967015 INFO (qtp1562951153-12208) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34639.solr.core.cdcr-source.shard1.replica_n1' (registry 'solr.core.cdcr-source.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47b7d3bc
[junit4] 2> 967015 INFO (qtp1562951153-12208) [ ] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 967016 INFO (qtp1562951153-12208) [ ] o.a.s.c.SolrCore [[cdcr-source_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.0-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_8DC1F601C865E878-001/cdcr-source-001/node1/cdcr-source_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.0-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_8DC1F601C865E878-001/cdcr-source-001/node1/./cdcr-source_shard1_replica_n1/data/]
[junit4] 2> 967083 INFO (qtp1562951153-12208) [ ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 967083 INFO (qtp1562951153-12208) [ ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 967084 INFO (qtp1562951153-12208) [ ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 967084 INFO (qtp1562951153-12208) [ ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 967085 INFO (qtp1562951153-12208) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@525c759[cdcr-source_shard1_replica_n1] main]
[junit4] 2> 967087 INFO (qtp1562951153-12208) [ ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/cdcr-source
[junit4] 2> 967087 INFO (qtp1562951153-12208) [ ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/cdcr-source
[junit4] 2> 967088 INFO (qtp1562951153-12208) [ ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 967088 INFO (searcherExecutor-3599-thread-1) [ ] o.a.s.c.SolrCore [cdcr-source_shard1_replica_n1] Registered new searcher Searcher@525c759[cdcr-source_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 967088 INFO (qtp1562951153-12208) [ ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1576649796063592448
[junit4] 2> 967092 INFO (qtp1562951153-12208) [ ] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 967092 INFO (qtp1562951153-12208) [ ] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 967092 INFO (qtp1562951153-12208) [ ] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:34639/solr/cdcr-source_shard1_replica_n1/
[junit4] 2> 967092 INFO (qtp1562951153-12208) [ ] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 967092 INFO (qtp1562951153-12208) [ ] o.a.s.c.SyncStrategy http://127.0.0.1:34639/solr/cdcr-source_shard1_replica_n1/ has no replicas
[junit4] 2> 967092 INFO (qtp1562951153-12208) [ ] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 967093 INFO (qtp1562951153-12208) [ ] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:34639/solr/cdcr-source_shard1_replica_n1/ shard1
[junit4] 2> 967194 INFO (zkCallback-1108-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/state.json] for collection [cdcr-source] has occurred - updating... (live nodes size: [1])
[junit4] 2> 967194 INFO (zkCallback-1108-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/state.json] for collection [cdcr-source] has occurred - updating... (live nodes size: [1])
[junit4] 2> 967243 INFO (qtp1562951153-12208) [ ] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 967245 INFO (qtp1562951153-12208) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=cdcr-source&newCollection=true&collection=cdcr-source&version=2&replicaType=NRT&coreNodeName=core_node1&name=cdcr-source_shard1_replica_n1&action=CREATE&numShards=1&shard=shard1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin} status=0 QTime=1249
[junit4] 2> 967247 INFO (qtp1562951153-12206) [ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 967347 INFO (zkCallback-1108-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/state.json] for collection [cdcr-source] has occurred - updating... (live nodes size: [1])
[junit4] 2> 967347 INFO (zkCallback-1108-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/state.json] for collection [cdcr-source] has occurred - updating... (live nodes size: [1])
[junit4] 2> 967692 INFO (OverseerCollectionConfigSetProcessor-98540612147871747-127.0.0.1:34639_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 968248 INFO (qtp1562951153-12206) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=cdcr-source&name=cdcr-source&nrtReplicas=1&action=CREATE&numShards=1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin&version=2} status=0 QTime=2557
[junit4] 1> Adding 10 docs with commit=true, numDocs=100
[junit4] 2> 968256 INFO (qtp1562951153-12207) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&wt=javabin&version=2}{add=[source_0 (1576649797285183488), source_1 (1576649797286232064), source_2 (1576649797286232065), source_3 (1576649797286232066), source_4 (1576649797286232067), source_5 (1576649797286232068), source_6 (1576649797286232069), source_7 (1576649797286232070), source_8 (1576649797286232071), source_9 (1576649797286232072), ... (100 adds)]} 0 3
[junit4] 2> 968256 INFO (qtp1562951153-12202) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1576649797288329217,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 968256 INFO (qtp1562951153-12202) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@69f82b14 commitCommandVersion:1576649797288329217
[junit4] 2> 968384 INFO (qtp1562951153-12202) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@325666e[cdcr-source_shard1_replica_n1] main]
[junit4] 2> 968384 INFO (qtp1562951153-12202) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 968385 INFO (searcherExecutor-3599-thread-1) [ ] o.a.s.c.SolrCore [cdcr-source_shard1_replica_n1] Registered new searcher Searcher@325666e[cdcr-source_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C100)))}
[junit4] 2> 968385 INFO (qtp1562951153-12202) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 128
[junit4] 1> Adding 10 docs with commit=true, numDocs=200
[junit4] 2> 968387 INFO (qtp1562951153-12208) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&wt=javabin&version=2}{add=[source_100 (1576649797423595520), source_101 (1576649797424644096), source_102 (1576649797424644097), source_103 (1576649797424644098), source_104 (1576649797424644099), source_105 (1576649797424644100), source_106 (1576649797425692672), source_107 (1576649797425692673), source_108 (1576649797425692674), source_109 (1576649797425692675), ... (100 adds)]} 0 2
[junit4] 2> 968388 INFO (qtp1562951153-12206) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1576649797426741248,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 968388 INFO (qtp1562951153-12206) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@69f82b14 commitCommandVersion:1576649797426741248
[junit4] 2> 968513 INFO (qtp1562951153-12206) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@4679cd75[cdcr-source_shard1_replica_n1] main]
[junit4] 2> 968514 INFO (qtp1562951153-12206) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 968515 INFO (searcherExecutor-3599-thread-1) [ ] o.a.s.c.SolrCore [cdcr-source_shard1_replica_n1] Registered new searcher Searcher@4679cd75[cdcr-source_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C100) Uninverting(_1(7.0.0):C100)))}
[junit4] 2> 968515 INFO (qtp1562951153-12206) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 127
[junit4] 1> Adding 10 docs with commit=true, numDocs=300
[junit4] 2> 968519 INFO (qtp1562951153-12207) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&wt=javabin&version=2}{add=[source_200 (1576649797560958976), source_201 (1576649797562007552), source_202 (1576649797562007553), source_203 (1576649797562007554), source_204 (1576649797562007555), source_205 (1576649797562007556), source_206 (1576649797562007557), source_207 (1576649797562007558), source_208 (1576649797563056128), source_209 (1576649797563056129), ... (100 adds)]} 0 3
[junit4] 2> 968520 INFO (qtp1562951153-12202) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1576649797565153280,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 968520 INFO (qtp1562951153-12202) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@69f82b14 commitCommandVersion:1576649797565153280
[junit4] 2> 968674 INFO (qtp1562951153-12202) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@53323ea8[cdcr-source_shard1_replica_n1] main]
[junit4] 2> 968675 INFO (qtp1562951153-12202) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 968675 INFO (searcherExecutor-3599-thread-1) [ ] o.a.s.c.SolrCore [cdcr-source_shard1_replica_n1] Registered new searcher Searcher@53323ea8[cdcr-source_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C100) Uninverting(_1(7.0.0):C100) Uninverting(_2(7.0.0):C100)))}
[junit4] 2> 968676 INFO (qtp1562951153-12202) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 155
[junit4] 1> Adding 10 docs with commit=true, numDocs=400
[junit4] 2> 968678 INFO (qtp1562951153-12208) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&wt=javabin&version=2}{add=[source_300 (1576649797728731136), source_301 (1576649797729779712), source_302 (1576649797729779713), source_303 (1576649797729779714), source_304 (1576649797729779715), source_305 (1576649797729779716), source_306 (1576649797729779717), source_307 (1576649797729779718), source_308 (1576649797729779719), source_309 (1576649797729779720), ... (100 adds)]} 0 1
[junit4] 2> 968678 INFO (qtp1562951153-12206) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1576649797730828325,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 968678 INFO (qtp1562951153-12206) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@69f82b14 commitCommandVersion:1576649797730828325
[junit4] 2> 968924 INFO (qtp1562951153-12206) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@143b33e0[cdcr-source_shard1_replica_n1] main]
[junit4] 2> 968925 INFO (qtp1562951153-12206) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 968926 INFO (searcherExecutor-3599-thread-1) [ ] o.a.s.c.SolrCore [cdcr-source_shard1_replica_n1] Registered new searcher Searcher@143b33e0[cdcr-source_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C100) Uninverting(_1(7.0.0):C100) Uninverting(_2(7.0.0):C100) Uninverting(_3(7.0.0):C100)))}
[junit4] 2> 968926 INFO (qtp1562951153-12206) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 247
[junit4] 1> Adding 10 docs with commit=true, numDocs=500
[junit4] 2> 968930 INFO (qtp1562951153-12207) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&wt=javabin&version=2}{add=[source_400 (1576649797991923712), source_401 (1576649797992972288), source_402 (1576649797992972289), source_403 (1576649797992972290), source_404 (1576649797992972291), source_405 (1576649797992972292), source_406 (1576649797992972293), source_407 (1576649797992972294), source_408 (1576649797992972295), source_409 (1576649797994020864), ... (100 adds)]} 0 2
[junit4] 2> 968930 INFO (qtp1562951153-12202) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1576649797995069466,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 968931 INFO (qtp1562951153-12202) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@69f82b14 commitCommandVersion:1576649797995069466
[junit4] 2> 969138 INFO (qtp1562951153-12202) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@6bf68aba[cdcr-source_shard1_replica_n1] main]
[junit4] 2> 969138 INFO (qtp1562951153-12202) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 969139 INFO (searcherExecutor-3599-thread-1) [ ] o.a.s.c.SolrCore [cdcr-source_shard1_replica_n1] Registered new searcher Searcher@6bf68aba[cdcr-source_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C100) Uninverting(_1(7.0.0):C100) Uninverting(_2(7.0.0):C100) Uninverting(_3(7.0.0):C100) Uninverting(_4(7.0.0):C100)))}
[junit4] 2> 969139 INFO (qtp1562951153-12202) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 208
[junit4] 1> Adding 10 docs with commit=true, numDocs=600
[junit4] 2> 969142 INFO (qtp1562951153-12208) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&wt=javabin&version=2}{add=[source_500 (1576649798215270400), source_501 (1576649798215270401), source_502 (1576649798215270402), source_503 (1576649798215270403), source_504 (1576649798215270404), source_505 (1576649798215270405), source_506 (1576649798215270406), source_507 (1576649798215270407), source_508 (1576649798215270408), source_509 (1576649798215270409), ... (100 adds)]} 0 1
[junit4] 2> 969142 INFO (qtp1562951153-12206) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1576649798217367553,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 969142 INFO (qtp1562951153-12206) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@69f82b14 commitCommandVersion:1576649798217367553
[junit4] 2> 969480 INFO (qtp1562951153-12206) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@23eb3551[cdcr-source_shard1_replica_n1] main]
[junit4] 2> 969480 INFO (qtp1562951153-12206) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 969481 INFO (searcherExecutor-3599-thread-1) [ ] o.a.s.c.SolrCore [cdcr-source_shard1_replica_n1] Registered new searcher Searcher@23eb3551[cdcr-source_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C100) Uninverting(_1(7.0.0):C100) Uninverting(_2(7.0.0):C100) Uninverting(_3(7.0.0):C100) Uninverting(_4(7.0.0):C100) Uninverting(_5(7.0.0):C100)))}
[junit4] 2> 969481 INFO (qtp1562951153-12206) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 338
[junit4] 1> Adding 10 docs with commit=true, numDocs=700
[junit4] 2> 969484 INFO (qtp1562951153-12207) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&wt=javabin&version=2}{add=[source_600 (1576649798572834816), source_601 (1576649798573883392), source_602 (1576649798573883393), source_603 (1576649798573883394), source_604 (1576649798573883395), source_605 (1576649798573883396), source_606 (1576649798573883397), source_607 (1576649798573883398), source_608 (1576649798573883399), source_609 (1576649798573883400), ... (100 adds)]} 0 2
[junit4] 2> 969485 INFO (qtp1562951153-12202) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1576649798577029120,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 969485 INFO (qtp1562951153-12202) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@69f82b14 commitCommandVersion:1576649798577029120
[junit4] 2> 969739 INFO (qtp1562951153-12202) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@c5451ed[cdcr-source_shard1_replica_n1] main]
[junit4] 2> 969739 INFO (qtp1562951153-12202) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 969742 INFO (searcherExecutor-3599-thread-1) [ ] o.a.s.c.SolrCore [cdcr-source_shard1_replica_n1] Registered new searcher Searcher@c5451ed[cdcr-source_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C100) Uninverting(_1(7.0.0):C100) Uninverting(_2(7.0.0):C100) Uninverting(_3(7.0.0):C100) Uninverting(_4(7.0.0):C100) Uninverting(_5(7.0.0):C100) Uninverting(_6(7.0.0):C100)))}
[junit4] 2> 969742 INFO (qtp1562951153-12202) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 256
[junit4] 1> Adding 10 docs with commit=true, numDocs=800
[junit4] 2> 969747 INFO (qtp1562951153-12208) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&wt=javabin&version=2}{add=[source_700 (1576649798847561728), source_701 (1576649798848610304), source_702 (1576649798849658880), source_703 (1576649798849658881), source_704 (1576649798849658882), source_705 (1576649798849658883), source_706 (1576649798849658884), source_707 (1576649798849658885), source_708 (1576649798849658886), source_709 (1576649798849658887), ... (100 adds)]} 0 3
[junit4] 2> 969748 INFO (qtp1562951153-12206) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1576649798852804608,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 969748 INFO (qtp1562951153-12206) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@69f82b14 commitCommandVersion:1576649798852804608
[junit4] 2> 970570 INFO (qtp1562951153-12206) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@5b758905[cdcr-source_shard1_replica_n1] main]
[junit4] 2> 970571 INFO (qtp1562951153-12206) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 970571 INFO (searcherExecutor-3599-thread-1) [ ] o.a.s.c.SolrCore [cdcr-source_shard1_replica_n1] Registered new searcher Searcher@5b758905[cdcr-source_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C100) Uninverting(_1(7.0.0):C100) Uninverting(_2(7.0.0):C100) Uninverting(_3(7.0.0):C100) Uninverting(_4(7.0.0):C100) Uninverting(_5(7.0.0):C100) Uninverting(_6(7.0.0):C100) Uninverting(_7(7.0.0):C100)))}
[junit4] 2> 970571 INFO (qtp1562951153-12206) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 823
[junit4] 1> Adding 10 docs with commit=true, numDocs=900
[junit4] 2> 970575 INFO (qtp1562951153-12207) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&wt=javabin&version=2}{add=[source_800 (1576649799716831232), source_801 (1576649799717879808), source_802 (1576649799717879809), source_803 (1576649799717879810), source_804 (1576649799717879811), source_805 (1576649799717879812), source_806 (1576649799717879813), source_807 (1576649799717879814), source_808 (1576649799717879815), source_809 (1576649799717879816), ... (100 adds)]} 0 2
[junit4] 2> 970579 INFO (qtp1562951153-12202) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1576649799724171264,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 970579 INFO (qtp1562951153-12202) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@69f82b14 commitCommandVersion:1576649799724171264
[junit4] 2> 970965 INFO (qtp1562951153-12202) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@62743616[cdcr-source_shard1_replica_n1] main]
[junit4] 2> 970965 INFO (qtp1562951153-12202) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 970968 INFO (searcherExecutor-3599-thread-1) [ ] o.a.s.c.SolrCore [cdcr-source_shard1_replica_n1] Registered new searcher Searcher@62743616[cdcr-source_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C100) Uninverting(_1(7.0.0):C100) Uninverting(_2(7.0.0):C100) Uninverting(_3(7.0.0):C100) Uninverting(_4(7.0.0):C100) Uninverting(_5(7.0.0):C100) Uninverting(_6(7.0.0):C100) Uninverting(_7(7.0.0):C100) Uninverting(_8(7.0.0):C100)))}
[junit4] 2> 970970 INFO (qtp1562951153-12202) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 390
[junit4] 1> Adding 10 docs with commit=true, numDocs=1000
[junit4] 2> 970973 INFO (qtp1562951153-12202) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&wt=javabin&version=2}{add=[source_900 (1576649800135213056), source_901 (1576649800136261632), source_902 (1576649800136261633), source_903 (1576649800136261634), source_904 (1576649800136261635), source_905 (1576649800136261636), source_906 (1576649800136261637), source_907 (1576649800136261638), source_908 (1576649800136261639), source_909 (1576649800136261640), ... (100 adds)]} 0 2
[junit4] 2> 970974 INFO (qtp1562951153-12202) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1576649800138358784,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 970974 INFO (qtp1562951153-12202) [ ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@69f82b14 commitCommandVersion:1576649800138358784
[junit4] 2> 971432 INFO (qtp1562951153-12202) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@624ed0fb[cdcr-source_shard1_replica_n1] main]
[junit4] 2> 971432 INFO (qtp1562951153-12202) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 971433 INFO (searcherExecutor-3599-thread-1) [ ] o.a.s.c.SolrCore [cdcr-source_shard1_replica_n1] Registered new searcher Searcher@624ed0fb[cdcr-source_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.0.0):C100) Uninverting(_1(7.0.0):C100) Uninverting(_2(7.0.0):C100) Uninverting(_3(7.0.0):C100) Uninverting(_4(7.0.0):C100) Uninverting(_5(7.0.0):C100) Uninverting(_6(7.0.0):C100) Uninverting(_7(7.0.0):C100) Uninverting(_8(7.0.0):C100) Uninverting(_9(7.0.0):C100)))}
[junit4] 2> 971433 INFO (qtp1562951153-12202) [ ] o.a.s.u.p.LogUpdateProcessorFactory [cdcr-source_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-source:4&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 459
[junit4] 2> 971434 INFO (qtp1562951153-12208) [ ] o.a.s.c.S.Request [cdcr-source_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&_stateVer_=cdcr-source:4&wt=javabin&version=2} hits=1000 status=0 QTime=0
[junit4] 2> 971435 INFO (qtp1562951153-12206) [ ] o.a.s.c.S.Request [cdcr-source_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&_stateVer_=cdcr-source:4&getVersions=1000&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 971446 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@20b8490e{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 971447 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1456962855
[junit4] 2> 971451 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 971451 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@51b99d53: rootName = solr_34639, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@712e0fe3
[junit4] 2> 971459 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 971459 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5d7ca7a6: rootName = solr_34639, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@24c319ed
[junit4] 2> 971465 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 971465 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@65347a6: rootName = solr_34639, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@3284286
[junit4] 2> 971466 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 971467 INFO (coreCloseExecutor-3604-thread-1) [ ] o.a.s.c.SolrCore [cdcr-source_shard1_replica_n1] CLOSING SolrCore org.apache.solr.core.SolrCore@51ea8225
[junit4] 2> 971467 INFO (coreCloseExecutor-3604-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.cdcr-source.shard1.replica_n1, tag=1374323237
[junit4] 2> 971468 INFO (coreCloseExecutor-3604-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@62e462be: rootName = solr_34639, domain = solr.core.cdcr-source.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.cdcr-source.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@14680508
[junit4] 2> 971468 INFO (zkCallback-1108-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/state.json] for collection [cdcr-source] has occurred - updating... (live nodes size: [1])
[junit4] 2> 971468 INFO (zkCallback-1108-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/state.json] for collection [cdcr-source] has occurred - updating... (live nodes size: [1])
[junit4] 2> 971477 INFO (coreCloseExecutor-3604-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.cdcr-source.shard1.leader, tag=1374323237
[junit4] 2> 971595 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.Overseer Overseer (id=98540612147871747-127.0.0.1:34639_solr-n_0000000000) closing
[junit4] 2> 971595 INFO (OverseerStateUpdate-98540612147871747-127.0.0.1:34639_solr-n_0000000000) [ ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:34639_solr
[junit4] 2> 971596 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15e162a27060003, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> 971596 WARN (zkCallback-1108-thread-2) [ ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 971597 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@96d8bb6{/solr,null,UNAVAILABLE}
[junit4] 2> 971597 INFO (zkCallback-1115-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 971597 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.e.j.s.Server jetty-9.3.14.v20161028
[junit4] 2> 971598 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@153cd956{/solr,null,AVAILABLE}
[junit4] 2> 971598 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3c04aa9b{HTTP/1.1,[http/1.1]}{127.0.0.1:41301}
[junit4] 2> 971598 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.e.j.s.Server Started @973482ms
[junit4] 2> 971599 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41301}
[junit4] 2> 971599 ERROR (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 971599 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.0.0
[junit4] 2> 971599 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 971599 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 971599 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-08-24T21:33:40.984640Z
[junit4] 2> 971604 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 971611 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:38715/solr
[junit4] 2> 971630 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 971630 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41301_solr
[junit4] 2> 971631 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.Overseer Overseer (id=98540612147871755-127.0.0.1:41301_solr-n_0000000001) starting
[junit4] 2> 971648 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41301_solr
[junit4] 2> 971649 INFO (zkCallback-1121-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 971649 INFO (zkCallback-1115-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 971718 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41301.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47b7d3bc
[junit4] 2> 971725 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41301.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47b7d3bc
[junit4] 2> 971725 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41301.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47b7d3bc
[junit4] 2> 971732 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /home/jenkins/workspace/Lucene-Solr-7.0-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_8DC1F601C865E878-001/cdcr-source-001/node1/.
[junit4] 2> 971732 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.CorePropertiesLocator Cores are: [cdcr-source_shard1_replica_n1]
[junit4] 2> 971733 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 971736 INFO (zkCallback-1121-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/state.json] for collection [cdcr-source] has occurred - updating... (live nodes size: [1])
[junit4] 2> 971736 INFO (zkCallback-1121-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/state.json] for collection [cdcr-source] has occurred - updating... (live nodes size: [1])
[junit4] 2> 972737 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 972740 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
[junit4] 2> 972741 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.s.IndexSchema [cdcr-source_shard1_replica_n1] Schema name=minimal
[junit4] 2> 972742 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 972743 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.s.ManagedIndexSchema Created and persisted managed schema znode at /configs/cdcr-source/managed-schema
[junit4] 2> 972743 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/cdcr-source/schema.xml to /configs/cdcr-source/schema.xml.bak
[junit4] 2> 972743 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.c.CoreContainer Creating SolrCore 'cdcr-source_shard1_replica_n1' using configuration from collection cdcr-source, trusted=true
[junit4] 2> 972744 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41301.solr.core.cdcr-source.shard1.replica_n1' (registry 'solr.core.cdcr-source.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47b7d3bc
[junit4] 2> 972744 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 972744 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.c.SolrCore [[cdcr-source_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.0-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_8DC1F601C865E878-001/cdcr-source-001/node1/cdcr-source_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.0-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_8DC1F601C865E878-001/cdcr-source-001/node1/./cdcr-source_shard1_replica_n1/data/]
[junit4] 2> 972801 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 972801 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 972803 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 972803 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 972824 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@1646f2ad[cdcr-source_shard1_replica_n1] main]
[junit4] 2> 972825 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/cdcr-source
[junit4] 2> 972825 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/cdcr-source
[junit4] 2> 972825 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/cdcr-source/managed-schema
[junit4] 2> 972825 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
[junit4] 2> 972826 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 972828 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.h.CdcrBufferStateManager Created znode /collections/cdcr-source/cdcr/state/buffer
[junit4] 2> 972831 INFO (coreLoadExecutor-3613-thread-1) [ ] o.a.s.h.CdcrProcessStateManager Created znode /collections/cdcr-source/cdcr/state/process
[junit4] 2> 972845 WARN (cdcr-update-log-synchronizer-3619-thread-1) [ ] o.a.s.h.CdcrUpdateLogSynchronizer Caught unexpected exception
[junit4] 2> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:41301/solr/cdcr-source_shard1_replica_n1: SolrCore is loading
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
[junit4] 2> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
[junit4] 2> at org.apache.solr.handler.CdcrUpdateLogSynchronizer$UpdateLogSynchronisation.run(CdcrUpdateLogSynchronizer.java:147)
[junit4] 2> at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
[junit4] 2> at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
[junit4] 2> at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:300)
[junit4] 2> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
[junit4] 2> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> 972846 INFO (searcherExecutor-3614-thread-1) [ ] o.a.s.c.SolrCore [cdcr-source_shard1_replica_n1] Registered new searcher Searcher@1646f2ad[cdcr-source_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_a(7.0.0):C1000)))}
[junit4] 2> 972849 INFO (coreZkRegister-3606-thread-1) [ ] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 972849 INFO (coreZkRegister-3606-thread-1) [ ] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 972849 INFO (coreZkRegister-3606-thread-1) [ ] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:41301/solr/cdcr-source_shard1_replica_n1/
[junit4] 2> 972849 INFO (coreZkRegister-3606-thread-1) [ ] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 972849 INFO (coreZkRegister-3606-thread-1) [ ] o.a.s.c.SyncStrategy http://127.0.0.1:41301/solr/cdcr-source_shard1_replica_n1/ has no replicas
[junit4] 2> 972849 INFO (coreZkRegister-3606-thread-1) [ ] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 972850 INFO (zkCallback-1121-thread-2) [ ] o.a.s.h.CdcrLeaderStateManager Received new leader state @ cdcr-source:shard1
[junit4] 2> 972855 INFO (coreZkRegister-3606-thread-1) [ ] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:41301/solr/cdcr-source_shard1_replica_n1/ shard1
[junit4] 2> 972856 INFO (coreZkRegister-3606-thread-1) [ ] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 972959 INFO (zkCallback-1121-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/state.json] for collection [cdcr-source] has occurred - updating... (live nodes size: [1])
[junit4] 2> 972959 INFO (zkCallback-1121-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/state.json] for collection [cdcr-source] has occurred - updating... (live nodes size: [1])
[junit4] 2> 973248 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: cdcr-source failOnTimeout:true timeout (sec):330
[junit4] 1> -
[junit4] 1> replica:core_node1 rstate:active live:true
[junit4] 1> no one is recoverying
[junit4] 2> 973249 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: cdcr-source
[junit4] 2> 973249 ERROR (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.s.i.CloudSolrClient Request to collection cdcr-source failed due to (510) org.apache.solr.common.SolrException: Could not find a healthy node to handle the request., retry? 0
[junit4] 2> 973250 WARN (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.s.i.CloudSolrClient Re-trying request to collection(s) cdcr-source after stale state error from server.
[junit4] 2> 973251 INFO (qtp2120512517-12260) [ ] o.a.s.c.S.Request [cdcr-source_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&_stateVer_=cdcr-source:7&wt=javabin&version=2} hits=1000 status=0 QTime=0
[junit4] 2> 973269 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 973269 INFO (TEST-CdcrBootstrapTest.testConvertClusterToCdcrAndBootstrap-seed#[8DC1F601C865E878]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37551/solr ready
[junit4] 2> 973271 INFO (qtp1003767237-12160) [ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=cdcr-target&name=cdcr-target&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 973280 INFO (OverseerThreadFactory-3588-thread-1) [ ] o.a.s.c.CreateCollectionCmd Create collection cdcr-target
[junit4] 2> 973395 INFO (OverseerStateUpdate-98540612129259523-127.0.0.1:41615_solr-n_0000000000) [ ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"cdcr-target",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"cdcr-target_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:41615/solr",
[junit4] 2> "type":"NRT"}
[junit4] 2> 973597 INFO (qtp1003767237-12166) [ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node1&collection.configName=cdcr-target&newCollection=true&name=cdcr-target_shard1_replica_n1&action=CREATE&numShards=1&collection=cdcr-target&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 973597 INFO (qtp1003767237-12166) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 973709 INFO (zkCallback-1096-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-target/state.json] for collection [cdcr-target] has occurred - updating... (live nodes size: [1])
[junit4] 2> 973709 INFO (zkCallback-1096-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-target/state.json] for collection [cdcr-target] has occurred - updating... (live nodes size: [1])
[junit4] 2> 974612 INFO (qtp1003767237-12166) [ ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.0.0
[junit4] 2> 974614 INFO (qtp1003767237-12166) [ ] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema not found - loading non-managed schema schema.xml instead
[junit4] 2> 974616 INFO (qtp1003767237-12166) [ ] o.a.s.s.IndexSchema [cdcr-target_shard1_replica_n1] Schema name=minimal
[junit4] 2> 974617 INFO (qtp1003767237-12166) [ ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 974617 INFO (qtp1003767237-12166) [ ] o.a.s.s.ManagedIndexSchema Created and persisted managed schema znode at /configs/cdcr-target/managed-schema
[junit4] 2> 974618 INFO (qtp1003767237-12166) [ ] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema in ZooKeeper, renamed the non-managed schema /configs/cdcr-target/schema.xml to /configs/cdcr-target/schema.xml.bak
[junit4] 2> 974618 INFO (qtp1003767237-12166) [ ] o.a.s.c.CoreContainer Creating SolrCore 'cdcr-target_shard1_replica_n1' using configuration from collection cdcr-target, trusted=true
[junit4] 2> 974618 INFO (qtp1003767237-12166) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41615.solr.core.cdcr-target.shard1.replica_n1' (registry 'solr.core.cdcr-target.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@47b7d3bc
[junit4] 2> 974619 INFO (qtp1003767237-12166) [ ] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 974619 INFO (qtp1003767237-12166) [ ] o.a.s.c.SolrCore [[cdcr-target_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-7.0-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_8DC1F601C865E878-001/cdcr-target-001/node1/cdcr-target_shard1_replica_n1], dataDir=[/home/jenkins/workspace/Lucene-Solr-7.0-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.CdcrBootstrapTest_8DC1F601C865E878-001/cdcr-target-001/node1/./cdcr-target_shard1_replica_n1/data/]
[junit4] 2> 974651 INFO (qtp1003767237-12166) [ ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 974651 INFO (qtp1003767237-12166) [ ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 974652 INFO (qtp1003767237-12166) [ ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 974652 INFO (qtp1003767237-12166) [ ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 974653 INFO (qtp1003767237-12166) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@116f1e87[cdcr-target_shard1_replica_n1] main]
[junit4] 2> 974654 INFO (qtp1003767237-12166) [ ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/cdcr-target
[junit4] 2> 974654 INFO (qtp1003767237-12166) [ ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/cdcr-target
[junit4] 2> 974655 INFO (qtp1003767237-12166) [ ] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/cdcr-target/managed-schema
[junit4] 2> 974655 INFO (qtp1003767237-12166) [ ] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
[junit4] 2> 974656 INFO (qtp1003767237-12166) [ ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 974658 INFO (qtp1003767237-12166) [ ] o.a.s.h.CdcrBufferStateManager Created znode /collections/cdcr-target/cdcr/state/buffer
[junit4] 2> 974664 INFO (qtp1003767237-12166) [ ] o.a.s.h.CdcrProcessStateManager Created znode /collections/cdcr-target/cdcr/state/process
[junit4] 2> 974665 INFO (searcherExecutor-3620-thread-1) [ ] o.a.s.c.SolrCore [cdcr-target_shard1_replica_n1] Registered new searcher Searcher@116f1e87[cdcr-target_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 974665 INFO (qtp1003767237-12166) [ ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1576649804008652800
[junit4] 2> 974669 INFO (qtp1003767237-12166) [ ] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 974669 INFO (qtp1003767237-12166) [ ] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 974669 INFO (qtp1003767237-12166) [ ] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:41615/solr/cdcr-target_shard1_replica_n1/
[junit4] 2> 974669 INFO (qtp1003767237-12166) [ ] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 974669 INFO (qtp1003767237-12166) [ ] o.a.s.c.SyncStrategy http://127.0.0.1:41615/solr/cdcr-target_shard1_replica_n1/ has no replicas
[junit4] 2> 974669 INFO (qtp1003767237-12166) [ ] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 974670 INFO (zkCallback-1096-thread-2) [ ] o.a.s.h.CdcrLeaderStateManager Received new leader state @ cdcr-target:shard1
[junit4] 2> 974674 INFO (qtp1003767237-12166) [ ] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:41615/solr/cdcr-target_shard1_replica_n1/ shard1
[junit4] 2> 974776 INFO (zkCallback-1096-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-target/state.json] for collection [cdcr-target] has occurred - updating... (live nodes size: [1])
[junit4] 2> 974776 INFO (zkCallback-1096-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-target/state.json] for collection [cdcr-target] has occurred - updating... (live nodes size: [1])
[junit4] 2> 974825 INFO (qtp1003767237-12166) [ ] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 974826 INFO (qtp1003767237-12166) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node1&collection.configName=cdcr-target&newCollection=true&name=cdcr-target_shard1_replica_n1&action=CREATE&numShards=1&collection=cdcr-target&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1228
[junit4] 2> 974827 INFO (qtp1003767237-12160) [ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 974927 INFO (zkCallback-1096-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-target/state.json] for collection [cdcr-target] has occurred - updating... (live nodes size: [1])
[junit4] 2> 974927 INFO (zkCallback-1096-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-target/state.json] for collection [cdcr-target] has occurred - updating... (live nodes size: [1])
[junit4] 2> 975284 INFO (OverseerCollectionConfigSetProcessor-98540612129259523-127.0.0.1:41615_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 975827 INFO (qtp1003767237-12160) [ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=cdcr-target&name=cdcr-target&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=2556
[junit4] 2> 976864 INFO (zkCallback-1096-thread-1) [ ] o.a.s.h.CdcrProcessStateManager The CDCR process state has changed: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-target/cdcr/state/process @ cdcr-target:shard1
[junit4] 2> 976865 INFO (qtp1003767237-12165) [ ] o.a.s.c.S.Request [cdcr-target_shard1_replica_n1] webapp=/solr path=/cdcr params={qt=/cdcr&_stateVer_=cdcr-target:4&action=start&wt=javabin&version=2} status=0 QTime=36
[junit4] 2> 976865 INFO (zkCallback-1096-thread-1) [ ] o.a.s.h.CdcrProcessStateManager Received new CDCR process state from watcher: STARTED @ cdcr-target:shard1
[junit4] 2> 976886 INFO (qtp2120512517-12261) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 976886 INFO (qtp2120512517-12261) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:37551/solr ready
[junit4] 2> 976922 INFO (qtp1003767237-12163) [ ] o.a.s.h.CdcrRequestHandler Found maxVersionFromRecent 0 maxVersionFromIndex 0
[junit4] 2> 976923 INFO (qtp1003767237-12163) [ ] o.a.s.c.S.Request [cdcr-target_shard1_replica_n1] webapp=/solr path=/cdcr params={action=SHARDCHECKPOINT&wt=javabin&version=2} status=0 QTime=2
[junit4] 2> 976923 INFO (qtp1003767237-12166) [ ] o.a.s.c.S.Request [cdcr-target_shard1_replica_n1] webapp=/solr path=/cdcr params={_stateVer_=cdcr-target:4&action=COLLECTIONCHECKPOINT&wt=javabin&version=2} status=0 QTime=35
[junit4] 2> 976924 INFO (qtp2120512517-12261) [ ] o.a.s.h.CdcrReplicatorManager Create new update log reader for target cdcr-target with checkpoint -1 @ cdcr-source:shard1
[junit4] 2> 976924 INFO (qtp2120512517-12261) [ ] o.a.s.h.CdcrReplicatorManager Attempting to bootstrap target collection: cdcr-target, shard: shard1
[junit4] 2> 976926 INFO (qtp2120512517-12261) [ ] o.a.s.h.CdcrReplicatorManager Submitting bootstrap task to executor
[junit4] 2> 976932 INFO (cdcr-bootstrap-status-1127-thread-1) [ ] o.a.s.h.CdcrReplicatorManager Attempting to bootstrap target collection: cdcr-target shard: shard1 leader: http://127.0.0.1:41615/solr/cdcr-target_shard1_replica_n1/
[junit4] 2> 976951 INFO (zkCallback-1121-thread-2) [ ] o.a.s.h.CdcrProcessStateManager The CDCR process state has changed: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-source/cdcr/state/process @ cdcr-source:shard1
[junit4] 2> 976951 INFO (qtp2120512517-12261) [ ] o.a.s.c.S.Request [cdcr-source_shard1_replica_n1] webapp=/solr path=/cdcr params={qt=/cdcr&_stateVer_=cdcr-source:7&action=start&wt=javabin&version=2} status=0 QTime=84
[junit4] 2> 976952 INFO (zkCallback-1121-thread-2) [ ] o.a.s.h.CdcrProcessStateManager Received new CDCR process state from watcher: STARTED @ cdcr-source:shard1
[junit4] 2> 976963 INFO (qtp1003767237-12160) [ ] o.a.s.c.S.Request [cdcr-target_shard1_replica_n1] webapp=/solr path=/cdcr params={qt=/cdcr&masterUrl=http://127.0.0.1:41301/solr/cdcr-source_shard1_replica_n1/&action=BOOTSTRAP&wt=javabin&version=2} status=0 QTime=21
[junit4] 2> 976964 INFO (qtp2120512517-12255) [ ] o.a.s.c.S.Request [cdcr-source_shard1_replica_n1] webapp=/solr path=/cdcr params={qt=/cdcr&_stateVer_=cdcr-source:7&action=queues&wt=javabin&version=2} status=0 QTime=12
[junit4] 1> Cdcr queue response: {responseHeader={status=0,QTime=12},queues={127.0.0.1:37551/solr={cdcr-target={queueSize=-932184065,lastTimestamp=}}},tlogTotalSize=4773,tlogTotalCount=1,updateLogSynchronizer=stopped}
[junit4] 2> 976966 INFO (qtp1003767237-12161) [ ] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1576649806421426176,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 976966 INFO (qtp1003767237-12161) [ ] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 976966 INFO (qtp1003767237-12163) [ ] o.a.s.c.S.Request [cdcr-target_shard1_replica_n1] webapp=/solr path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 976966 WARN (cdcr-bootstrap-status-1127-thread-1) [ ] o.a.s.h.CdcrReplicatorManager Bootstrap process was not found on target collection: cdcr-target shard: shard1, leader: http://127.0.0.1:41615/solr/cdcr-target_shard1_replica_n1/
[junit4] 2> 976966 INFO (cdcr-bootstrap-status-1127-thread-1) [ ] o.a.s.h.CdcrReplicatorManager Attempting to bootstrap target collection: cdcr-target shard: shard1 leader: http://127.0.0.1:41615/solr/cdcr-target_shard1_replica_n1/
[junit4] 2> 976967 INFO (qtp1003767237-12163) [ ] o.a.s.c.S.Request [cdcr-target_shard1_replica_n1] webapp=/solr path=/cdcr params={qt=/cdcr&masterUrl=http://127.0.0.1:41301/solr/cdcr-source_shard1_replica_n1/&action=BOOTSTRAP&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 976967 INFO (qtp1003767237-12160) [ ] o.a.s.c.S.Request [cdcr-target_shard1_replica_n1] webapp=/solr path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 976968 INFO (cdcr-bootstrap-status-1127-thread-1) [ ] o.a.s.h.CdcrReplicatorManager CDCR bootstrap running for 1 seconds, sleeping for 2000 ms
[junit4] 2> 976978 INFO (qtp1003767237-12161) [ ] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 976978 INFO (qtp1003767237-12161) [ ] o.a.s.c.S.Request [cdcr-target_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-target:4&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2} status=0 QTime=12
[junit4] 2> 976978 INFO (qtp1003767237-12162) [ ] o.a.s.c.S.Request [cdcr-target_shard1_replica_n1] webapp=/solr path=/select params={q=*:*&_stateVer_=cdcr-target:4&wt=javabin&version=2} hits=0 status=0 QTime=0
[junit4] 2> 976979 INFO (recoveryExecutor-1094-thread-1) [ ] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 976984 INFO (qtp2120512517-12260)
[...truncated too long message...]
Impl.checkConnect(Native Method)
[junit4] 2> at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
[junit4] 2> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
[junit4] 2> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[junit4] 2> Aug 24, 2017 9:45:29 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> SEVERE: There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=15008, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[8DC1F601C865E878]-SendThread(127.0.0.1:39369), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
[junit4] 2> at java.base@9/java.lang.Thread.sleep(Native Method)
[junit4] 2> at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {rnd_b=Lucene50(blocksize=128), a_t=PostingsFormat(name=LuceneVarGapFixedInterval), id=Lucene50(blocksize=128)}, docValues:{rnd_b=DocValuesFormat(name=Lucene70), _version_=DocValuesFormat(name=Direct), a_t=DocValuesFormat(name=Direct), a_i=DocValuesFormat(name=Lucene70), id=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=473, maxMBSortInHeap=7.929576306989999, sim=RandomSimilarity(queryNorm=true): {}, locale=seh, timezone=Pacific/Guadalcanal
[junit4] 2> NOTE: Linux 4.10.0-27-generic amd64/Oracle Corporation 9 (64-bit)/cpus=8,threads=2,free=400190568,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [TestExtendedDismaxParser, TestLazyCores, TestLRUStatsCache, TestFastOutputStream, StatsReloadRaceTest, TestExportWriter, TestPhraseSuggestions, DistributedExpandComponentTest, VMParamsZkACLAndCredentialsProvidersTest, ConcurrentDeleteAndCreateCollectionTest, TestPHPSerializedResponseWriter, ChaosMonkeySafeLeaderWithPullReplicasTest, CurrencyFieldTypeTest, TestFieldCacheSortRandom, TestSolrDeletionPolicy1, UpdateLogTest, TestUnifiedSolrHighlighter, TestSimpleQParserPlugin, LeaderFailureAfterFreshStartTest, HdfsSyncSliceTest, TestCloudManagedSchema, TestRealTimeGet, TestManagedSchemaAPI, SpellCheckCollatorTest, TestHighlightDedupGrouping, IndexBasedSpellCheckerTest, CdcrRequestHandlerTest, MoveReplicaHDFSUlogDirTest, TestReplicationHandler, BlobRepositoryCloudTest, HttpSolrCallGetCoreTest, DistributedQueryElevationComponentTest, XmlUpdateRequestHandlerTest, ExitableDirectoryReaderTest, RankQueryTest, TestRangeQuery, BasicZkTest, TestBadConfig, RecoveryAfterSoftCommitTest, TestSolrCloudWithSecureImpersonation, TestLegacyFieldCache, TestSolrQueryParser, TestPayloadCheckQParserPlugin, BlockCacheTest, TestConfigSetsAPI, SynonymTokenizerTest, SolrRequestParserTest, TestCloudRecovery, TestCrossCoreJoin, CreateCollectionCleanupTest, TestBackupRepositoryFactory, SolrGraphiteReporterTest, TestConfigsApi, TestFaceting, TestDocumentBuilder, SolrIndexMetricsTest, TestSolrCoreParser, OutputWriterTest, TestBulkSchemaAPI, BadCopyFieldTest, MetricsConfigTest, TestBinaryField, TestJavabinTupleStreamParser, TestHdfsUpdateLog, TestGraphTermsQParserPlugin, PeerSyncTest, TestOmitPositions, CoreMergeIndexesAdminHandlerTest, TestSolrCoreSnapshots, TestRecoveryHdfs, HdfsChaosMonkeySafeLeaderTest, TestRTimerTree, TestMinMaxOnMultiValuedField, TestCustomSort, TestRandomDVFaceting, SpellingQueryConverterTest, EnumFieldTest, RequestHandlersTest, TestDistribIDF, TestNestedDocsSort, TestCustomStream, TestCloudJSONFacetJoinDomain, TestDownShardTolerantSearch, SolrCLIZkUtilsTest, PluginInfoTest, HdfsRecoverLeaseTest, TestRuleBasedAuthorizationPlugin, TestFieldCache, TestFunctionQuery, TestStressLucene, MigrateRouteKeyTest, TestUninvertingReader, DistributedFacetPivotLongTailTest, TestImpersonationWithHadoopAuth, SolrPluginUtilsTest, DistributedSuggestComponentTest, TestIBSimilarityFactory, WordBreakSolrSpellCheckerTest, TestConfigSetProperties, CdcrBootstrapTest, TestSSLRandomization, DeleteShardTest, TestWriterPerf, TestComponentsName, ConfigureRecoveryStrategyTest, TestIndexingPerformance, UUIDFieldTest, CloudExitableDirectoryReaderTest, BlockJoinFacetSimpleTest, UtilsToolTest, TestWordDelimiterFilterFactory, TestEmbeddedSolrServerConstructors, SmileWriterTest, DocExpirationUpdateProcessorFactoryTest, FullHLLTest, TestDistributedSearch, TestCollapseQParserPlugin, BJQParserTest, BigEndianAscendingWordSerializerTest, ExternalFileFieldSortTest, TestManagedStopFilterFactory, TestRecovery, DistributedSpellCheckComponentTest, TestStressReorder, TestJoin, SpellCheckComponentTest, BadIndexSchemaTest, ShowFileRequestHandlerTest, SolrIndexSplitterTest, AnalysisAfterCoreReloadTest, TestTrie, SuggesterWFSTTest, PolyFieldTest, TestUpdate, FieldMutatingUpdateProcessorTest, DirectUpdateHandlerOptimizeTest, StatelessScriptUpdateProcessorFactoryTest, DistanceFunctionTest, DebugComponentTest, DocumentAnalysisRequestHandlerTest, TestValueSourceCache, TermVectorComponentTest, RequiredFieldsTest, FastVectorHighlighterTest, TestJmxIntegration, TestCSVResponseWriter, TestAnalyzedSuggestions, JsonLoaderTest, TestPartialUpdateDeduplication, CSVRequestHandlerTest, SearchHandlerTest, UpdateParamsTest, CopyFieldTest, BadComponentTest, MultiTermTest, SampleTest, NumericFieldsTest, TestElisionMultitermQuery, TestConfig, TestLuceneMatchVersion, EchoParamsTest, TestSweetSpotSimilarityFactory, TestPerFieldSimilarity, TestLMDirichletSimilarityFactory, TimeZoneUtilsTest, ScriptEngineTest, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest, AssignTest, AsyncCallRequestStatusResponseTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ChaosMonkeyNothingIsSafeWithPullReplicasTest -Dtests.seed=8DC1F601C865E878 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=seh -Dtests.timezone=Pacific/Guadalcanal -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J0 | ChaosMonkeyNothingIsSafeWithPullReplicasTest (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest:
[junit4] > 1) Thread[id=15009, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[8DC1F601C865E878]-EventThread, state=WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
[junit4] > at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
[junit4] > at java.base@9/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
[junit4] > at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2062)
[junit4] > at java.base@9/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
[junit4] > at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
[junit4] > 2) Thread[id=15007, name=Connection evictor, state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
[junit4] > at java.base@9/java.lang.Thread.sleep(Native Method)
[junit4] > at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
[junit4] > at java.base@9/java.lang.Thread.run(Thread.java:844)
[junit4] > 3) Thread[id=15008, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[8DC1F601C865E878]-SendThread(127.0.0.1:39369), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
[junit4] > at java.base@9/java.lang.Thread.sleep(Native Method)
[junit4] > at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
[junit4] > at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
[junit4] > at __randomizedtesting.SeedInfo.seed([8DC1F601C865E878]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=15008, name=TEST-ChaosMonkeyNothingIsSafeWithPullReplicasTest.test-seed#[8DC1F601C865E878]-SendThread(127.0.0.1:39369), state=TIMED_WAITING, group=TGRP-ChaosMonkeyNothingIsSafeWithPullReplicasTest]
[junit4] > at java.base@9/java.lang.Thread.sleep(Native Method)
[junit4] > at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1051)
[junit4] > at __randomizedtesting.SeedInfo.seed([8DC1F601C865E878]:0)
[junit4] Completed [518/729 (2!)] on J0 in 252.67s, 1 test, 1 failure, 2 errors <<< FAILURES!
[...truncated 704 lines...]
[junit4] JVM J0: stdout was not empty, see: /home/jenkins/workspace/Lucene-Solr-7.0-Linux/solr/build/solr-core/test/temp/junit4-J0-20170824_211727_47111827124921430272764.sysout
[junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
[junit4] java.lang.OutOfMemoryError: Java heap space
[junit4] Dumping heap to /home/jenkins/workspace/Lucene-Solr-7.0-Linux/heapdumps/java_pid23742.hprof ...
[junit4] Heap dump file created [357277657 bytes in 0.604 secs]
[junit4] <<< JVM J0: EOF ----
[...truncated 7537 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-7.0-Linux/build.xml:810: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-7.0-Linux/build.xml:762: Some of the tests produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? Dumps created:
* java_pid23742.hprof
Total time: 57 minutes 20 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any