You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2018/09/12 08:57:52 UTC
[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 813 -
Still Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/813/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC
2 tests failed.
FAILED: org.apache.solr.cloud.cdcr.CdcrBidirectionalTest.testBiDir
Error Message:
Captured an uncaught exception in thread: Thread[id=715, name=cdcr-replicator-201-thread-1, state=RUNNABLE, group=TGRP-CdcrBidirectionalTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=715, name=cdcr-replicator-201-thread-1, state=RUNNABLE, group=TGRP-CdcrBidirectionalTest]
Caused by: java.lang.AssertionError: 1611391313648812032 != 1611391313647763456
at __randomizedtesting.SeedInfo.seed([7E5970A0C6AFE24E]:0)
at org.apache.solr.update.CdcrUpdateLog$CdcrLogReader.forwardSeek(CdcrUpdateLog.java:611)
at org.apache.solr.handler.CdcrReplicator.run(CdcrReplicator.java:125)
at org.apache.solr.handler.CdcrReplicatorScheduler.lambda$null$0(CdcrReplicatorScheduler.java:81)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.cdcr.CdcrBidirectionalTest.testBiDir
Error Message:
Captured an uncaught exception in thread: Thread[id=13041, name=cdcr-replicator-6186-thread-1, state=RUNNABLE, group=TGRP-CdcrBidirectionalTest]
Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=13041, name=cdcr-replicator-6186-thread-1, state=RUNNABLE, group=TGRP-CdcrBidirectionalTest]
Caused by: java.lang.AssertionError: 1611386873032212480 != 1611386872339103744
at __randomizedtesting.SeedInfo.seed([7E5970A0C6AFE24E]:0)
at org.apache.solr.update.CdcrUpdateLog$CdcrLogReader.forwardSeek(CdcrUpdateLog.java:611)
at org.apache.solr.handler.CdcrReplicator.run(CdcrReplicator.java:125)
at org.apache.solr.handler.CdcrReplicatorScheduler.lambda$null$0(CdcrReplicatorScheduler.java:81)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 13366 lines...]
[junit4] Suite: org.apache.solr.cloud.cdcr.CdcrBidirectionalTest
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.cdcr.CdcrBidirectionalTest_7E5970A0C6AFE24E-001/init-core-data-001
[junit4] 2> 1430649 INFO (SUITE-CdcrBidirectionalTest-seed#[7E5970A0C6AFE24E]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 1430650 INFO (SUITE-CdcrBidirectionalTest-seed#[7E5970A0C6AFE24E]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 1430650 INFO (SUITE-CdcrBidirectionalTest-seed#[7E5970A0C6AFE24E]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 1430651 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testBiDir
[junit4] 2> 1430652 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.cdcr.CdcrBidirectionalTest_7E5970A0C6AFE24E-001/cdcr-cluster2-001
[junit4] 2> 1430652 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1430652 INFO (Thread-2782) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1430652 INFO (Thread-2782) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1430654 ERROR (Thread-2782) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1430752 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.c.ZkTestServer start zk server on port:49671
[junit4] 2> 1430754 INFO (zkConnectionManagerCallback-3365-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1430758 INFO (jetty-launcher-3362-thread-1) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 1430759 INFO (jetty-launcher-3362-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1430759 INFO (jetty-launcher-3362-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1430760 INFO (jetty-launcher-3362-thread-1) [ ] o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 1430760 INFO (jetty-launcher-3362-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7141ab42{/solr,null,AVAILABLE}
[junit4] 2> 1430763 INFO (jetty-launcher-3362-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@692cd7fb{HTTP/1.1,[http/1.1]}{127.0.0.1:34429}
[junit4] 2> 1430763 INFO (jetty-launcher-3362-thread-1) [ ] o.e.j.s.Server Started @1431460ms
[junit4] 2> 1430763 INFO (jetty-launcher-3362-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34429}
[junit4] 2> 1430763 ERROR (jetty-launcher-3362-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1430763 INFO (jetty-launcher-3362-thread-1) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 1430763 INFO (jetty-launcher-3362-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.6.0
[junit4] 2> 1430763 INFO (jetty-launcher-3362-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1430763 INFO (jetty-launcher-3362-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1430763 INFO (jetty-launcher-3362-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-12T07:44:09.245Z
[junit4] 2> 1430765 INFO (zkConnectionManagerCallback-3367-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1430765 INFO (jetty-launcher-3362-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1431875 INFO (jetty-launcher-3362-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49671/solr
[junit4] 2> 1431876 INFO (zkConnectionManagerCallback-3371-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1431880 INFO (zkConnectionManagerCallback-3373-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1432013 INFO (jetty-launcher-3362-thread-1) [n:127.0.0.1:34429_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34429_solr
[junit4] 2> 1432014 INFO (jetty-launcher-3362-thread-1) [n:127.0.0.1:34429_solr ] o.a.s.c.Overseer Overseer (id=72322430259232771-127.0.0.1:34429_solr-n_0000000000) starting
[junit4] 2> 1432025 INFO (zkConnectionManagerCallback-3380-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1432029 INFO (jetty-launcher-3362-thread-1) [n:127.0.0.1:34429_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:49671/solr ready
[junit4] 2> 1432030 INFO (jetty-launcher-3362-thread-1) [n:127.0.0.1:34429_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34429_solr
[junit4] 2> 1432030 INFO (OverseerStateUpdate-72322430259232771-127.0.0.1:34429_solr-n_0000000000) [n:127.0.0.1:34429_solr ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:34429_solr
[junit4] 2> 1432041 INFO (OverseerStateUpdate-72322430259232771-127.0.0.1:34429_solr-n_0000000000) [n:127.0.0.1:34429_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1432041 INFO (zkCallback-3379-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1432049 INFO (jetty-launcher-3362-thread-1) [n:127.0.0.1:34429_solr ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 1432073 INFO (jetty-launcher-3362-thread-1) [n:127.0.0.1:34429_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34429.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7e2aa632
[junit4] 2> 1432078 INFO (jetty-launcher-3362-thread-1) [n:127.0.0.1:34429_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34429.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7e2aa632
[junit4] 2> 1432078 INFO (jetty-launcher-3362-thread-1) [n:127.0.0.1:34429_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34429.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7e2aa632
[junit4] 2> 1432078 INFO (jetty-launcher-3362-thread-1) [n:127.0.0.1:34429_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.cdcr.CdcrBidirectionalTest_7E5970A0C6AFE24E-001/cdcr-cluster2-001/node1/.
[junit4] 2> 1432104 INFO (zkConnectionManagerCallback-3383-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1432109 INFO (zkConnectionManagerCallback-3386-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1432111 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.cdcr.CdcrBidirectionalTest_7E5970A0C6AFE24E-001/cdcr-cluster1-001
[junit4] 2> 1432111 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1432111 INFO (Thread-2792) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1432111 INFO (Thread-2792) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1432114 ERROR (Thread-2792) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1432211 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.c.ZkTestServer start zk server on port:58986
[junit4] 2> 1432214 INFO (zkConnectionManagerCallback-3390-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1432219 INFO (jetty-launcher-3387-thread-1) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 1432220 INFO (jetty-launcher-3387-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1432220 INFO (jetty-launcher-3387-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1432220 INFO (jetty-launcher-3387-thread-1) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 1432221 INFO (jetty-launcher-3387-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@17eb164c{/solr,null,AVAILABLE}
[junit4] 2> 1432221 INFO (jetty-launcher-3387-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@e73a313{HTTP/1.1,[http/1.1]}{127.0.0.1:35341}
[junit4] 2> 1432221 INFO (jetty-launcher-3387-thread-1) [ ] o.e.j.s.Server Started @1432918ms
[junit4] 2> 1432221 INFO (jetty-launcher-3387-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35341}
[junit4] 2> 1432221 ERROR (jetty-launcher-3387-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1432222 INFO (jetty-launcher-3387-thread-1) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 1432222 INFO (jetty-launcher-3387-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.6.0
[junit4] 2> 1432222 INFO (jetty-launcher-3387-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1432222 INFO (jetty-launcher-3387-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1432222 INFO (jetty-launcher-3387-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-12T07:44:10.704Z
[junit4] 2> 1432224 INFO (zkConnectionManagerCallback-3392-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1432225 INFO (jetty-launcher-3387-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1432533 INFO (jetty-launcher-3387-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:58986/solr
[junit4] 2> 1432535 INFO (zkConnectionManagerCallback-3396-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1432538 INFO (zkConnectionManagerCallback-3398-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1432683 INFO (jetty-launcher-3387-thread-1) [n:127.0.0.1:35341_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35341_solr
[junit4] 2> 1432684 INFO (jetty-launcher-3387-thread-1) [n:127.0.0.1:35341_solr ] o.a.s.c.Overseer Overseer (id=72322430354849795-127.0.0.1:35341_solr-n_0000000000) starting
[junit4] 2> 1432700 INFO (zkConnectionManagerCallback-3405-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1432704 INFO (jetty-launcher-3387-thread-1) [n:127.0.0.1:35341_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58986/solr ready
[junit4] 2> 1432705 INFO (jetty-launcher-3387-thread-1) [n:127.0.0.1:35341_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35341_solr
[junit4] 2> 1432706 INFO (OverseerStateUpdate-72322430354849795-127.0.0.1:35341_solr-n_0000000000) [n:127.0.0.1:35341_solr ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:35341_solr
[junit4] 2> 1432708 INFO (zkCallback-3404-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1432713 INFO (OverseerStateUpdate-72322430354849795-127.0.0.1:35341_solr-n_0000000000) [n:127.0.0.1:35341_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1432728 INFO (jetty-launcher-3387-thread-1) [n:127.0.0.1:35341_solr ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 1432742 INFO (jetty-launcher-3387-thread-1) [n:127.0.0.1:35341_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35341.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7e2aa632
[junit4] 2> 1432747 INFO (jetty-launcher-3387-thread-1) [n:127.0.0.1:35341_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35341.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7e2aa632
[junit4] 2> 1432747 INFO (jetty-launcher-3387-thread-1) [n:127.0.0.1:35341_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35341.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7e2aa632
[junit4] 2> 1432748 INFO (jetty-launcher-3387-thread-1) [n:127.0.0.1:35341_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.cdcr.CdcrBidirectionalTest_7E5970A0C6AFE24E-001/cdcr-cluster1-001/node1/.
[junit4] 2> 1432767 INFO (zkConnectionManagerCallback-3408-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1432770 INFO (zkConnectionManagerCallback-3411-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1432772 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.c.c.CdcrBidirectionalTest cluster2 zkHost = 127.0.0.1:49671/solr
[junit4] 2> 1432772 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.c.c.CdcrBidirectionalTest cluster1 zkHost = 127.0.0.1:58986/solr
[junit4] 2> 1432774 INFO (zkConnectionManagerCallback-3413-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1432781 INFO (zkConnectionManagerCallback-3417-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1432783 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1432784 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:58986/solr ready
[junit4] 2> 1432786 INFO (qtp223663899-12903) [n:127.0.0.1:35341_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=cdcr-cluster1&maxShardsPerNode=2&name=cdcr-cluster1&nrtReplicas=1&action=CREATE&numShards=2&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1432790 INFO (OverseerThreadFactory-6147-thread-1-processing-n:127.0.0.1:35341_solr) [n:127.0.0.1:35341_solr ] o.a.s.c.a.c.CreateCollectionCmd Create collection cdcr-cluster1
[junit4] 2> 1432907 INFO (OverseerStateUpdate-72322430354849795-127.0.0.1:35341_solr-n_0000000000) [n:127.0.0.1:35341_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"cdcr-cluster1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"cdcr-cluster1_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:35341/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1432910 INFO (OverseerStateUpdate-72322430354849795-127.0.0.1:35341_solr-n_0000000000) [n:127.0.0.1:35341_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"cdcr-cluster1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"cdcr-cluster1_shard2_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:35341/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1433115 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr x:cdcr-cluster1_shard2_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=cdcr-cluster1&newCollection=true&collection=cdcr-cluster1&version=2&replicaType=NRT&coreNodeName=core_node4&name=cdcr-cluster1_shard2_replica_n2&action=CREATE&numShards=2&shard=shard2&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin
[junit4] 2> 1433115 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1433115 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr x:cdcr-cluster1_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=cdcr-cluster1&newCollection=true&collection=cdcr-cluster1&version=2&replicaType=NRT&coreNodeName=core_node3&name=cdcr-cluster1_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin
[junit4] 2> 1434133 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
[junit4] 2> 1434141 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.s.IndexSchema [cdcr-cluster1_shard1_replica_n1] Schema name=minimal
[junit4] 2> 1434143 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1434143 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'cdcr-cluster1_shard1_replica_n1' using configuration from collection cdcr-cluster1, trusted=true
[junit4] 2> 1434144 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35341.solr.core.cdcr-cluster1.shard1.replica_n1' (registry 'solr.core.cdcr-cluster1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7e2aa632
[junit4] 2> 1434144 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.SolrCore [[cdcr-cluster1_shard1_replica_n1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.cdcr.CdcrBidirectionalTest_7E5970A0C6AFE24E-001/cdcr-cluster1-001/node1/cdcr-cluster1_shard1_replica_n1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.cdcr.CdcrBidirectionalTest_7E5970A0C6AFE24E-001/cdcr-cluster1-001/node1/./cdcr-cluster1_shard1_replica_n1/data/]
[junit4] 2> 1434151 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
[junit4] 2> 1434160 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.s.IndexSchema [cdcr-cluster1_shard2_replica_n2] Schema name=minimal
[junit4] 2> 1434160 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1434160 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'cdcr-cluster1_shard2_replica_n2' using configuration from collection cdcr-cluster1, trusted=true
[junit4] 2> 1434160 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35341.solr.core.cdcr-cluster1.shard2.replica_n2' (registry 'solr.core.cdcr-cluster1.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7e2aa632
[junit4] 2> 1434160 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.SolrCore [[cdcr-cluster1_shard2_replica_n2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.cdcr.CdcrBidirectionalTest_7E5970A0C6AFE24E-001/cdcr-cluster1-001/node1/cdcr-cluster1_shard2_replica_n2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.cdcr.CdcrBidirectionalTest_7E5970A0C6AFE24E-001/cdcr-cluster1-001/node1/./cdcr-cluster1_shard2_replica_n2/data/]
[junit4] 2> 1434251 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 1434251 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1434253 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1434253 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1434255 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6022c4c6[cdcr-cluster1_shard1_replica_n1] main]
[junit4] 2> 1434256 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/cdcr-cluster1
[junit4] 2> 1434257 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/cdcr-cluster1
[junit4] 2> 1434257 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/cdcr-cluster1/managed-schema
[junit4] 2> 1434257 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
[junit4] 2> 1434258 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1434265 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 1434266 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1434267 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.h.CdcrBufferStateManager Created znode /collections/cdcr-cluster1/cdcr/state/buffer
[junit4] 2> 1434267 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1434267 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1434269 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@332601a7[cdcr-cluster1_shard2_replica_n2] main]
[junit4] 2> 1434270 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/cdcr-cluster1
[junit4] 2> 1434271 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/cdcr-cluster1
[junit4] 2> 1434271 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/cdcr-cluster1/managed-schema
[junit4] 2> 1434271 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
[junit4] 2> 1434272 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1434275 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.h.CdcrProcessStateManager Created znode /collections/cdcr-cluster1/cdcr/state/process
[junit4] 2> 1434310 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1611386850159624192
[junit4] 2> 1434310 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1611386850159624192
[junit4] 2> 1434312 INFO (searcherExecutor-6152-thread-1-processing-n:127.0.0.1:35341_solr x:cdcr-cluster1_shard1_replica_n1 c:cdcr-cluster1 s:shard1 r:core_node3) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.SolrCore [cdcr-cluster1_shard1_replica_n1] Registered new searcher Searcher@6022c4c6[cdcr-cluster1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1434313 INFO (searcherExecutor-6153-thread-1-processing-n:127.0.0.1:35341_solr x:cdcr-cluster1_shard2_replica_n2 c:cdcr-cluster1 s:shard2 r:core_node4) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.SolrCore [cdcr-cluster1_shard2_replica_n2] Registered new searcher Searcher@332601a7[cdcr-cluster1_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1434317 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/cdcr-cluster1/terms/shard1 to Terms{values={core_node3=0}, version=0}
[junit4] 2> 1434323 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/cdcr-cluster1/terms/shard2 to Terms{values={core_node4=0}, version=0}
[junit4] 2> 1434324 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1434325 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1434325 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:35341/solr/cdcr-cluster1_shard1_replica_n1/
[junit4] 2> 1434325 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1434325 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:35341/solr/cdcr-cluster1_shard1_replica_n1/ has no replicas
[junit4] 2> 1434325 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1434334 INFO (zkCallback-3397-thread-1) [ ] o.a.s.h.CdcrLeaderStateManager Received new leader state @ cdcr-cluster1:shard1
[junit4] 2> 1434334 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1434334 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1434334 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:35341/solr/cdcr-cluster1_shard2_replica_n2/
[junit4] 2> 1434335 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1434335 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.SyncStrategy http://127.0.0.1:35341/solr/cdcr-cluster1_shard2_replica_n2/ has no replicas
[junit4] 2> 1434335 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1434339 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:35341/solr/cdcr-cluster1_shard2_replica_n2/ shard2
[junit4] 2> 1434352 INFO (zkCallback-3397-thread-2) [ ] o.a.s.h.CdcrLeaderStateManager Received new leader state @ cdcr-cluster1:shard2
[junit4] 2> 1434358 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:35341/solr/cdcr-cluster1_shard1_replica_n1/ shard1
[junit4] 2> 1434494 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1434496 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=cdcr-cluster1&newCollection=true&collection=cdcr-cluster1&version=2&replicaType=NRT&coreNodeName=core_node4&name=cdcr-cluster1_shard2_replica_n2&action=CREATE&numShards=2&shard=shard2&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin} status=0 QTime=1383
[junit4] 2> 1434511 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1434514 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=cdcr-cluster1&newCollection=true&collection=cdcr-cluster1&version=2&replicaType=NRT&coreNodeName=core_node3&name=cdcr-cluster1_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin} status=0 QTime=1401
[junit4] 2> 1434518 INFO (qtp223663899-12903) [n:127.0.0.1:35341_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 1434614 INFO (zkCallback-3397-thread-3) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-cluster1/state.json] for collection [cdcr-cluster1] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1434792 INFO (OverseerCollectionConfigSetProcessor-72322430354849795-127.0.0.1:35341_solr-n_0000000000) [n:127.0.0.1:35341_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1435518 INFO (qtp223663899-12903) [n:127.0.0.1:35341_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=cdcr-cluster1&maxShardsPerNode=2&name=cdcr-cluster1&nrtReplicas=1&action=CREATE&numShards=2&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin&version=2} status=0 QTime=2733
[junit4] 2> 1435520 INFO (zkConnectionManagerCallback-3421-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1435526 INFO (zkConnectionManagerCallback-3425-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1435527 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1435528 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:49671/solr ready
[junit4] 2> 1435530 INFO (qtp2126542339-12849) [n:127.0.0.1:34429_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=cdcr-cluster2&maxShardsPerNode=2&name=cdcr-cluster2&nrtReplicas=1&action=CREATE&numShards=2&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1435535 INFO (OverseerThreadFactory-6134-thread-1-processing-n:127.0.0.1:34429_solr) [n:127.0.0.1:34429_solr ] o.a.s.c.a.c.CreateCollectionCmd Create collection cdcr-cluster2
[junit4] 2> 1435648 INFO (OverseerStateUpdate-72322430259232771-127.0.0.1:34429_solr-n_0000000000) [n:127.0.0.1:34429_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"cdcr-cluster2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"cdcr-cluster2_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:34429/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1435649 INFO (OverseerStateUpdate-72322430259232771-127.0.0.1:34429_solr-n_0000000000) [n:127.0.0.1:34429_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"cdcr-cluster2",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"cdcr-cluster2_shard2_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:34429/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1435851 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr x:cdcr-cluster2_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=cdcr-cluster2&newCollection=true&collection=cdcr-cluster2&version=2&replicaType=NRT&coreNodeName=core_node3&name=cdcr-cluster2_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin
[junit4] 2> 1435853 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1435854 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr x:cdcr-cluster2_shard2_replica_n2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=cdcr-cluster2&newCollection=true&collection=cdcr-cluster2&version=2&replicaType=NRT&coreNodeName=core_node4&name=cdcr-cluster2_shard2_replica_n2&action=CREATE&numShards=2&shard=shard2&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin
[junit4] 2> 1436888 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
[junit4] 2> 1436889 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
[junit4] 2> 1436896 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.s.IndexSchema [cdcr-cluster2_shard2_replica_n2] Schema name=minimal
[junit4] 2> 1436897 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.s.IndexSchema [cdcr-cluster2_shard1_replica_n1] Schema name=minimal
[junit4] 2> 1436897 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1436897 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'cdcr-cluster2_shard2_replica_n2' using configuration from collection cdcr-cluster2, trusted=true
[junit4] 2> 1436897 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34429.solr.core.cdcr-cluster2.shard2.replica_n2' (registry 'solr.core.cdcr-cluster2.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7e2aa632
[junit4] 2> 1436897 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.SolrCore [[cdcr-cluster2_shard2_replica_n2] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.cdcr.CdcrBidirectionalTest_7E5970A0C6AFE24E-001/cdcr-cluster2-001/node1/cdcr-cluster2_shard2_replica_n2], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.cdcr.CdcrBidirectionalTest_7E5970A0C6AFE24E-001/cdcr-cluster2-001/node1/./cdcr-cluster2_shard2_replica_n2/data/]
[junit4] 2> 1436899 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1436901 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'cdcr-cluster2_shard1_replica_n1' using configuration from collection cdcr-cluster2, trusted=true
[junit4] 2> 1436902 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_34429.solr.core.cdcr-cluster2.shard1.replica_n1' (registry 'solr.core.cdcr-cluster2.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@7e2aa632
[junit4] 2> 1436902 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.SolrCore [[cdcr-cluster2_shard1_replica_n1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.cdcr.CdcrBidirectionalTest_7E5970A0C6AFE24E-001/cdcr-cluster2-001/node1/cdcr-cluster2_shard1_replica_n1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.cdcr.CdcrBidirectionalTest_7E5970A0C6AFE24E-001/cdcr-cluster2-001/node1/./cdcr-cluster2_shard1_replica_n1/data/]
[junit4] 2> 1436969 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 1436969 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1436972 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.CdcrUpdateLog
[junit4] 2> 1436972 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1436973 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1436973 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1436974 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1436974 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1436974 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@29f4678e[cdcr-cluster2_shard1_replica_n1] main]
[junit4] 2> 1436975 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@4006f7b6[cdcr-cluster2_shard2_replica_n2] main]
[junit4] 2> 1436975 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/cdcr-cluster2
[junit4] 2> 1436976 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/cdcr-cluster2
[junit4] 2> 1436976 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/cdcr-cluster2/managed-schema
[junit4] 2> 1436976 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
[junit4] 2> 1436977 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1436979 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.h.CdcrBufferStateManager Created znode /collections/cdcr-cluster2/cdcr/state/buffer
[junit4] 2> 1436980 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.h.CdcrProcessStateManager Created znode /collections/cdcr-cluster2/cdcr/state/process
[junit4] 2> 1436981 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/cdcr-cluster2
[junit4] 2> 1436982 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1611386852961419264
[junit4] 2> 1436982 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/cdcr-cluster2
[junit4] 2> 1436982 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/cdcr-cluster2/managed-schema
[junit4] 2> 1436982 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
[junit4] 2> 1436982 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1436985 INFO (searcherExecutor-6165-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard1_replica_n1 c:cdcr-cluster2 s:shard1 r:core_node3) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.SolrCore [cdcr-cluster2_shard1_replica_n1] Registered new searcher Searcher@29f4678e[cdcr-cluster2_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1436985 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1611386852964564992
[junit4] 2> 1436987 INFO (searcherExecutor-6164-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard2_replica_n2 c:cdcr-cluster2 s:shard2 r:core_node4) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.SolrCore [cdcr-cluster2_shard2_replica_n2] Registered new searcher Searcher@4006f7b6[cdcr-cluster2_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1436993 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/cdcr-cluster2/terms/shard1 to Terms{values={core_node3=0}, version=0}
[junit4] 2> 1436995 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/cdcr-cluster2/terms/shard2 to Terms{values={core_node4=0}, version=0}
[junit4] 2> 1437001 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1437001 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1437001 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:34429/solr/cdcr-cluster2_shard2_replica_n2/
[junit4] 2> 1437001 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1437001 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.SyncStrategy http://127.0.0.1:34429/solr/cdcr-cluster2_shard2_replica_n2/ has no replicas
[junit4] 2> 1437001 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1437017 INFO (zkCallback-3372-thread-2) [ ] o.a.s.h.CdcrLeaderStateManager Received new leader state @ cdcr-cluster2:shard2
[junit4] 2> 1437017 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1437017 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1437018 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:34429/solr/cdcr-cluster2_shard1_replica_n1/
[junit4] 2> 1437038 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1437039 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:34429/solr/cdcr-cluster2_shard2_replica_n2/ shard2
[junit4] 2> 1437039 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:34429/solr/cdcr-cluster2_shard1_replica_n1/ has no replicas
[junit4] 2> 1437039 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1437043 INFO (zkCallback-3372-thread-2) [ ] o.a.s.h.CdcrLeaderStateManager Received new leader state @ cdcr-cluster2:shard1
[junit4] 2> 1437046 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:34429/solr/cdcr-cluster2_shard1_replica_n1/ shard1
[junit4] 2> 1437150 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1437151 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=cdcr-cluster2&newCollection=true&collection=cdcr-cluster2&version=2&replicaType=NRT&coreNodeName=core_node3&name=cdcr-cluster2_shard1_replica_n1&action=CREATE&numShards=2&shard=shard1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin} status=0 QTime=1300
[junit4] 2> 1437153 INFO (zkCallback-3372-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-cluster2/state.json] for collection [cdcr-cluster2] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1437195 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1437198 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=cdcr-cluster2&newCollection=true&collection=cdcr-cluster2&version=2&replicaType=NRT&coreNodeName=core_node4&name=cdcr-cluster2_shard2_replica_n2&action=CREATE&numShards=2&shard=shard2&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin} status=0 QTime=1345
[junit4] 2> 1437207 INFO (qtp2126542339-12849) [n:127.0.0.1:34429_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 1437300 INFO (zkCallback-3372-thread-3) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-cluster2/state.json] for collection [cdcr-cluster2] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1437301 INFO (zkCallback-3372-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-cluster2/state.json] for collection [cdcr-cluster2] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1437536 INFO (OverseerCollectionConfigSetProcessor-72322430259232771-127.0.0.1:34429_solr-n_0000000000) [n:127.0.0.1:34429_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1438208 INFO (qtp2126542339-12849) [n:127.0.0.1:34429_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=cdcr-cluster2&maxShardsPerNode=2&name=cdcr-cluster2&nrtReplicas=1&action=CREATE&numShards=2&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin&version=2} status=0 QTime=2678
[junit4] 2> 1438224 INFO (zkConnectionManagerCallback-3432-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1438226 INFO (qtp223663899-12906) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1438227 INFO (qtp223663899-12906) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:49671/solr ready
[junit4] 2> 1438236 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.h.CdcrRequestHandler Found maxVersionFromRecent 0 maxVersionFromIndex 0
[junit4] 2> 1438236 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.S.Request [cdcr-cluster2_shard2_replica_n2] webapp=/solr path=/cdcr params={action=SHARDCHECKPOINT&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1438284 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.h.CdcrRequestHandler Found maxVersionFromRecent 0 maxVersionFromIndex 0
[junit4] 2> 1438284 INFO (qtp2126542339-12845) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.S.Request [cdcr-cluster2_shard1_replica_n1] webapp=/solr path=/cdcr params={action=SHARDCHECKPOINT&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1438285 INFO (qtp2126542339-12852) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.S.Request [cdcr-cluster2_shard1_replica_n1] webapp=/solr path=/cdcr params={_stateVer_=cdcr-cluster2:4&action=COLLECTIONCHECKPOINT&wt=javabin&version=2} status=0 QTime=56
[junit4] 2> 1438286 INFO (qtp223663899-12906) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.h.CdcrReplicatorManager Create new update log reader for target cdcr-cluster2 with checkpoint -1 @ cdcr-cluster1:shard1
[junit4] 2> 1438286 INFO (qtp223663899-12906) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.h.CdcrReplicatorManager Attempting to bootstrap target collection: cdcr-cluster2, shard: shard1
[junit4] 2> 1438293 INFO (qtp223663899-12906) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.h.CdcrReplicatorManager Submitting bootstrap task to executor
[junit4] 2> 1438296 INFO (qtp223663899-12906) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.S.Request [cdcr-cluster1_shard1_replica_n1] webapp=/solr path=/cdcr params={qt=/cdcr&_stateVer_=cdcr-cluster1:4&action=start&wt=javabin&version=2} status=0 QTime=76
[junit4] 2> 1438296 INFO (zkCallback-3397-thread-3) [ ] o.a.s.h.CdcrProcessStateManager The CDCR process state has changed: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-cluster1/cdcr/state/process @ cdcr-cluster1:shard2
[junit4] 2> 1438296 INFO (zkCallback-3397-thread-3) [ ] o.a.s.h.CdcrProcessStateManager Received new CDCR process state from watcher: STARTED @ cdcr-cluster1:shard2
[junit4] 2> 1438297 INFO (zkCallback-3397-thread-2) [ ] o.a.s.h.CdcrProcessStateManager The CDCR process state has changed: WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/cdcr-cluster1/cdcr/state/process @ cdcr-cluster1:shard1
[junit4] 2> 1438297 INFO (zkCallback-3397-thread-2) [ ] o.a.s.h.CdcrProcessStateManager Received new CDCR process state from watcher: STARTED @ cdcr-cluster1:shard1
[junit4] 2> 1438298 INFO (zkConnectionManagerCallback-3437-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1438299 INFO (zkCallback-3397-thread-3) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1438300 INFO (zkCallback-3397-thread-3) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:49671/solr ready
[junit4] 2> 1438347 INFO (qtp2126542339-12850) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.h.CdcrRequestHandler Found maxVersionFromRecent 0 maxVersionFromIndex 0
[junit4] 2> 1438347 INFO (qtp2126542339-12850) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.S.Request [cdcr-cluster2_shard2_replica_n2] webapp=/solr path=/cdcr params={action=SHARDCHECKPOINT&wt=javabin&version=2} status=0 QTime=41
[junit4] 2> 1438347 INFO (qtp2126542339-12849) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.h.CdcrRequestHandler Found maxVersionFromRecent 0 maxVersionFromIndex 0
[junit4] 2> 1438347 INFO (qtp2126542339-12849) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.S.Request [cdcr-cluster2_shard1_replica_n1] webapp=/solr path=/cdcr params={action=SHARDCHECKPOINT&wt=javabin&version=2} status=0 QTime=41
[junit4] 2> 1438353 INFO (cdcr-bootstrap-status-3428-thread-1-processing-n:127.0.0.1:35341_solr x:cdcr-cluster1_shard1_replica_n1 c:cdcr-cluster1 s:shard1 r:core_node3) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.h.CdcrReplicatorManager Attempting to bootstrap target collection: cdcr-cluster2 shard: shard1 leader: http://127.0.0.1:34429/solr/cdcr-cluster2_shard1_replica_n1/
[junit4] 2> 1438386 INFO (qtp2126542339-12848) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.S.Request [cdcr-cluster2_shard2_replica_n2] webapp=/solr path=/cdcr params={_stateVer_=cdcr-cluster2:4&action=COLLECTIONCHECKPOINT&wt=javabin&version=2} status=0 QTime=84
[junit4] 2> 1438387 INFO (zkCallback-3397-thread-3) [ ] o.a.s.h.CdcrReplicatorManager Create new update log reader for target cdcr-cluster2 with checkpoint -1 @ cdcr-cluster1:shard2
[junit4] 2> 1438387 INFO (zkCallback-3397-thread-3) [ ] o.a.s.h.CdcrReplicatorManager Attempting to bootstrap target collection: cdcr-cluster2, shard: shard2
[junit4] 2> 1438387 INFO (zkCallback-3397-thread-3) [ ] o.a.s.h.CdcrReplicatorManager Submitting bootstrap task to executor
[junit4] 2> 1438412 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard1_replica_n1 c:cdcr-cluster2 s:shard1 r:core_node3) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 1438418 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1611386854467174400,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1438418 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1438418 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1438418 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.S.Request [cdcr-cluster1_shard1_replica_n1] webapp=/solr path=/update params={waitSearcher=true&openSearcher=false&commit=true&softCommit=false&commit_end_point=true&wt=javabin&version=2} status=0 QTime=2
[junit4] 2> 1438422 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.S.Request [cdcr-cluster1_shard1_replica_n1] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1438422 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard1_replica_n1 c:cdcr-cluster2 s:shard1 r:core_node3) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1438422 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard1_replica_n1 c:cdcr-cluster2 s:shard1 r:core_node3) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1438422 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard1_replica_n1 c:cdcr-cluster2 s:shard1 r:core_node3) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 1438422 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard1_replica_n1 c:cdcr-cluster2 s:shard1 r:core_node3) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 1438422 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard1_replica_n1 c:cdcr-cluster2 s:shard1 r:core_node3) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 1438422 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard1_replica_n1 c:cdcr-cluster2 s:shard1 r:core_node3) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7391a623[cdcr-cluster2_shard1_replica_n1] main]
[junit4] 2> 1438424 INFO (searcherExecutor-6165-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard1_replica_n1 c:cdcr-cluster2 s:shard1 r:core_node3) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.SolrCore [cdcr-cluster2_shard1_replica_n1] Registered new searcher Searcher@7391a623[cdcr-cluster2_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1438425 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard1_replica_n1 c:cdcr-cluster2 s:shard1 r:core_node3) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.h.CdcrRequestHandler No replay needed.
[junit4] 2> 1438425 INFO (qtp2126542339-12849) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.S.Request [cdcr-cluster2_shard1_replica_n1] webapp=/solr path=/cdcr params={qt=/cdcr&masterUrl=http://127.0.0.1:35341/solr/cdcr-cluster1_shard1_replica_n1/&action=BOOTSTRAP&wt=javabin&version=2} status=0 QTime=71
[junit4] 2> 1438429 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.S.Request [cdcr-cluster2_shard1_replica_n1] webapp=/solr path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1438429 INFO (cdcr-bootstrap-status-3428-thread-1-processing-n:127.0.0.1:35341_solr x:cdcr-cluster1_shard1_replica_n1 c:cdcr-cluster1 s:shard1 r:core_node3) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.h.CdcrReplicatorManager CDCR bootstrap successful in 1 seconds
[junit4] 2> 1438430 INFO (cdcr-bootstrap-status-3433-thread-1) [ ] o.a.s.h.CdcrReplicatorManager Attempting to bootstrap target collection: cdcr-cluster2 shard: shard2 leader: http://127.0.0.1:34429/solr/cdcr-cluster2_shard2_replica_n2/
[junit4] 2> 1438430 INFO (qtp2126542339-12848) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.S.Request [cdcr-cluster2_shard2_replica_n2] webapp=/solr path=/cdcr params={qt=/cdcr&masterUrl=http://127.0.0.1:35341/solr/cdcr-cluster1_shard2_replica_n2/&action=BOOTSTRAP&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1438431 INFO (qtp2126542339-12848) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.S.Request [cdcr-cluster2_shard2_replica_n2] webapp=/solr path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1438431 INFO (cdcr-bootstrap-status-3433-thread-1) [ ] o.a.s.h.CdcrReplicatorManager CDCR bootstrap running for 1 seconds, sleeping for 2000 ms
[junit4] 2> 1438431 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard2_replica_n2 c:cdcr-cluster2 s:shard2 r:core_node4) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.u.UpdateLog Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4] 2> 1438433 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1611386854482903040,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1438434 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1438434 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1438434 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.S.Request [cdcr-cluster1_shard2_replica_n2] webapp=/solr path=/update params={waitSearcher=true&openSearcher=false&commit=true&softCommit=false&commit_end_point=true&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 1438435 INFO (qtp223663899-12906) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.S.Request [cdcr-cluster1_shard2_replica_n2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1438436 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard2_replica_n2 c:cdcr-cluster2 s:shard2 r:core_node4) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1438436 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard2_replica_n2 c:cdcr-cluster2 s:shard2 r:core_node4) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1438436 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard2_replica_n2 c:cdcr-cluster2 s:shard2 r:core_node4) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 1438438 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard2_replica_n2 c:cdcr-cluster2 s:shard2 r:core_node4) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 1438438 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard2_replica_n2 c:cdcr-cluster2 s:shard2 r:core_node4) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 1438438 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard2_replica_n2 c:cdcr-cluster2 s:shard2 r:core_node4) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@3d3fe09c[cdcr-cluster2_shard2_replica_n2] main]
[junit4] 2> 1438439 INFO (searcherExecutor-6164-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard2_replica_n2 c:cdcr-cluster2 s:shard2 r:core_node4) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.SolrCore [cdcr-cluster2_shard2_replica_n2] Registered new searcher Searcher@3d3fe09c[cdcr-cluster2_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1438440 INFO (recoveryExecutor-3369-thread-1-processing-n:127.0.0.1:34429_solr x:cdcr-cluster2_shard2_replica_n2 c:cdcr-cluster2 s:shard2 r:core_node4) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.h.CdcrRequestHandler No replay needed.
[junit4] 2> 1438456 INFO (qtp2126542339-12848) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.h.CdcrRequestHandler Found maxVersionFromRecent 0 maxVersionFromIndex 0
[junit4] 2> 1438456 INFO (qtp2126542339-12848) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.S.Request [cdcr-cluster2_shard2_replica_n2] webapp=/solr path=/cdcr params={action=SHARDCHECKPOINT&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1438459 INFO (qtp2126542339-12848) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.h.CdcrRequestHandler Found maxVersionFromRecent 0 maxVersionFromIndex 0
[junit4] 2> 1438459 INFO (qtp2126542339-12848) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.S.Request [cdcr-cluster2_shard1_replica_n1] webapp=/solr path=/cdcr params={action=SHARDCHECKPOINT&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1438459 INFO (qtp2126542339-12847) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.S.Request [cdcr-cluster2_shard2_replica_n2] webapp=/solr path=/cdcr params={_stateVer_=cdcr-cluster2:4&action=COLLECTIONCHECKPOINT&wt=javabin&version=2} status=0 QTime=29
[junit4] 2> 1438471 INFO (cdcr-bootstrap-status-3428-thread-1-processing-n:127.0.0.1:35341_solr x:cdcr-cluster1_shard1_replica_n1 c:cdcr-cluster1 s:shard1 r:core_node3) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.h.CdcrReplicatorManager Create new update log reader for target cdcr-cluster2 with checkpoint -1 @ cdcr-cluster1:shard1
[junit4] 2> 1438472 INFO (cdcr-bootstrap-status-3428-thread-1-processing-n:127.0.0.1:35341_solr x:cdcr-cluster1_shard1_replica_n1 c:cdcr-cluster1 s:shard1 r:core_node3) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.h.CdcrReplicatorManager Bootstrap successful, giving the go-ahead to replicator
[junit4] 2> 1439357 INFO (cdcr-replicator-6178-thread-1) [ ] o.a.s.h.CdcrReplicator Forwarded 0 updates to target cdcr-cluster2
[junit4] 2> 1440346 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.c.c.CdcrBidirectionalTest Adding 10 docs with commit=true, numDocs=100
[junit4] 2> 1440374 INFO (qtp2126542339-12848) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/cdcr-cluster2/terms/shard1 to Terms{values={core_node3=1}, version=1}
[junit4] 2> 1440374 INFO (qtp2126542339-12848) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.S.Request [cdcr-cluster2_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-cluster2:4&cdcr.update=&wt=javabin&version=2} status=0 QTime=10
[junit4] 2> 1440375 INFO (cdcr-replicator-6178-thread-1) [ ] o.a.s.h.CdcrReplicator Forwarded 43 updates to target cdcr-cluster2
[junit4] 2> 1440375 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/cdcr-cluster1/terms/shard2 to Terms{values={core_node4=1}, version=1}
[junit4] 2> 1440375 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.S.Request [cdcr-cluster1_shard2_replica_n2] webapp=/solr path=/update params={_stateVer_=cdcr-cluster1:4&wt=javabin&version=2} status=0 QTime=27
[junit4] 2> 1440376 INFO (qtp223663899-12906) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/cdcr-cluster1/terms/shard1 to Terms{values={core_node3=1}, version=1}
[junit4] 2> 1440376 INFO (qtp223663899-12906) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.S.Request [cdcr-cluster1_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-cluster1:4&wt=javabin&version=2} status=0 QTime=29
[junit4] 2> 1440386 INFO (qtp223663899-12906) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1611386856530771968,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1440386 INFO (qtp223663899-12906) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@55145717 commitCommandVersion:1611386856530771968
[junit4] 2> 1440392 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1611386856537063424,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1440392 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@22a9e700 commitCommandVersion:1611386856537063424
[junit4] 2> 1440437 INFO (qtp2126542339-12850) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.S.Request [cdcr-cluster2_shard2_replica_n2] webapp=/solr path=/cdcr params={qt=/cdcr&action=BOOTSTRAP_STATUS&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1440437 INFO (cdcr-bootstrap-status-3433-thread-1) [ ] o.a.s.h.CdcrReplicatorManager CDCR bootstrap successful in 3 seconds
[junit4] 2> 1440440 INFO (qtp2126542339-12848) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.h.CdcrRequestHandler Found maxVersionFromRecent 0 maxVersionFromIndex 0
[junit4] 2> 1440440 INFO (qtp2126542339-12848) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard2 r:core_node4 x:cdcr-cluster2_shard2_replica_n2] o.a.s.c.S.Request [cdcr-cluster2_shard2_replica_n2] webapp=/solr path=/cdcr params={action=SHARDCHECKPOINT&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1440444 INFO (qtp2126542339-12848) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.h.CdcrRequestHandler Found maxVersionFromRecent 1611386856493023266 maxVersionFromIndex 0
[junit4] 2> 1440444 INFO (qtp2126542339-12848) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.S.Request [cdcr-cluster2_shard1_replica_n1] webapp=/solr path=/cdcr params={action=SHARDCHECKPOINT&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1440445 INFO (qtp2126542339-12850) [n:127.0.0.1:34429_solr c:cdcr-cluster2 s:shard1 r:core_node3 x:cdcr-cluster2_shard1_replica_n1] o.a.s.c.S.Request [cdcr-cluster2_shard1_replica_n1] webapp=/solr path=/cdcr params={_stateVer_=cdcr-cluster2:4&action=COLLECTIONCHECKPOINT&wt=javabin&version=2} status=0 QTime=7
[junit4] 2> 1440448 INFO (cdcr-bootstrap-status-3433-thread-1) [ ] o.a.s.h.CdcrReplicatorManager Create new update log reader for target cdcr-cluster2 with checkpoint -1 @ cdcr-cluster1:shard2
[junit4] 2> 1440448 INFO (cdcr-bootstrap-status-3433-thread-1) [ ] o.a.s.h.CdcrReplicatorManager Bootstrap successful, giving the go-ahead to replicator
[junit4] 2> 1440815 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6ee68857[cdcr-cluster1_shard1_replica_n1] main]
[junit4] 2> 1440820 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1440821 INFO (searcherExecutor-6152-thread-1-processing-n:127.0.0.1:35341_solr x:cdcr-cluster1_shard1_replica_n1 c:cdcr-cluster1 s:shard1 r:core_node3) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.SolrCore [cdcr-cluster1_shard1_replica_n1] Registered new searcher Searcher@6ee68857[cdcr-cluster1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.6.0):C43)))}
[junit4] 2> 1440821 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.S.Request [cdcr-cluster1_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:35341/solr/cdcr-cluster1_shard2_replica_n2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false} status=0 QTime=429
[junit4] 2> 1440832 INFO (qtp223663899-12906) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@53559b26[cdcr-cluster1_shard2_replica_n2] main]
[junit4] 2> 1440832 INFO (qtp223663899-12906) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1440847 INFO (searcherExecutor-6153-thread-1-processing-n:127.0.0.1:35341_solr x:cdcr-cluster1_shard2_replica_n2 c:cdcr-cluster1 s:shard2 r:core_node4) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.SolrCore [cdcr-cluster1_shard2_replica_n2] Registered new searcher Searcher@53559b26[cdcr-cluster1_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(7.6.0):C57)))}
[junit4] 2> 1440847 INFO (qtp223663899-12906) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.S.Request [cdcr-cluster1_shard2_replica_n2] webapp=/solr path=/update params={update.distrib=FROMLEADER&update.chain=cdcr-processor-chain&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:35341/solr/cdcr-cluster1_shard2_replica_n2/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false} status=0 QTime=461
[junit4] 2> 1440848 INFO (qtp223663899-12901) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.S.Request [cdcr-cluster1_shard2_replica_n2] webapp=/solr path=/update params={_stateVer_=cdcr-cluster1:4&waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2} status=0 QTime=470
[junit4] 2> 1440848 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.c.c.CdcrBidirectionalTest Adding 10 docs with commit=true, numDocs=200
[junit4] 2> 1440859 INFO (qtp223663899-12899) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.c.S.Request [cdcr-cluster1_shard1_replica_n1] webapp=/solr path=/update params={_stateVer_=cdcr-cluster1:4&wt=javabin&version=2} status=0 QTime=10
[junit4] 2> 1440862 INFO (qtp223663899-12906) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard2 r:core_node4 x:cdcr-cluster1_shard2_replica_n2] o.a.s.c.S.Request [cdcr-cluster1_shard2_replica_n2] webapp=/solr path=/update params={_stateVer_=cdcr-cluster1:4&wt=javabin&version=2} status=0 QTime=12
[junit4] 2> 1440864 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1611386857031991296,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1440864 INFO (qtp223663899-12902) [n:127.0.0.1:35341_solr c:cdcr-cluster1 s:shard1 r:core_node3 x:cdcr-cluster1_shard1_replica_n1] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@22a9e700 commitCommandVersion:1611386857031991296
[junit4] 2> 1440877 INFO (qtp223663899-12906)
[...truncated too long message...]
[junit4] 2> 87270 WARN (OverseerAutoScalingTriggerThread-72322708277035011-127.0.0.1:52480_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 87277 INFO (jetty-closer-174-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@199c11f2{/solr,null,UNAVAILABLE}
[junit4] 2> 87277 INFO (jetty-closer-174-thread-1) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 87278 ERROR (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 87279 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:62272 62272
[junit4] 2> 87319 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxnFactory Ignoring unexpected runtime exception
[junit4] 2> java.nio.channels.CancelledKeyException: null
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) ~[?:1.8.0_172]
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87) ~[?:1.8.0_172]
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:205) [zookeeper-3.4.11.jar:3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0]
[junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
[junit4] 2> 87320 INFO (Thread-54) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:62272 62272
[junit4] 2> 87321 WARN (Thread-54) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/aliases.json
[junit4] 2> 5 /solr/clusterprops.json
[junit4] 2> 3 /solr/collections/cdcr-cluster2/terms/shard2
[junit4] 2> 3 /solr/collections/cdcr-cluster2/terms/shard1
[junit4] 2> 2 /solr/configs/cdcr-cluster2/managed-schema
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/collections/cdcr-cluster2/cdcr/state/process
[junit4] 2> 5 /solr/clusterstate.json
[junit4] 2> 3 /solr/collections/cdcr-cluster2/state.json
[junit4] 2> 2 /solr/collections/cdcr-cluster2/cdcr/state/buffer
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/live_nodes
[junit4] 2> 5 /solr/collections
[junit4] 2>
[junit4] 2> 87322 INFO (TEST-CdcrBidirectionalTest.testBiDir-seed#[7E5970A0C6AFE24E]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testBiDir
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=CdcrBidirectionalTest -Dtests.method=testBiDir -Dtests.seed=7E5970A0C6AFE24E -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ja-JP -Dtests.timezone=Asia/Katmandu -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 22.9s J0 | CdcrBidirectionalTest.testBiDir <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=715, name=cdcr-replicator-201-thread-1, state=RUNNABLE, group=TGRP-CdcrBidirectionalTest]
[junit4] > Caused by: java.lang.AssertionError: 1611391313648812032 != 1611391313647763456
[junit4] > at __randomizedtesting.SeedInfo.seed([7E5970A0C6AFE24E]:0)
[junit4] > at org.apache.solr.update.CdcrUpdateLog$CdcrLogReader.forwardSeek(CdcrUpdateLog.java:611)
[junit4] > at org.apache.solr.handler.CdcrReplicator.run(CdcrReplicator.java:125)
[junit4] > at org.apache.solr.handler.CdcrReplicatorScheduler.lambda$null$0(CdcrReplicatorScheduler.java:81)
[junit4] > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=724, name=cdcr-replicator-204-thread-1, state=RUNNABLE, group=TGRP-CdcrBidirectionalTest]
[junit4] > Caused by: java.lang.AssertionError: 1611391313648812032 != 1611391313647763456
[junit4] > at __randomizedtesting.SeedInfo.seed([7E5970A0C6AFE24E]:0)
[junit4] > at org.apache.solr.update.CdcrUpdateLog$CdcrLogReader.forwardSeek(CdcrUpdateLog.java:611)
[junit4] > at org.apache.solr.handler.CdcrReplicator.run(CdcrReplicator.java:105)
[junit4] > at org.apache.solr.handler.CdcrReplicatorScheduler.lambda$null$0(CdcrReplicatorScheduler.java:81)
[junit4] > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=747, name=cdcr-replicator-204-thread-2, state=RUNNABLE, group=TGRP-CdcrBidirectionalTest]
[junit4] > Caused by: java.lang.AssertionError: 1611391313648812032 != 1611391313647763456
[junit4] > at __randomizedtesting.SeedInfo.seed([7E5970A0C6AFE24E]:0)
[junit4] > at org.apache.solr.update.CdcrUpdateLog$CdcrLogReader.forwardSeek(CdcrUpdateLog.java:611)
[junit4] > at org.apache.solr.handler.CdcrReplicator.run(CdcrReplicator.java:105)
[junit4] > at org.apache.solr.handler.CdcrReplicatorScheduler.lambda$null$0(CdcrReplicatorScheduler.java:81)
[junit4] > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 87326 INFO (SUITE-CdcrBidirectionalTest-seed#[7E5970A0C6AFE24E]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
[junit4] 2> Sep 12, 2018 8:55:12 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 4 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {abc=PostingsFormat(name=LuceneFixedGap), xyz=BlockTreeOrds(blocksize=128), id=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{_version_=DocValuesFormat(name=Lucene70), xyz=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=241, maxMBSortInHeap=6.364468224506482, sim=RandomSimilarity(queryNorm=false): {}, locale=ja-JP, timezone=Asia/Katmandu
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=3,threads=1,free=48752544,total=100663296
[junit4] 2> NOTE: All tests run in this JVM: [CdcrBidirectionalTest, CdcrBidirectionalTest, CdcrBidirectionalTest]
[junit4] Completed [5/5 (1!)] on J0 in 23.93s, 1 test, 1 error <<< FAILURES!
[junit4]
[junit4]
[junit4] Tests with failures [seed: 7E5970A0C6AFE24E]:
[junit4] - org.apache.solr.cloud.cdcr.CdcrBidirectionalTest.testBiDir
[junit4]
[junit4]
[junit4] JVM J0: 0.74 .. 89.26 = 88.51s
[junit4] JVM J1: 0.74 .. 65.57 = 64.83s
[junit4] Execution time total: 1 minute 29 seconds
[junit4] Tests summary: 5 suites, 5 tests, 1 error
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/lucene/common-build.xml:1567: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/lucene/common-build.xml:1092: There were test failures: 5 suites, 5 tests, 1 error [seed: 7E5970A0C6AFE24E]
Total time: 1 minute 31 seconds
[repro] Setting last failure code to 256
[repro] Failures:
[repro] 1/5 failed: org.apache.solr.cloud.cdcr.CdcrBidirectionalTest
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=0789a77c2590f716fc3cedb247309068c3fc5d85, workspace=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris
[WARNINGS] Computing warning deltas based on reference build #812
Recording test results
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 815 -
Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/815/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC
5 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicZkTest
Error Message:
SolrCore.getOpenCount()==2
Stack Trace:
java.lang.RuntimeException: SolrCore.getOpenCount()==2
at __randomizedtesting.SeedInfo.seed([7C92AACABAC9980E]:0)
at org.apache.solr.util.TestHarness.close(TestHarness.java:380)
at org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:802)
at org.apache.solr.cloud.AbstractZkTestCase.azt_afterClass(AbstractZkTestCase.java:147)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicZkTest
Error Message:
SolrCore.getOpenCount()==2
Stack Trace:
java.lang.RuntimeException: SolrCore.getOpenCount()==2
at __randomizedtesting.SeedInfo.seed([7C92AACABAC9980E]:0)
at org.apache.solr.util.TestHarness.close(TestHarness.java:380)
at org.apache.solr.SolrTestCaseJ4.deleteCore(SolrTestCaseJ4.java:802)
at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:297)
at sun.reflect.GeneratedMethodAccessor64.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicZkTest
Error Message:
26 threads leaked from SUITE scope at org.apache.solr.cloud.BasicZkTest: 1) Thread[id=19356, name=MetricsHistoryHandler-7511-thread-1, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=19335, name=Connection evictor, state=TIMED_WAITING, group=TGRP-BasicZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=19344, name=SUITE-BasicZkTest-seed#[7C92AACABAC9980E]-worker-SendThread(127.0.0.1:49399), state=RUNNABLE, group=TGRP-BasicZkTest] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144) 4) Thread[id=19355, name=SolrRrdBackendFactory-7510-thread-1, state=TIMED_WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=19358, name=Thread-4800, state=WAITING, group=TGRP-BasicZkTest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1850) 6) Thread[id=19379, name=SolrRrdBackendFactory-7510-thread-2, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=19348, name=OverseerAutoScalingTriggerThread-72329213934108675-127.0.0.1:8983_-n_0000000000, state=WAITING, group=Overseer autoscaling triggers] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189) at java.lang.Thread.run(Thread.java:748) 8) Thread[id=19340, name=SUITE-BasicZkTest-seed#[7C92AACABAC9980E]-worker-EventThread, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 9) Thread[id=19353, name=ScheduledTrigger-7506-thread-3, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 10) Thread[id=19354, name=ScheduledTrigger-7506-thread-4, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 11) Thread[id=19372, name=Thread-4801, state=WAITING, group=TGRP-BasicZkTest] at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:313) at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:313) at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:496) 12) Thread[id=19360, name=searcherExecutor-7513-thread-1, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 13) Thread[id=19373, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:49399, state=RUNNABLE, group=TGRP-BasicZkTest] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196) at java.lang.Thread.run(Thread.java:748) 14) Thread[id=19339, name=SUITE-BasicZkTest-seed#[7C92AACABAC9980E]-worker-SendThread(127.0.0.1:49399), state=RUNNABLE, group=TGRP-BasicZkTest] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144) 15) Thread[id=19343, name=Connection evictor, state=TIMED_WAITING, group=TGRP-BasicZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 16) Thread[id=19333, name=Connection evictor, state=TIMED_WAITING, group=TGRP-BasicZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 17) Thread[id=19346, name=zkConnectionManagerCallback-4881-thread-1, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 18) Thread[id=19334, name=Connection evictor, state=TIMED_WAITING, group=TGRP-BasicZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 19) Thread[id=19341, name=zkConnectionManagerCallback-4874-thread-1, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 20) Thread[id=19342, name=OverseerStateUpdate-72329213934108675-127.0.0.1:8983_-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463) at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:207) at java.lang.Thread.run(Thread.java:748) 21) Thread[id=19352, name=ScheduledTrigger-7506-thread-2, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 22) Thread[id=19347, name=OverseerCollectionConfigSetProcessor-72329213934108675-127.0.0.1:8983_-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463) at org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:243) at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:233) at java.lang.Thread.run(Thread.java:748) 23) Thread[id=19351, name=ScheduledTrigger-7506-thread-1, state=TIMED_WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 24) Thread[id=19345, name=SUITE-BasicZkTest-seed#[7C92AACABAC9980E]-worker-EventThread, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 25) Thread[id=19376, name=ProcessThread(sid:0 cport:49399):, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:123) 26) Thread[id=19374, name=SessionTracker, state=TIMED_WAITING, group=TGRP-BasicZkTest] at java.lang.Object.wait(Native Method) at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:147)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 26 threads leaked from SUITE scope at org.apache.solr.cloud.BasicZkTest:
1) Thread[id=19356, name=MetricsHistoryHandler-7511-thread-1, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2) Thread[id=19335, name=Connection evictor, state=TIMED_WAITING, group=TGRP-BasicZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
3) Thread[id=19344, name=SUITE-BasicZkTest-seed#[7C92AACABAC9980E]-worker-SendThread(127.0.0.1:49399), state=RUNNABLE, group=TGRP-BasicZkTest]
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
4) Thread[id=19355, name=SolrRrdBackendFactory-7510-thread-1, state=TIMED_WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
5) Thread[id=19358, name=Thread-4800, state=WAITING, group=TGRP-BasicZkTest]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1850)
6) Thread[id=19379, name=SolrRrdBackendFactory-7510-thread-2, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
7) Thread[id=19348, name=OverseerAutoScalingTriggerThread-72329213934108675-127.0.0.1:8983_-n_0000000000, state=WAITING, group=Overseer autoscaling triggers]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189)
at java.lang.Thread.run(Thread.java:748)
8) Thread[id=19340, name=SUITE-BasicZkTest-seed#[7C92AACABAC9980E]-worker-EventThread, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
9) Thread[id=19353, name=ScheduledTrigger-7506-thread-3, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
10) Thread[id=19354, name=ScheduledTrigger-7506-thread-4, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
11) Thread[id=19372, name=Thread-4801, state=WAITING, group=TGRP-BasicZkTest]
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1252)
at java.lang.Thread.join(Thread.java:1326)
at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:313)
at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:313)
at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:496)
12) Thread[id=19360, name=searcherExecutor-7513-thread-1, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
13) Thread[id=19373, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:49399, state=RUNNABLE, group=TGRP-BasicZkTest]
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:196)
at java.lang.Thread.run(Thread.java:748)
14) Thread[id=19339, name=SUITE-BasicZkTest-seed#[7C92AACABAC9980E]-worker-SendThread(127.0.0.1:49399), state=RUNNABLE, group=TGRP-BasicZkTest]
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)
15) Thread[id=19343, name=Connection evictor, state=TIMED_WAITING, group=TGRP-BasicZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
16) Thread[id=19333, name=Connection evictor, state=TIMED_WAITING, group=TGRP-BasicZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
17) Thread[id=19346, name=zkConnectionManagerCallback-4881-thread-1, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
18) Thread[id=19334, name=Connection evictor, state=TIMED_WAITING, group=TGRP-BasicZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
19) Thread[id=19341, name=zkConnectionManagerCallback-4874-thread-1, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
20) Thread[id=19342, name=OverseerStateUpdate-72329213934108675-127.0.0.1:8983_-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)
at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:207)
at java.lang.Thread.run(Thread.java:748)
21) Thread[id=19352, name=ScheduledTrigger-7506-thread-2, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
22) Thread[id=19347, name=OverseerCollectionConfigSetProcessor-72329213934108675-127.0.0.1:8983_-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:463)
at org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:243)
at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:233)
at java.lang.Thread.run(Thread.java:748)
23) Thread[id=19351, name=ScheduledTrigger-7506-thread-1, state=TIMED_WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
24) Thread[id=19345, name=SUITE-BasicZkTest-seed#[7C92AACABAC9980E]-worker-EventThread, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
25) Thread[id=19376, name=ProcessThread(sid:0 cport:49399):, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:123)
26) Thread[id=19374, name=SessionTracker, state=TIMED_WAITING, group=TGRP-BasicZkTest]
at java.lang.Object.wait(Native Method)
at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:147)
at __randomizedtesting.SeedInfo.seed([7C92AACABAC9980E]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.BasicZkTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=19356, name=MetricsHistoryHandler-7511-thread-1, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=19341, name=zkConnectionManagerCallback-4874-thread-1, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=19344, name=SUITE-BasicZkTest-seed#[7C92AACABAC9980E]-worker-SendThread(127.0.0.1:49399), state=TIMED_WAITING, group=TGRP-BasicZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054) 4) Thread[id=19355, name=SolrRrdBackendFactory-7510-thread-1, state=TIMED_WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=19352, name=ScheduledTrigger-7506-thread-2, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=19358, name=Thread-4800, state=WAITING, group=TGRP-BasicZkTest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1850) 7) Thread[id=19379, name=SolrRrdBackendFactory-7510-thread-2, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 8) Thread[id=19351, name=ScheduledTrigger-7506-thread-1, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 9) Thread[id=19353, name=ScheduledTrigger-7506-thread-3, state=TIMED_WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 10) Thread[id=19354, name=ScheduledTrigger-7506-thread-4, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 11) Thread[id=19360, name=searcherExecutor-7513-thread-1, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 12) Thread[id=19339, name=SUITE-BasicZkTest-seed#[7C92AACABAC9980E]-worker-SendThread(127.0.0.1:49399), state=TIMED_WAITING, group=TGRP-BasicZkTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 13) Thread[id=19346, name=zkConnectionManagerCallback-4881-thread-1, state=WAITING, group=TGRP-BasicZkTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=19356, name=MetricsHistoryHandler-7511-thread-1, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2) Thread[id=19341, name=zkConnectionManagerCallback-4874-thread-1, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
3) Thread[id=19344, name=SUITE-BasicZkTest-seed#[7C92AACABAC9980E]-worker-SendThread(127.0.0.1:49399), state=TIMED_WAITING, group=TGRP-BasicZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
4) Thread[id=19355, name=SolrRrdBackendFactory-7510-thread-1, state=TIMED_WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
5) Thread[id=19352, name=ScheduledTrigger-7506-thread-2, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
6) Thread[id=19358, name=Thread-4800, state=WAITING, group=TGRP-BasicZkTest]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1850)
7) Thread[id=19379, name=SolrRrdBackendFactory-7510-thread-2, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
8) Thread[id=19351, name=ScheduledTrigger-7506-thread-1, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
9) Thread[id=19353, name=ScheduledTrigger-7506-thread-3, state=TIMED_WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
10) Thread[id=19354, name=ScheduledTrigger-7506-thread-4, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
11) Thread[id=19360, name=searcherExecutor-7513-thread-1, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
12) Thread[id=19339, name=SUITE-BasicZkTest-seed#[7C92AACABAC9980E]-worker-SendThread(127.0.0.1:49399), state=TIMED_WAITING, group=TGRP-BasicZkTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
13) Thread[id=19346, name=zkConnectionManagerCallback-4881-thread-1, state=WAITING, group=TGRP-BasicZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([7C92AACABAC9980E]:0)
FAILED: org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove
Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:37839/solr/MoveReplicaHDFSTest_failed_coll_true, http://127.0.0.1:45114/solr/MoveReplicaHDFSTest_failed_coll_true]
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:37839/solr/MoveReplicaHDFSTest_failed_coll_true, http://127.0.0.1:45114/solr/MoveReplicaHDFSTest_failed_coll_true]
at __randomizedtesting.SeedInfo.seed([7C92AACABAC9980E:D65F79380D1A4DDE]:0)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:994)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
at org.apache.solr.cloud.MoveReplicaTest.testFailedMove(MoveReplicaTest.java:289)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:37839/solr/MoveReplicaHDFSTest_failed_coll_true: no servers hosting shard: shard2
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:436)
... 46 more
Build Log:
[...truncated 12970 lines...]
[junit4] Suite: org.apache.solr.cloud.MoveReplicaHDFSTest
[junit4] 2> 1277867 INFO (SUITE-MoveReplicaHDFSTest-seed#[7C92AACABAC9980E]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.MoveReplicaHDFSTest_7C92AACABAC9980E-001/init-core-data-001
[junit4] 2> 1277867 WARN (SUITE-MoveReplicaHDFSTest-seed#[7C92AACABAC9980E]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
[junit4] 2> 1277867 INFO (SUITE-MoveReplicaHDFSTest-seed#[7C92AACABAC9980E]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 1277869 INFO (SUITE-MoveReplicaHDFSTest-seed#[7C92AACABAC9980E]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
[junit4] 2> 1277869 INFO (SUITE-MoveReplicaHDFSTest-seed#[7C92AACABAC9980E]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.MoveReplicaHDFSTest_7C92AACABAC9980E-001/tempDir-001
[junit4] 2> 1277869 INFO (SUITE-MoveReplicaHDFSTest-seed#[7C92AACABAC9980E]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1277870 INFO (Thread-2386) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1277870 INFO (Thread-2386) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1277871 ERROR (Thread-2386) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1277973 INFO (SUITE-MoveReplicaHDFSTest-seed#[7C92AACABAC9980E]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:32770
[junit4] 2> 1277976 INFO (zkConnectionManagerCallback-3267-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1277981 INFO (jetty-launcher-3264-thread-1) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 1277981 INFO (jetty-launcher-3264-thread-2) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 1277982 INFO (jetty-launcher-3264-thread-3) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 1277982 INFO (jetty-launcher-3264-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1277982 INFO (jetty-launcher-3264-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1277982 INFO (jetty-launcher-3264-thread-1) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 1277983 INFO (jetty-launcher-3264-thread-3) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1277983 INFO (jetty-launcher-3264-thread-3) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1277983 INFO (jetty-launcher-3264-thread-3) [ ] o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 1277983 INFO (jetty-launcher-3264-thread-3) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4f5da5aa{/solr,null,AVAILABLE}
[junit4] 2> 1277984 INFO (jetty-launcher-3264-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1277984 INFO (jetty-launcher-3264-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1277984 INFO (jetty-launcher-3264-thread-2) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 1277985 INFO (jetty-launcher-3264-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@58e3a7dc{/solr,null,AVAILABLE}
[junit4] 2> 1277986 INFO (jetty-launcher-3264-thread-3) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3303e556{HTTP/1.1,[http/1.1]}{127.0.0.1:45114}
[junit4] 2> 1277986 INFO (jetty-launcher-3264-thread-3) [ ] o.e.j.s.Server Started @1278612ms
[junit4] 2> 1277986 INFO (jetty-launcher-3264-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=45114}
[junit4] 2> 1277986 ERROR (jetty-launcher-3264-thread-3) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1277986 INFO (jetty-launcher-3264-thread-3) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 1277986 INFO (jetty-launcher-3264-thread-3) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.6.0
[junit4] 2> 1277986 INFO (jetty-launcher-3264-thread-3) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1277986 INFO (jetty-launcher-3264-thread-3) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1277986 INFO (jetty-launcher-3264-thread-3) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-13T12:18:30.984Z
[junit4] 2> 1277986 INFO (jetty-launcher-3264-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2d3f734a{/solr,null,AVAILABLE}
[junit4] 2> 1277986 INFO (jetty-launcher-3264-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@66f1889f{HTTP/1.1,[http/1.1]}{127.0.0.1:62305}
[junit4] 2> 1277987 INFO (jetty-launcher-3264-thread-1) [ ] o.e.j.s.Server Started @1278613ms
[junit4] 2> 1277987 INFO (jetty-launcher-3264-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=62305}
[junit4] 2> 1277987 ERROR (jetty-launcher-3264-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1277987 INFO (jetty-launcher-3264-thread-1) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 1277987 INFO (jetty-launcher-3264-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.6.0
[junit4] 2> 1277987 INFO (jetty-launcher-3264-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1277987 INFO (jetty-launcher-3264-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1277987 INFO (jetty-launcher-3264-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-13T12:18:30.985Z
[junit4] 2> 1277987 INFO (jetty-launcher-3264-thread-4) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 1277990 INFO (jetty-launcher-3264-thread-4) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1277990 INFO (jetty-launcher-3264-thread-4) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1277990 INFO (jetty-launcher-3264-thread-4) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 1277990 INFO (jetty-launcher-3264-thread-4) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@68b4512e{/solr,null,AVAILABLE}
[junit4] 2> 1277991 INFO (jetty-launcher-3264-thread-4) [ ] o.e.j.s.AbstractConnector Started ServerConnector@2344cb68{HTTP/1.1,[http/1.1]}{127.0.0.1:58021}
[junit4] 2> 1277991 INFO (jetty-launcher-3264-thread-4) [ ] o.e.j.s.Server Started @1278617ms
[junit4] 2> 1277991 INFO (jetty-launcher-3264-thread-4) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=58021}
[junit4] 2> 1277991 ERROR (jetty-launcher-3264-thread-4) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1277991 INFO (jetty-launcher-3264-thread-4) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 1277991 INFO (jetty-launcher-3264-thread-4) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.6.0
[junit4] 2> 1277991 INFO (jetty-launcher-3264-thread-4) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1277992 INFO (jetty-launcher-3264-thread-4) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1277992 INFO (jetty-launcher-3264-thread-4) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-13T12:18:30.990Z
[junit4] 2> 1277992 INFO (zkConnectionManagerCallback-3269-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1277992 INFO (jetty-launcher-3264-thread-3) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1277995 INFO (zkConnectionManagerCallback-3271-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1277996 INFO (jetty-launcher-3264-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1278028 INFO (zkConnectionManagerCallback-3273-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278029 INFO (jetty-launcher-3264-thread-4) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1278035 INFO (jetty-launcher-3264-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@67e1c2e5{HTTP/1.1,[http/1.1]}{127.0.0.1:37839}
[junit4] 2> 1278035 INFO (jetty-launcher-3264-thread-2) [ ] o.e.j.s.Server Started @1278661ms
[junit4] 2> 1278035 INFO (jetty-launcher-3264-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37839}
[junit4] 2> 1278035 ERROR (jetty-launcher-3264-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1278035 INFO (jetty-launcher-3264-thread-2) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 1278035 INFO (jetty-launcher-3264-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.6.0
[junit4] 2> 1278035 INFO (jetty-launcher-3264-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1278035 INFO (jetty-launcher-3264-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1278035 INFO (jetty-launcher-3264-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-13T12:18:31.033Z
[junit4] 2> 1278121 INFO (jetty-launcher-3264-thread-3) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:32770/solr
[junit4] 2> 1278124 INFO (zkConnectionManagerCallback-3275-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278124 INFO (jetty-launcher-3264-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1278132 INFO (zkConnectionManagerCallback-3279-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278348 INFO (jetty-launcher-3264-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:32770/solr
[junit4] 2> 1278349 INFO (zkConnectionManagerCallback-3281-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278360 INFO (zkConnectionManagerCallback-3285-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278365 INFO (zkConnectionManagerCallback-3287-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278447 INFO (jetty-launcher-3264-thread-1) [n:127.0.0.1:62305_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:62305_solr
[junit4] 2> 1278448 INFO (jetty-launcher-3264-thread-1) [n:127.0.0.1:62305_solr ] o.a.s.c.Overseer Overseer (id=72329174612181000-127.0.0.1:62305_solr-n_0000000000) starting
[junit4] 2> 1278466 INFO (jetty-launcher-3264-thread-3) [n:127.0.0.1:45114_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45114_solr
[junit4] 2> 1278485 INFO (zkCallback-3286-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1278486 INFO (zkConnectionManagerCallback-3294-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278488 INFO (jetty-launcher-3264-thread-1) [n:127.0.0.1:62305_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1278488 INFO (jetty-launcher-3264-thread-1) [n:127.0.0.1:62305_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:32770/solr ready
[junit4] 2> 1278490 INFO (jetty-launcher-3264-thread-1) [n:127.0.0.1:62305_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1278490 INFO (jetty-launcher-3264-thread-1) [n:127.0.0.1:62305_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:62305_solr
[junit4] 2> 1278503 INFO (jetty-launcher-3264-thread-1) [n:127.0.0.1:62305_solr ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 1278521 INFO (jetty-launcher-3264-thread-1) [n:127.0.0.1:62305_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_62305.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@896b69
[junit4] 2> 1278528 INFO (jetty-launcher-3264-thread-1) [n:127.0.0.1:62305_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_62305.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@896b69
[junit4] 2> 1278528 INFO (jetty-launcher-3264-thread-1) [n:127.0.0.1:62305_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_62305.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@896b69
[junit4] 2> 1278529 INFO (jetty-launcher-3264-thread-1) [n:127.0.0.1:62305_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.MoveReplicaHDFSTest_7C92AACABAC9980E-001/tempDir-001/node1/.
[junit4] 2> 1278531 INFO (zkCallback-3286-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1278531 INFO (OverseerStateUpdate-72329174612181000-127.0.0.1:62305_solr-n_0000000000) [n:127.0.0.1:62305_solr ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:62305_solr
[junit4] 2> 1278535 INFO (zkConnectionManagerCallback-3301-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278536 INFO (zkCallback-3280-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1278537 INFO (jetty-launcher-3264-thread-3) [n:127.0.0.1:45114_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1278538 INFO (jetty-launcher-3264-thread-3) [n:127.0.0.1:45114_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:32770/solr ready
[junit4] 2> 1278539 INFO (jetty-launcher-3264-thread-3) [n:127.0.0.1:45114_solr ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 1278551 INFO (jetty-launcher-3264-thread-3) [n:127.0.0.1:45114_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45114.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@896b69
[junit4] 2> 1278556 INFO (jetty-launcher-3264-thread-3) [n:127.0.0.1:45114_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45114.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@896b69
[junit4] 2> 1278556 INFO (jetty-launcher-3264-thread-3) [n:127.0.0.1:45114_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_45114.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@896b69
[junit4] 2> 1278557 INFO (jetty-launcher-3264-thread-3) [n:127.0.0.1:45114_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.MoveReplicaHDFSTest_7C92AACABAC9980E-001/tempDir-001/node3/.
[junit4] 2> 1278558 INFO (zkCallback-3293-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1278578 INFO (jetty-launcher-3264-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:32770/solr
[junit4] 2> 1278581 INFO (zkConnectionManagerCallback-3307-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278583 INFO (zkConnectionManagerCallback-3309-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278592 INFO (jetty-launcher-3264-thread-2) [n:127.0.0.1:37839_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1278598 INFO (jetty-launcher-3264-thread-2) [n:127.0.0.1:37839_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1278598 INFO (jetty-launcher-3264-thread-2) [n:127.0.0.1:37839_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37839_solr
[junit4] 2> 1278600 INFO (zkCallback-3300-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1278600 INFO (zkCallback-3286-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1278600 INFO (zkCallback-3308-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1278601 INFO (zkCallback-3293-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1278602 INFO (zkCallback-3280-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1278618 INFO (zkConnectionManagerCallback-3316-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278620 INFO (jetty-launcher-3264-thread-2) [n:127.0.0.1:37839_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 1278621 INFO (jetty-launcher-3264-thread-2) [n:127.0.0.1:37839_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:32770/solr ready
[junit4] 2> 1278622 INFO (jetty-launcher-3264-thread-2) [n:127.0.0.1:37839_solr ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 1278636 INFO (jetty-launcher-3264-thread-2) [n:127.0.0.1:37839_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37839.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@896b69
[junit4] 2> 1278642 INFO (jetty-launcher-3264-thread-2) [n:127.0.0.1:37839_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37839.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@896b69
[junit4] 2> 1278642 INFO (jetty-launcher-3264-thread-2) [n:127.0.0.1:37839_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37839.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@896b69
[junit4] 2> 1278643 INFO (jetty-launcher-3264-thread-2) [n:127.0.0.1:37839_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.MoveReplicaHDFSTest_7C92AACABAC9980E-001/tempDir-001/node2/.
[junit4] 2> 1278689 INFO (jetty-launcher-3264-thread-4) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:32770/solr
[junit4] 2> 1278691 INFO (zkConnectionManagerCallback-3321-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278695 INFO (zkConnectionManagerCallback-3323-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278705 INFO (jetty-launcher-3264-thread-4) [n:127.0.0.1:58021_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 1278711 INFO (jetty-launcher-3264-thread-4) [n:127.0.0.1:58021_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1278711 INFO (jetty-launcher-3264-thread-4) [n:127.0.0.1:58021_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58021_solr
[junit4] 2> 1278713 INFO (zkCallback-3286-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1278713 INFO (zkCallback-3322-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1278713 INFO (zkCallback-3293-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1278713 INFO (zkCallback-3280-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1278713 INFO (zkCallback-3315-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1278713 INFO (zkCallback-3308-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1278714 INFO (zkCallback-3300-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1278748 INFO (zkConnectionManagerCallback-3330-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278749 INFO (jetty-launcher-3264-thread-4) [n:127.0.0.1:58021_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 1278754 INFO (jetty-launcher-3264-thread-4) [n:127.0.0.1:58021_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:32770/solr ready
[junit4] 2> 1278754 INFO (jetty-launcher-3264-thread-4) [n:127.0.0.1:58021_solr ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 1278766 INFO (jetty-launcher-3264-thread-4) [n:127.0.0.1:58021_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58021.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@896b69
[junit4] 2> 1278771 INFO (jetty-launcher-3264-thread-4) [n:127.0.0.1:58021_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58021.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@896b69
[junit4] 2> 1278771 INFO (jetty-launcher-3264-thread-4) [n:127.0.0.1:58021_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_58021.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@896b69
[junit4] 2> 1278772 INFO (jetty-launcher-3264-thread-4) [n:127.0.0.1:58021_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.MoveReplicaHDFSTest_7C92AACABAC9980E-001/tempDir-001/node4/.
[junit4] 2> 1278809 INFO (zkConnectionManagerCallback-3333-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278814 INFO (zkConnectionManagerCallback-3338-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1278815 INFO (SUITE-MoveReplicaHDFSTest-seed#[7C92AACABAC9980E]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 1278817 INFO (SUITE-MoveReplicaHDFSTest-seed#[7C92AACABAC9980E]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:32770/solr ready
[junit4] 2> 1278822 INFO (qtp960022103-12401) [n:127.0.0.1:58021_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1278831 INFO (qtp960022103-12401) [n:127.0.0.1:58021_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=8
[junit4] 1> Formatting using clusterid: testClusterID
[junit4] 2> 1279030 WARN (SUITE-MoveReplicaHDFSTest-seed#[7C92AACABAC9980E]-worker) [ ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
[junit4] 2> 1279039 WARN (SUITE-MoveReplicaHDFSTest-seed#[7C92AACABAC9980E]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 1279041 INFO (SUITE-MoveReplicaHDFSTest-seed#[7C92AACABAC9980E]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 1279052 INFO (SUITE-MoveReplicaHDFSTest-seed#[7C92AACABA
[...truncated too long message...]
----------
-init-totals:
-test:
[mkdir] Created dir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test
[junit4:pickseed] Seed property 'tests.seed' already defined: 7C92AACABAC9980E
[mkdir] Created dir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/temp
[junit4] <JUnit4> says ahoj! Master seed: 7C92AACABAC9980E
[junit4] Executing 10 suites with 2 JVMs.
[junit4]
[junit4] Started J0 PID(11199@serv1-vm3).
[junit4] Started J1 PID(11200@serv1-vm3).
[junit4] Suite: org.apache.solr.cloud.MoveReplicaHDFSTest
[junit4] OK 24.3s J0 | MoveReplicaHDFSTest.testFailedMove
[junit4] OK 17.4s J0 | MoveReplicaHDFSTest.testNormalFailedMove
[junit4] OK 25.9s J0 | MoveReplicaHDFSTest.test
[junit4] OK 23.2s J0 | MoveReplicaHDFSTest.testNormalMove
[junit4] Completed [1/10] on J0 in 126.13s, 4 tests
[junit4]
[junit4] Suite: org.apache.solr.cloud.MoveReplicaHDFSTest
[junit4] OK 24.1s J1 | MoveReplicaHDFSTest.testFailedMove
[junit4] OK 16.2s J1 | MoveReplicaHDFSTest.testNormalFailedMove
[junit4] OK 27.3s J1 | MoveReplicaHDFSTest.test
[junit4] OK 26.2s J1 | MoveReplicaHDFSTest.testNormalMove
[junit4] Completed [2/10] on J1 in 128.96s, 4 tests
[junit4]
[junit4] Duplicate suite name used with XML reports: org.apache.solr.cloud.MoveReplicaHDFSTest. This may confuse tools that process XML reports. Set 'ignoreDuplicateSuites' to true to skip this message.
[junit4] Suite: org.apache.solr.cloud.BasicZkTest
[junit4] OK 1.92s J0 | BasicZkTest.testBasic
[junit4] Completed [3/10] on J0 in 8.37s, 1 test
[junit4]
[junit4] Suite: org.apache.solr.cloud.MoveReplicaHDFSTest
[junit4] OK 20.2s J1 | MoveReplicaHDFSTest.testFailedMove
[junit4] OK 15.8s J1 | MoveReplicaHDFSTest.testNormalFailedMove
[junit4] OK 22.5s J1 | MoveReplicaHDFSTest.test
[junit4] OK 25.3s J1 | MoveReplicaHDFSTest.testNormalMove
[junit4] Completed [4/10] on J1 in 99.44s, 4 tests
[junit4]
[junit4] Suite: org.apache.solr.cloud.MoveReplicaHDFSTest
[junit4] OK 18.0s J0 | MoveReplicaHDFSTest.testFailedMove
[junit4] OK 17.4s J0 | MoveReplicaHDFSTest.testNormalFailedMove
[junit4] OK 22.9s J0 | MoveReplicaHDFSTest.test
[junit4] OK 24.2s J0 | MoveReplicaHDFSTest.testNormalMove
[junit4] Completed [5/10] on J0 in 96.53s, 4 tests
[junit4]
[junit4] Suite: org.apache.solr.cloud.BasicZkTest
[junit4] OK 3.14s J0 | BasicZkTest.testBasic
[junit4] Completed [6/10] on J0 in 6.36s, 1 test
[junit4]
[junit4] Duplicate suite name used with XML reports: org.apache.solr.cloud.BasicZkTest. This may confuse tools that process XML reports. Set 'ignoreDuplicateSuites' to true to skip this message.
[junit4] Suite: org.apache.solr.cloud.BasicZkTest
[junit4] OK 1.89s J0 | BasicZkTest.testBasic
[junit4] Completed [7/10] on J0 in 4.90s, 1 test
[junit4]
[junit4] Suite: org.apache.solr.cloud.BasicZkTest
[junit4] OK 2.35s J0 | BasicZkTest.testBasic
[junit4] Completed [8/10] on J0 in 5.66s, 1 test
[junit4]
[junit4] Suite: org.apache.solr.cloud.BasicZkTest
[junit4] OK 2.92s J0 | BasicZkTest.testBasic
[junit4] Completed [9/10] on J0 in 5.85s, 1 test
[junit4]
[junit4] Suite: org.apache.solr.cloud.MoveReplicaHDFSTest
[junit4] OK 40.5s J1 | MoveReplicaHDFSTest.testFailedMove
[junit4] OK 13.5s J1 | MoveReplicaHDFSTest.testNormalFailedMove
[junit4] OK 24.1s J1 | MoveReplicaHDFSTest.test
[junit4] OK 19.0s J1 | MoveReplicaHDFSTest.testNormalMove
[junit4] Completed [10/10] on J1 in 111.65s, 4 tests
[junit4]
[junit4] JVM J0: 0.76 .. 256.59 = 255.83s
[junit4] JVM J1: 0.77 .. 342.16 = 341.39s
[junit4] Execution time total: 5 minutes 42 seconds
[junit4] Tests summary: 10 suites, 25 tests
[junit4] Could not remove temporary path: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0 (java.nio.file.DirectoryNotEmptyException: Remaining files: [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J0/temp])
[junit4] Could not remove temporary path: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1 (java.nio.file.DirectoryNotEmptyException: Remaining files: [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp])
[echo] 5 slowest tests:
[junit4:tophints] 609.63s | org.apache.solr.cloud.BasicDistributedZkTest
[junit4:tophints] 460.54s | org.apache.solr.cloud.api.collections.ShardSplitTest
[junit4:tophints] 340.74s | org.apache.solr.cloud.cdcr.CdcrWithNodesRestartsTest
[junit4:tophints] 130.77s | org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest
[junit4:tophints] 109.70s | org.apache.solr.cloud.ZkFailoverTest
-check-totals:
test-nocompile:
BUILD SUCCESSFUL
Total time: 5 minutes 45 seconds
[repro] Failures:
[repro] 0/5 failed: org.apache.solr.cloud.BasicZkTest
[repro] 0/5 failed: org.apache.solr.cloud.MoveReplicaHDFSTest
[repro] Exiting with code 0
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
java.lang.InterruptedException: no matches found within 10000
at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2845)
at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2724)
at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2705)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3084)
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to Solaris VBOX
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:955)
at hudson.FilePath.act(FilePath.java:1070)
at hudson.FilePath.act(FilePath.java:1059)
at hudson.FilePath.validateAntFileMask(FilePath.java:2703)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
at hudson.model.Run.execute(Run.java:1840)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
at hudson.FilePath.act(FilePath.java:1072)
at hudson.FilePath.act(FilePath.java:1059)
at hudson.FilePath.validateAntFileMask(FilePath.java:2703)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
at hudson.model.Run.execute(Run.java:1840)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/*_pid*.log". Configuration error?
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=f76a424aa2b1a29eda229e0e7b292551d96e9d29, workspace=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris
[WARNINGS] Computing warning deltas based on reference build #814
Recording test results
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 814 -
Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/814/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
4 tests failed.
FAILED: org.apache.lucene.index.TestAddIndexes.testAddIndicesWithSoftDeletes
Error Message:
Index: 2, Size: 2
Stack Trace:
java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
at __randomizedtesting.SeedInfo.seed([B80C15673B91141:27D43E12C6BC3559]:0)
at java.util.ArrayList.rangeCheck(ArrayList.java:657)
at java.util.ArrayList.get(ArrayList.java:433)
at java.util.Collections$UnmodifiableList.get(Collections.java:1309)
at org.apache.lucene.index.TestAddIndexes.testAddIndicesWithSoftDeletes(TestAddIndexes.java:1455)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.UnloadDistributedZkTest.test
Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:40258
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:40258
at __randomizedtesting.SeedInfo.seed([28027971633A8DD1:A05646ABCDC6E029]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.cloud.UnloadDistributedZkTest.testCoreUnloadAndLeaders(UnloadDistributedZkTest.java:307)
at org.apache.solr.cloud.UnloadDistributedZkTest.test(UnloadDistributedZkTest.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1008)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:983)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
... 52 more
FAILED: org.apache.solr.core.TestLazyCores.testLazyLoad
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([28027971633A8DD1]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.core.TestLazyCores
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([28027971633A8DD1]:0)
Build Log:
[...truncated 1215 lines...]
[junit4] Suite: org.apache.lucene.index.TestAddIndexes
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestAddIndexes -Dtests.method=testAddIndicesWithSoftDeletes -Dtests.seed=B80C15673B91141 -Dtests.slow=true -Dtests.locale=es-SV -Dtests.timezone=Asia/Jakarta -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.02s J0 | TestAddIndexes.testAddIndicesWithSoftDeletes <<<
[junit4] > Throwable #1: java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
[junit4] > at __randomizedtesting.SeedInfo.seed([B80C15673B91141:27D43E12C6BC3559]:0)
[junit4] > at java.util.ArrayList.rangeCheck(ArrayList.java:657)
[junit4] > at java.util.ArrayList.get(ArrayList.java:433)
[junit4] > at java.util.Collections$UnmodifiableList.get(Collections.java:1309)
[junit4] > at org.apache.lucene.index.TestAddIndexes.testAddIndicesWithSoftDeletes(TestAddIndexes.java:1455)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/lucene/build/core/test/J0/temp/lucene.index.TestAddIndexes_B80C15673B91141-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {c=PostingsFormat(name=LuceneVarGapDocFreqInterval), id=PostingsFormat(name=LuceneVarGapDocFreqInterval), f1=PostingsFormat(name=LuceneVarGapDocFreqInterval), f2=FSTOrd50, version=FSTOrd50, content=BlockTreeOrds(blocksize=128)}, docValues:{dv=DocValuesFormat(name=Direct), soft_delete=DocValuesFormat(name=Asserting), doc=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Lucene70), content=DocValuesFormat(name=Memory), doc2d=DocValuesFormat(name=Direct)}, maxPointsInLeafNode=584, maxMBSortInHeap=7.761726159162388, sim=RandomSimilarity(queryNorm=true): {foo=DFR I(ne)BZ(0.3), id=DFR GL2, content=IB SPL-L1}, locale=es-SV, timezone=Asia/Jakarta
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=3,threads=1,free=350279888,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [TestExitableDirectoryReader, TestTrackingDirectoryWrapper, TestIOUtils, TestDocIdSetIterator, TestRollback, TestMathUtil, TestManyFields, TestMutableValues, TestLucene50StoredFieldsFormatHighCompression, TestPostingsOffsets, TestCharsRef, TestSortedSetDocValues, TestSortRandom, TestMultiMMap, TestTermRangeQuery, TestReqExclBulkScorer, TestIndexWriterMerging, TermInSetQueryTest, TestForceMergeForever, TestMultiThreadTermVectors, TestTerm, TestDocumentsWriterStallControl, TestIndexManyDocuments, TestIsCurrent, TestDirectPacked, TestWordlistLoader, TestRAMDirectory, TestSoftDeletesDirectoryReaderWrapper, TestSimpleExplanationsOfNonMatches, TestDocValuesFieldUpdates, TestSizeBoundedForceMerge, TestFilterMergePolicy, TestLockFactory, TestPackedTokenAttributeImpl, TestSpanFirstQuery, TestMultiPhraseEnum, TestAssertions, TestBufferedIndexInput, TestAllFilesCheckIndexHeader, TestParallelTermEnum, Test2BPositions, TestSpanTermQuery, TestMultiTermConstantScore, TestHighCompressionMode, TestRegexpRandom, TestTermsEnum2, TestPagedBytes, TestSpanCollection, TestRollingBuffer, TestOmitTf, TestUniqueTermCount, TestPayloadsOnVectors, TestIntroSorter, TestSameTokenSamePosition, TestExternalCodecs, TestFlushByRamOrCountsPolicy, TestMinShouldMatch2, TestNRTThreads, TestControlledRealTimeReopenThread, TestSpanNearQuery, TestMixedCodecs, TestPhrasePrefixQuery, TestMultiDocValues, TestRamUsageEstimator, Test2BSortedDocValuesOrds, TestDuelingCodecsAtNight, TestBooleanScorer, TestBoolean2ScorerSupplier, TestFieldMaskingSpanQuery, TestElevationComparator, TestIndexWriterDeleteByQuery, TestRecyclingIntBlockAllocator, TestFixedBitDocIdSet, TestLogMergePolicy, TestBytesRefArray, TestBlendedTermQuery, TestAllFilesHaveChecksumFooter, TestWildcardRandom, TestUnicodeUtil, TestNGramPhraseQuery, TestIndexSorting, TestEarlyTerminatingSortingCollector, TestLucene50LiveDocsFormat, TestCloseableThreadLocal, TestSumDocFreq, TestPositiveScoresOnlyCollector, TestUsageTrackingFilterCachingPolicy, TestPerFieldDocValuesFormat, TestOmitNorms, TestBytesRefAttImpl, TestIndexWriterUnicode, FuzzyTermOnShortTermsTest, TestDirectoryReaderReopen, TestFixedLengthBytesRefArray, TestTieredMergePolicy, TestSearchAfter, TestSloppyMath, TestDocsAndPositions, TestBinaryTerms, TestSloppyPhraseQuery2, TestNamedSPILoader, TestCachingTokenFilter, TestPolygon, TestSpanExplanations, TestCrashCausesCorruptIndex, TestDoubleRange, TestPointQueries, TestNormsFieldExistsQuery, TestShardSearching, TestMatchNoDocsQuery, TestSimpleSearchEquivalence, TestDocumentWriter, TestDocIdsWriter, TestNRTReaderCleanup, TestForUtil, TestVirtualMethod, TestIndexWriter, TestIntArrayDocIdSet, TestCharacterUtils, TestFSTs, TestLucene70NormsFormat, TestLucene70SegmentInfoFormat, TestPerFieldPostingsFormat2, TestBinaryDocument, TestDateTools, TestDocument, TestField, TestFieldType, TestFloatRange, TestIntRange, TestLatLonDocValuesField, TestLatLonPoint, TestLatLonPointDistanceSort, TestLongRange, TestGeoEncodingUtils, TestGeoUtils, Test2BBinaryDocValues, Test2BDocs, Test2BNumericDocValues, Test2BPoints, Test2BPostingsBytes, Test2BSortedDocValuesFixedSorted, Test4GBStoredFields, TestAddIndexes]
[junit4] Completed [283/486 (1!)] on J0 in 1.39s, 28 tests, 1 error <<< FAILURES!
[...truncated 11422 lines...]
[junit4] Suite: org.apache.solr.cloud.UnloadDistributedZkTest
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/init-core-data-001
[junit4] 2> 212852 WARN (SUITE-UnloadDistributedZkTest-seed#[28027971633A8DD1]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
[junit4] 2> 212852 INFO (SUITE-UnloadDistributedZkTest-seed#[28027971633A8DD1]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 212853 INFO (SUITE-UnloadDistributedZkTest-seed#[28027971633A8DD1]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=https://issues.apache.org/jira/browse/SOLR-5776)
[junit4] 2> 212854 INFO (SUITE-UnloadDistributedZkTest-seed#[28027971633A8DD1]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 212854 INFO (SUITE-UnloadDistributedZkTest-seed#[28027971633A8DD1]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
[junit4] 2> 212861 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 212862 INFO (Thread-190) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 212862 INFO (Thread-190) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 212863 ERROR (Thread-190) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 212962 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.ZkTestServer start zk server on port:60445
[junit4] 2> 212968 INFO (zkConnectionManagerCallback-317-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 212974 INFO (zkConnectionManagerCallback-319-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 212984 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 212987 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4] 2> 212989 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 212994 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 212996 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 212998 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 213000 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 213002 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 213004 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 213006 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 213008 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.AbstractZkTestCase put /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 213011 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
[junit4] 2> 213204 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 213207 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 213209 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 213209 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 213210 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7aafff82{/,null,AVAILABLE}
[junit4] 2> 213211 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@54bc32a{HTTP/1.1,[http/1.1]}{127.0.0.1:58778}
[junit4] 2> 213212 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.Server Started @213536ms
[junit4] 2> 213212 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=58778, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/control-001/cores}
[junit4] 2> 213212 ERROR (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 213212 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 213213 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.6.0
[junit4] 2> 213213 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 213213 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 213213 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-12T20:27:21.332Z
[junit4] 2> 213215 INFO (zkConnectionManagerCallback-321-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 213217 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 213218 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/control-001/solr.xml
[junit4] 2> 213221 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 213221 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 213221 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 213968 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60445/solr
[junit4] 2> 213970 INFO (zkConnectionManagerCallback-325-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 213981 INFO (zkConnectionManagerCallback-327-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 214149 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [n:127.0.0.1:58778_ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:58778_
[junit4] 2> 214150 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [n:127.0.0.1:58778_ ] o.a.s.c.Overseer Overseer (id=72325432703975428-127.0.0.1:58778_-n_0000000000) starting
[junit4] 2> 214164 INFO (zkConnectionManagerCallback-334-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 214171 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [n:127.0.0.1:58778_ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:60445/solr ready
[junit4] 2> 214173 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [n:127.0.0.1:58778_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58778_
[junit4] 2> 214173 INFO (OverseerStateUpdate-72325432703975428-127.0.0.1:58778_-n_0000000000) [n:127.0.0.1:58778_ ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:58778_
[junit4] 2> 214180 INFO (zkCallback-333-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 214183 INFO (OverseerStateUpdate-72325432703975428-127.0.0.1:58778_-n_0000000000) [n:127.0.0.1:58778_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 214202 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [n:127.0.0.1:58778_ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 214214 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [n:127.0.0.1:58778_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e
[junit4] 2> 214219 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [n:127.0.0.1:58778_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e
[junit4] 2> 214220 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [n:127.0.0.1:58778_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e
[junit4] 2> 214221 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [n:127.0.0.1:58778_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/control-001/cores
[junit4] 2> 214238 INFO (zkConnectionManagerCallback-340-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 214239 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 214241 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:60445/solr ready
[junit4] 2> 214244 INFO (qtp24597319-1203) [n:127.0.0.1:58778_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:58778_&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 214248 INFO (OverseerThreadFactory-617-thread-1-processing-n:127.0.0.1:58778_) [n:127.0.0.1:58778_ ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
[junit4] 2> 214364 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ x:control_collection_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 214364 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ x:control_collection_shard1_replica_n1] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 215396 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
[junit4] 2> 215408 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
[junit4] 2> 215497 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 215525 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
[junit4] 2> 215526 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e
[junit4] 2> 215526 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/control-001/cores/control_collection_shard1_replica_n1/data/]
[junit4] 2> 215528 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=37, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 215533 WARN (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 215592 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 215592 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 215593 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 215593 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 215595 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 215595 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@743b60f8[control_collection_shard1_replica_n1] main]
[junit4] 2> 215597 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 215597 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 215598 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 215598 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1611434865424596992
[junit4] 2> 215600 INFO (searcherExecutor-622-thread-1-processing-n:127.0.0.1:58778_ x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@743b60f8[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 215607 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
[junit4] 2> 215612 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 215612 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 215612 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:58778/control_collection_shard1_replica_n1/
[junit4] 2> 215613 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 215614 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:58778/control_collection_shard1_replica_n1/ has no replicas
[junit4] 2> 215614 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 215621 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:58778/control_collection_shard1_replica_n1/ shard1
[junit4] 2> 215775 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 215779 INFO (qtp24597319-1200) [n:127.0.0.1:58778_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1416
[junit4] 2> 215782 INFO (qtp24597319-1203) [n:127.0.0.1:58778_ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 215880 INFO (zkCallback-326-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 216255 INFO (OverseerCollectionConfigSetProcessor-72325432703975428-127.0.0.1:58778_-n_0000000000) [n:127.0.0.1:58778_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 216783 INFO (qtp24597319-1203) [n:127.0.0.1:58778_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:58778_&wt=javabin&version=2} status=0 QTime=2540
[junit4] 2> 216789 INFO (zkConnectionManagerCallback-345-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 216791 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 216792 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:60445/solr ready
[junit4] 2> 216795 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 216797 INFO (qtp24597319-1201) [n:127.0.0.1:58778_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 216801 INFO (OverseerThreadFactory-617-thread-2-processing-n:127.0.0.1:58778_) [n:127.0.0.1:58778_ ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 217008 WARN (OverseerThreadFactory-617-thread-2-processing-n:127.0.0.1:58778_) [n:127.0.0.1:58778_ ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
[junit4] 2> 217011 INFO (qtp24597319-1201) [n:127.0.0.1:58778_ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 217012 INFO (qtp24597319-1201) [n:127.0.0.1:58778_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=215
[junit4] 2> 217139 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-1-001 of type NRT
[junit4] 2> 217140 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 217141 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 217141 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 217141 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 217142 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@35f6f0b3{/,null,AVAILABLE}
[junit4] 2> 217142 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@40088a7{HTTP/1.1,[http/1.1]}{127.0.0.1:35304}
[junit4] 2> 217142 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.Server Started @217467ms
[junit4] 2> 217142 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/tempDir-001/jetty1, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=35304, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-1-001/cores}
[junit4] 2> 217143 ERROR (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 217143 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 217143 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.6.0
[junit4] 2> 217143 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 217143 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 217143 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-12T20:27:25.262Z
[junit4] 2> 217147 INFO (zkConnectionManagerCallback-347-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 217148 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 217148 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-1-001/solr.xml
[junit4] 2> 217157 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 217157 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 217157 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 217485 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60445/solr
[junit4] 2> 217487 INFO (zkConnectionManagerCallback-351-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 217491 INFO (zkConnectionManagerCallback-353-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 217504 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 217511 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 217511 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35304_
[junit4] 2> 217513 INFO (zkCallback-333-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 217518 INFO (zkCallback-344-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 217518 INFO (zkCallback-326-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 217519 INFO (zkCallback-352-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 217535 INFO (zkConnectionManagerCallback-360-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 217537 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 217538 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:60445/solr ready
[junit4] 2> 217542 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 217555 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e
[junit4] 2> 217560 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e
[junit4] 2> 217560 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e
[junit4] 2> 217562 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-1-001/cores
[junit4] 2> 217580 INFO (qtp1128205673-1256) [n:127.0.0.1:35304_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:35304_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 217583 INFO (OverseerCollectionConfigSetProcessor-72325432703975428-127.0.0.1:58778_-n_0000000000) [n:127.0.0.1:58778_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 217585 INFO (OverseerThreadFactory-617-thread-3-processing-n:127.0.0.1:58778_) [n:127.0.0.1:58778_ c:collection1 s:shard2 ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:35304_ for creating new replica of shard shard2
[junit4] 2> 217594 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ x:collection1_shard2_replica_n41] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 218620 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
[junit4] 2> 218629 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema [collection1_shard2_replica_n41] Schema name=test
[junit4] 2> 218698 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 218721 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n41' using configuration from collection collection1, trusted=true
[junit4] 2> 218721 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n41' (registry 'solr.core.collection1.shard2.replica_n41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e
[junit4] 2> 218721 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [[collection1_shard2_replica_n41] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-1-001/cores/collection1_shard2_replica_n41], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-1-001/cores/collection1_shard2_replica_n41/data/]
[junit4] 2> 218723 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=37, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 218724 WARN (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 218765 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 218765 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 218767 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 218767 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 218768 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 218768 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.s.SolrIndexSearcher Opening [Searcher@27243d3[collection1_shard2_replica_n41] main]
[junit4] 2> 218770 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 218770 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 218771 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 218771 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1611434868751728640
[junit4] 2> 218772 INFO (searcherExecutor-636-thread-1-processing-n:127.0.0.1:35304_ x:collection1_shard2_replica_n41 c:collection1 s:shard2) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [collection1_shard2_replica_n41] Registered new searcher Searcher@27243d3[collection1_shard2_replica_n41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 218779 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node42=0}, version=0}
[junit4] 2> 218784 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 218784 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 218784 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:35304/collection1_shard2_replica_n41/
[junit4] 2> 218784 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 218785 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy http://127.0.0.1:35304/collection1_shard2_replica_n41/ has no replicas
[junit4] 2> 218785 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 218790 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:35304/collection1_shard2_replica_n41/ shard2
[junit4] 2> 218894 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 218897 INFO (qtp1128205673-1254) [n:127.0.0.1:35304_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1304
[junit4] 2> 218900 INFO (qtp1128205673-1256) [n:127.0.0.1:35304_ c:collection1 ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:35304_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1320
[junit4] 2> 218997 INFO (zkCallback-352-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 219027 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-2-001 of type NRT
[junit4] 2> 219028 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 219030 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 219030 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 219030 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 219030 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@12cb1933{/,null,AVAILABLE}
[junit4] 2> 219036 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@52308e8b{HTTP/1.1,[http/1.1]}{127.0.0.1:53013}
[junit4] 2> 219036 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.Server Started @219361ms
[junit4] 2> 219036 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=53013, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-2-001/cores}
[junit4] 2> 219037 ERROR (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 219037 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 219037 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.6.0
[junit4] 2> 219037 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 219037 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 219037 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-12T20:27:27.156Z
[junit4] 2> 219040 INFO (zkConnectionManagerCallback-363-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 219042 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 219042 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-2-001/solr.xml
[junit4] 2> 219045 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 219045 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 219045 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 219586 INFO (OverseerCollectionConfigSetProcessor-72325432703975428-127.0.0.1:58778_-n_0000000000) [n:127.0.0.1:58778_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 219991 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60445/solr
[junit4] 2> 219994 INFO (zkConnectionManagerCallback-367-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 219997 INFO (zkConnectionManagerCallback-369-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 220008 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 220015 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 220015 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53013_
[junit4] 2> 220018 INFO (zkCallback-333-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 220019 INFO (zkCallback-326-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 220019 INFO (zkCallback-344-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 220019 INFO (zkCallback-352-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 220020 INFO (zkCallback-359-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 220020 INFO (zkCallback-368-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 220037 INFO (zkConnectionManagerCallback-376-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 220038 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 220040 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:60445/solr ready
[junit4] 2> 220040 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 220053 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e
[junit4] 2> 220059 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e
[junit4] 2> 220059 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e
[junit4] 2> 220060 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-2-001/cores
[junit4] 2> 220081 INFO (qtp1128205673-1260) [n:127.0.0.1:35304_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:53013_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 220087 INFO (OverseerThreadFactory-617-thread-4-processing-n:127.0.0.1:58778_) [n:127.0.0.1:58778_ c:collection1 s:shard1 ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:53013_ for creating new replica of shard shard1
[junit4] 2> 220091 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ x:collection1_shard1_replica_n43] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 220198 INFO (zkCallback-352-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 221119 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
[junit4] 2> 221130 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema [collection1_shard1_replica_n43] Schema name=test
[junit4] 2> 221199 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
[junit4] 2> 221221 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n43' using configuration from collection collection1, trusted=true
[junit4] 2> 221221 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n43' (registry 'solr.core.collection1.shard1.replica_n43') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e
[junit4] 2> 221221 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [[collection1_shard1_replica_n43] ] Opening new SolrCore at [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-2-001/cores/collection1_shard1_replica_n43], dataDir=[/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-2-001/cores/collection1_shard1_replica_n43/data/]
[junit4] 2> 221223 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=37, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 221225 WARN (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 221265 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 221265 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 221266 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 221266 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 221267 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=42, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
[junit4] 2> 221267 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.s.SolrIndexSearcher Opening [Searcher@5f79382c[collection1_shard1_replica_n43] main]
[junit4] 2> 221269 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 221269 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 221270 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 221270 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1611434871372120064
[junit4] 2> 221271 INFO (searcherExecutor-650-thread-1-processing-n:127.0.0.1:53013_ x:collection1_shard1_replica_n43 c:collection1 s:shard1) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [collection1_shard1_replica_n43] Registered new searcher Searcher@5f79382c[collection1_shard1_replica_n43] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 221277 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node44=0}, version=0}
[junit4] 2> 221282 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 221282 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 221282 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:53013/collection1_shard1_replica_n43/
[junit4] 2> 221283 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 221283 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy http://127.0.0.1:53013/collection1_shard1_replica_n43/ has no replicas
[junit4] 2> 221283 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 221288 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:53013/collection1_shard1_replica_n43/ shard1
[junit4] 2> 221391 INFO (zkCallback-352-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 221443 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 221446 INFO (qtp1243916603-1296) [n:127.0.0.1:53013_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1355
[junit4] 2> 221450 INFO (qtp1128205673-1260) [n:127.0.0.1:35304_ c:collection1 ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:53013_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1369
[junit4] 2> 221550 INFO (zkCallback-368-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 221550 INFO (zkCallback-352-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 221587 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-3-001 of type NRT
[junit4] 2> 221588 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 221590 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 221590 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 221590 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 221591 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4636edfc{/,null,AVAILABLE}
[junit4] 2> 221591 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@24b0a1ac{HTTP/1.1,[http/1.1]}{127.0.0.1:63535}
[junit4] 2> 221591 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.e.j.s.Server Started @221917ms
[junit4] 2> 221591 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/tempDir-001/jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=63535, coreRootDirectory=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-3-001/cores}
[junit4] 2> 221592 ERROR (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 221593 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 221593 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.6.0
[junit4] 2> 221593 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 221593 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 221593 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-12T20:27:29.712Z
[junit4] 2> 221595 INFO (zkConnectionManagerCallback-379-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 221597 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 221597 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-3-001/solr.xml
[junit4] 2> 221599 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 221599 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 221600 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 221791 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60445/solr
[junit4] 2> 221793 INFO (zkConnectionManagerCallback-383-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 221797 INFO (zkConnectionManagerCallback-385-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 221809 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 221816 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 221816 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63535_
[junit4] 2> 221819 INFO (zkCallback-333-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 221819 INFO (zkCallback-344-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 221819 INFO (zkCallback-368-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 221819 INFO (zkCallback-326-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 221819 INFO (zkCallback-375-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 221820 INFO (zkCallback-359-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 221820 INFO (zkCallback-352-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 221821 INFO (zkCallback-384-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 221838 INFO (zkConnectionManagerCallback-392-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 221839 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 221841 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:60445/solr ready
[junit4] 2> 221841 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 221855 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e
[junit4] 2> 221860 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e
[junit4] 2> 221860 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4092929e
[junit4] 2> 221861 INFO (TEST-UnloadDistributedZkTest.test-seed#[28027971633A8DD1]) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/../../../../../../../../../../export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.UnloadDistributedZkTest_28027971633A8DD1-001/shard-3-001/cores
[junit4] 2> 221898 INFO (qtp1128205673-1259) [n:127.0.0.1:35304_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:63535_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 221901 INFO (OverseerCollectionConfigSetProcessor-72325432703975428-127.0.0.1:58778_-n_0000000000) [n:127.0.0.1:58778_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 221903 INFO (OverseerThreadFactory-617-thread-5-processing-n:127.0.0.1:58778_) [n:127.0.0.1:58778_ c:collection1 s:shard2 ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:63535_ for creating new replica of shard shard2
[junit4] 2> 221907 INFO (qtp1999607545-1334) [n:127.0.0.1:63535_ x:collection1_shard2_replica_n45] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n45&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 222018 INFO (zkCallback-368-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 222018 INFO (zkCallback-352-thread-1) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 222928 INFO (qtp1999607545-1334) [n:127.0.0.1:63535_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.6.0
[junit4] 2> 222938 INFO (qtp1999607545-1334) [n:127.0.0.1:63535_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.s.IndexSchema [collection1_shard2_replica_n45] Schema name=test
[junit4] 2> 223006 INFO (qtp1999607545-1334) [n:127.0.0.1
[...truncated too long message...]
[junit4] OK 0.44s J0 | TestAddIndexes.testAddIndexesWithCloseNoWait
[junit4] OK 0.50s J0 | TestAddIndexes.testAddIndexesWithThreads
[junit4] OK 0.06s J0 | TestAddIndexes.testSimpleCaseCustomCodec
[junit4] OK 0.02s J0 | TestAddIndexes.testAddEmpty
[junit4] OK 0.01s J0 | TestAddIndexes.testExistingDeletes
[junit4] OK 0.12s J0 | TestAddIndexes.testWithPendingDeletes
[junit4] OK 0.02s J0 | TestAddIndexes.testNonCFSLeftovers
[junit4] OK 0.04s J0 | TestAddIndexes.testNoTailSegments
[junit4] OK 0.08s J0 | TestAddIndexes.testWithPendingDeletes2
[junit4] OK 0.01s J0 | TestAddIndexes.testIllegalIndexSortChange1
[junit4] OK 0.10s J0 | TestAddIndexes.testAddIndicesWithSoftDeletes
[junit4] OK 0.04s J0 | TestAddIndexes.testWithPendingDeletes3
[junit4] OK 0.00s J0 | TestAddIndexes.testLocksBlock
[junit4] OK 0.17s J0 | TestAddIndexes.testNoCopySegments
[junit4] OK 0.05s J0 | TestAddIndexes.testFieldNamesChanged
[junit4] OK 0.15s J0 | TestAddIndexes.testHangOnClose
[junit4] OK 0.00s J0 | TestAddIndexes.testFakeAllDeleted
[junit4] OK 0.11s J0 | TestAddIndexes.testNoMergeAfterCopy
[junit4] OK 0.08s J0 | TestAddIndexes.testAddIndexesDVUpdateNewSegmentName
[junit4] OK 0.05s J0 | TestAddIndexes.testAddIndexesWithClose
[junit4] OK 0.00s J0 | TestAddIndexes.testAddIndexMissingCodec
[junit4] OK 0.08s J0 | TestAddIndexes.testMoreMerges
[junit4] OK 0.04s J0 | TestAddIndexes.testMergeAfterCopy
[junit4] Completed [4/5] on J0 in 3.05s, 28 tests
[junit4]
[junit4] Suite: org.apache.lucene.index.TestAddIndexes
[junit4] OK 0.02s J1 | TestAddIndexes.testIllegalIndexSortChange2
[junit4] OK 0.06s J1 | TestAddIndexes.testAddSelf
[junit4] OK 0.04s J1 | TestAddIndexes.testSimpleCase
[junit4] OK 0.01s J1 | TestAddIndexes.testAddIndexesDVUpdateSameSegmentName
[junit4] OK 0.44s J1 | TestAddIndexes.testAddIndexesWithRollback
[junit4] OK 0.19s J1 | TestAddIndexes.testAddIndexesWithCloseNoWait
[junit4] OK 0.12s J1 | TestAddIndexes.testAddIndexesWithThreads
[junit4] OK 0.02s J1 | TestAddIndexes.testSimpleCaseCustomCodec
[junit4] OK 0.00s J1 | TestAddIndexes.testAddEmpty
[junit4] OK 0.00s J1 | TestAddIndexes.testExistingDeletes
[junit4] OK 0.02s J1 | TestAddIndexes.testWithPendingDeletes
[junit4] OK 0.01s J1 | TestAddIndexes.testNonCFSLeftovers
[junit4] OK 0.02s J1 | TestAddIndexes.testNoTailSegments
[junit4] OK 0.02s J1 | TestAddIndexes.testWithPendingDeletes2
[junit4] OK 0.00s J1 | TestAddIndexes.testIllegalIndexSortChange1
[junit4] OK 0.02s J1 | TestAddIndexes.testAddIndicesWithSoftDeletes
[junit4] OK 0.02s J1 | TestAddIndexes.testWithPendingDeletes3
[junit4] OK 0.00s J1 | TestAddIndexes.testLocksBlock
[junit4] OK 0.02s J1 | TestAddIndexes.testNoCopySegments
[junit4] OK 0.01s J1 | TestAddIndexes.testFieldNamesChanged
[junit4] OK 0.03s J1 | TestAddIndexes.testHangOnClose
[junit4] OK 0.00s J1 | TestAddIndexes.testFakeAllDeleted
[junit4] OK 0.02s J1 | TestAddIndexes.testNoMergeAfterCopy
[junit4] OK 0.00s J1 | TestAddIndexes.testAddIndexesDVUpdateNewSegmentName
[junit4] OK 0.02s J1 | TestAddIndexes.testAddIndexesWithClose
[junit4] OK 0.00s J1 | TestAddIndexes.testAddIndexMissingCodec
[junit4] OK 0.03s J1 | TestAddIndexes.testMoreMerges
[junit4] OK 0.02s J1 | TestAddIndexes.testMergeAfterCopy
[junit4] Completed [5/5] on J1 in 1.16s, 28 tests
[junit4]
[junit4] JVM J0: 0.69 .. 9.92 = 9.23s
[junit4] JVM J1: 0.68 .. 11.19 = 10.52s
[junit4] Execution time total: 11 seconds
[junit4] Tests summary: 5 suites, 140 tests
[junit4] Could not remove temporary path: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/lucene/build/core/test/J0 (java.nio.file.DirectoryNotEmptyException: Remaining files: [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/lucene/build/core/test/J0/temp])
[junit4] Could not remove temporary path: /export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/lucene/build/core/test/J1 (java.nio.file.DirectoryNotEmptyException: Remaining files: [/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris/lucene/build/core/test/J1/temp])
[echo] 5 slowest tests:
[junit4:tophints] 67.78s | org.apache.lucene.codecs.lucene70.TestLucene70DocValuesFormat
[junit4:tophints] 54.37s | org.apache.lucene.index.TestDocumentsWriterStallControl
[junit4:tophints] 16.29s | org.apache.lucene.search.TestLatLonPointQueries
[junit4:tophints] 13.53s | org.apache.lucene.search.TestBoolean2
[junit4:tophints] 11.73s | org.apache.lucene.search.TestSimpleExplanationsWithFillerDocs
-check-totals:
test-nocompile:
BUILD SUCCESSFUL
Total time: 13 seconds
[repro] Failures:
[repro] 0/5 failed: org.apache.lucene.index.TestAddIndexes
[repro] 0/5 failed: org.apache.solr.cloud.UnloadDistributedZkTest
[repro] 0/5 failed: org.apache.solr.core.TestLazyCores
[repro] Exiting with code 0
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
java.lang.InterruptedException: no matches found within 10000
at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2845)
at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2724)
at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2705)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3084)
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to Solaris VBOX
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:955)
at hudson.FilePath.act(FilePath.java:1070)
at hudson.FilePath.act(FilePath.java:1059)
at hudson.FilePath.validateAntFileMask(FilePath.java:2703)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
at hudson.model.Run.execute(Run.java:1840)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
at hudson.FilePath.act(FilePath.java:1072)
at hudson.FilePath.act(FilePath.java:1059)
at hudson.FilePath.validateAntFileMask(FilePath.java:2703)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
at hudson.model.Run.execute(Run.java:1840)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/*_pid*.log". Configuration error?
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=0789a77c2590f716fc3cedb247309068c3fc5d85, workspace=/export/home/jenkins/workspace/Lucene-Solr-7.x-Solaris
[WARNINGS] Computing warning deltas based on reference build #813
Recording test results
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2