You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2017/09/06 21:10:13 UTC
[JENKINS] Lucene-Solr-7.x-MacOSX (64bit/jdk1.8.0) - Build # 162 -
Failure!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-MacOSX/162/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest
Error Message:
Timeout waiting for all live and active
Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
at __randomizedtesting.SeedInfo.seed([FA8D96852459F7C3:79FBC977F220F962]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest(TestCloudRecovery.java:184)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12976 lines...]
[junit4] Suite: org.apache.solr.cloud.TestCloudRecovery
[junit4] 2> 2486449 INFO (SUITE-TestCloudRecovery-seed#[FA8D96852459F7C3]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_FA8D96852459F7C3-001/init-core-data-001
[junit4] 2> 2486449 WARN (SUITE-TestCloudRecovery-seed#[FA8D96852459F7C3]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=21 numCloses=21
[junit4] 2> 2486450 INFO (SUITE-TestCloudRecovery-seed#[FA8D96852459F7C3]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 2486452 INFO (SUITE-TestCloudRecovery-seed#[FA8D96852459F7C3]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
[junit4] 2> 2486453 INFO (SUITE-TestCloudRecovery-seed#[FA8D96852459F7C3]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_FA8D96852459F7C3-001/tempDir-001
[junit4] 2> 2486453 INFO (SUITE-TestCloudRecovery-seed#[FA8D96852459F7C3]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 2486453 INFO (Thread-3916) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2486453 INFO (Thread-3916) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 2486457 ERROR (Thread-3916) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2486563 INFO (SUITE-TestCloudRecovery-seed#[FA8D96852459F7C3]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:60590
[junit4] 2> 2486591 INFO (jetty-launcher-3008-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 2486591 INFO (jetty-launcher-3008-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 2486593 INFO (jetty-launcher-3008-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@559c41e9{/solr,null,AVAILABLE}
[junit4] 2> 2486594 INFO (jetty-launcher-3008-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@876dd2{/solr,null,AVAILABLE}
[junit4] 2> 2486595 INFO (jetty-launcher-3008-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@fc02b50{HTTP/1.1,[http/1.1]}{127.0.0.1:60592}
[junit4] 2> 2486595 INFO (jetty-launcher-3008-thread-2) [ ] o.e.j.s.Server Started @2490244ms
[junit4] 2> 2486595 INFO (jetty-launcher-3008-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@7e382d21{HTTP/1.1,[http/1.1]}{127.0.0.1:60593}
[junit4] 2> 2486595 INFO (jetty-launcher-3008-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60592}
[junit4] 2> 2486595 INFO (jetty-launcher-3008-thread-1) [ ] o.e.j.s.Server Started @2490244ms
[junit4] 2> 2486595 INFO (jetty-launcher-3008-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60593}
[junit4] 2> 2486595 ERROR (jetty-launcher-3008-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2486595 ERROR (jetty-launcher-3008-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2486596 INFO (jetty-launcher-3008-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.1.0
[junit4] 2> 2486596 INFO (jetty-launcher-3008-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 7.1.0
[junit4] 2> 2486596 INFO (jetty-launcher-3008-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2486596 INFO (jetty-launcher-3008-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2486596 INFO (jetty-launcher-3008-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 2486596 INFO (jetty-launcher-3008-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 2486596 INFO (jetty-launcher-3008-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-06T20:37:58.241Z
[junit4] 2> 2486596 INFO (jetty-launcher-3008-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-06T20:37:58.241Z
[junit4] 2> 2486598 INFO (jetty-launcher-3008-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 2486599 INFO (jetty-launcher-3008-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 2486610 INFO (jetty-launcher-3008-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60590/solr
[junit4] 2> 2486610 INFO (jetty-launcher-3008-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60590/solr
[junit4] 2> 2486744 INFO (jetty-launcher-3008-thread-1) [n:127.0.0.1:60593_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2486744 INFO (jetty-launcher-3008-thread-2) [n:127.0.0.1:60592_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2486745 INFO (jetty-launcher-3008-thread-1) [n:127.0.0.1:60593_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60593_solr
[junit4] 2> 2486745 INFO (jetty-launcher-3008-thread-2) [n:127.0.0.1:60592_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60592_solr
[junit4] 2> 2486746 INFO (jetty-launcher-3008-thread-1) [n:127.0.0.1:60593_solr ] o.a.s.c.Overseer Overseer (id=98614003505561605-127.0.0.1:60593_solr-n_0000000000) starting
[junit4] 2> 2486747 INFO (zkCallback-3019-thread-1-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2486747 INFO (zkCallback-3020-thread-1-processing-n:127.0.0.1:60592_solr) [n:127.0.0.1:60592_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2486756 INFO (jetty-launcher-3008-thread-1) [n:127.0.0.1:60593_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60593_solr
[junit4] 2> 2486758 INFO (zkCallback-3019-thread-1-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2486758 INFO (zkCallback-3020-thread-1-processing-n:127.0.0.1:60592_solr) [n:127.0.0.1:60592_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2486843 INFO (jetty-launcher-3008-thread-1) [n:127.0.0.1:60593_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60593.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@58dd76c3
[junit4] 2> 2486843 INFO (jetty-launcher-3008-thread-2) [n:127.0.0.1:60592_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60592.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@58dd76c3
[junit4] 2> 2486849 INFO (jetty-launcher-3008-thread-1) [n:127.0.0.1:60593_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60593.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@58dd76c3
[junit4] 2> 2486849 INFO (jetty-launcher-3008-thread-1) [n:127.0.0.1:60593_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60593.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@58dd76c3
[junit4] 2> 2486849 INFO (jetty-launcher-3008-thread-2) [n:127.0.0.1:60592_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60592.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@58dd76c3
[junit4] 2> 2486850 INFO (jetty-launcher-3008-thread-2) [n:127.0.0.1:60592_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60592.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@58dd76c3
[junit4] 2> 2486850 INFO (jetty-launcher-3008-thread-1) [n:127.0.0.1:60593_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_FA8D96852459F7C3-001/tempDir-001/node1/.
[junit4] 2> 2486850 INFO (jetty-launcher-3008-thread-2) [n:127.0.0.1:60592_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_FA8D96852459F7C3-001/tempDir-001/node2/.
[junit4] 2> 2486890 INFO (SUITE-TestCloudRecovery-seed#[FA8D96852459F7C3]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2486892 INFO (SUITE-TestCloudRecovery-seed#[FA8D96852459F7C3]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:60590/solr ready
[junit4] 2> 2486900 INFO (qtp1750240443-19825) [n:127.0.0.1:60592_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=0&collection.configName=config&maxShardsPerNode=2&name=collection1&nrtReplicas=0&action=CREATE&numShards=2&tlogReplicas=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2486903 INFO (OverseerThreadFactory-8313-thread-1-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 2487019 INFO (OverseerStateUpdate-98614003505561605-127.0.0.1:60593_solr-n_0000000000) [n:127.0.0.1:60593_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1_shard1_replica_t1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:60593/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 2487022 INFO (OverseerStateUpdate-98614003505561605-127.0.0.1:60593_solr-n_0000000000) [n:127.0.0.1:60593_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1_shard1_replica_t2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:60592/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 2487025 INFO (OverseerStateUpdate-98614003505561605-127.0.0.1:60593_solr-n_0000000000) [n:127.0.0.1:60593_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"collection1_shard2_replica_t4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:60593/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 2487029 INFO (OverseerStateUpdate-98614003505561605-127.0.0.1:60593_solr-n_0000000000) [n:127.0.0.1:60593_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"collection1_shard2_replica_t6",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:60592/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 2487246 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t2&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 2487246 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 2487246 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t6&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 2487246 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 2487247 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 2487249 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t4&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 2487370 INFO (zkCallback-3019-thread-1-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2487370 INFO (zkCallback-3020-thread-1-processing-n:127.0.0.1:60592_solr) [n:127.0.0.1:60592_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2487371 INFO (zkCallback-3020-thread-2-processing-n:127.0.0.1:60592_solr) [n:127.0.0.1:60592_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2487371 INFO (zkCallback-3019-thread-2-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2488268 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2488269 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2488269 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2488291 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 2488294 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.IndexSchema [collection1_shard2_replica_t4] Schema name=minimal
[junit4] 2> 2488294 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.IndexSchema [collection1_shard1_replica_t1] Schema name=minimal
[junit4] 2> 2488297 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.IndexSchema [collection1_shard2_replica_t6] Schema name=minimal
[junit4] 2> 2488298 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.IndexSchema [collection1_shard1_replica_t2] Schema name=minimal
[junit4] 2> 2488299 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 2488299 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t1' using configuration from collection collection1, trusted=true
[junit4] 2> 2488299 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 2488299 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t6' using configuration from collection collection1, trusted=true
[junit4] 2> 2488300 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60593.solr.core.collection1.shard1.replica_t1' (registry 'solr.core.collection1.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@58dd76c3
[junit4] 2> 2488300 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2488300 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SolrCore [[collection1_shard1_replica_t1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_FA8D96852459F7C3-001/tempDir-001/node1/collection1_shard1_replica_t1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_FA8D96852459F7C3-001/tempDir-001/node1/./collection1_shard1_replica_t1/data/]
[junit4] 2> 2488300 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 2488300 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t2' using configuration from collection collection1, trusted=true
[junit4] 2> 2488300 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60592.solr.core.collection1.shard2.replica_t6' (registry 'solr.core.collection1.shard2.replica_t6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@58dd76c3
[junit4] 2> 2488300 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60592.solr.core.collection1.shard1.replica_t2' (registry 'solr.core.collection1.shard1.replica_t2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@58dd76c3
[junit4] 2> 2488301 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2488301 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2488301 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrCore [[collection1_shard1_replica_t2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_FA8D96852459F7C3-001/tempDir-001/node2/collection1_shard1_replica_t2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_FA8D96852459F7C3-001/tempDir-001/node2/./collection1_shard1_replica_t2/data/]
[junit4] 2> 2488301 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrCore [[collection1_shard2_replica_t6] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_FA8D96852459F7C3-001/tempDir-001/node2/collection1_shard2_replica_t6], dataDir=[/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_FA8D96852459F7C3-001/tempDir-001/node2/./collection1_shard2_replica_t6/data/]
[junit4] 2> 2488305 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 2488305 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t4' using configuration from collection collection1, trusted=true
[junit4] 2> 2488306 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60593.solr.core.collection1.shard2.replica_t4' (registry 'solr.core.collection1.shard2.replica_t4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@58dd76c3
[junit4] 2> 2488306 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 2488306 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrCore [[collection1_shard2_replica_t4] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_FA8D96852459F7C3-001/tempDir-001/node1/collection1_shard2_replica_t4], dataDir=[/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_FA8D96852459F7C3-001/tempDir-001/node1/./collection1_shard2_replica_t4/data/]
[junit4] 2> 2488392 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2488392 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2488396 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2488396 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2488402 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2488402 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2488404 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2488404 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2488408 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2488408 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2488410 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2488410 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2488411 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 2488411 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.SolrIndexSearcher Opening [Searcher@f2f4664[collection1_shard2_replica_t4] main]
[junit4] 2> 2488411 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 2488412 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 2488413 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 2488413 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 2488413 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 2488413 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2488413 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1577824058142097408
[junit4] 2> 2488414 INFO (searcherExecutor-8321-thread-1-processing-n:127.0.0.1:60593_solr x:collection1_shard2_replica_t4 s:shard2 c:collection1 r:core_node7) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrCore [collection1_shard2_replica_t4] Registered new searcher Searcher@f2f4664[collection1_shard2_replica_t4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2488418 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7773d003[collection1_shard1_replica_t1] main]
[junit4] 2> 2488418 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.SolrIndexSearcher Opening [Searcher@5ea774ff[collection1_shard2_replica_t6] main]
[junit4] 2> 2488419 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@355d14bc[collection1_shard1_replica_t2] main]
[junit4] 2> 2488420 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
[junit4] 2> 2488420 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 2488420 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 2488421 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 2488421 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 2488421 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 2488422 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2488422 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 2488422 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1577824058151534592
[junit4] 2> 2488423 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2488422 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 2488423 INFO (searcherExecutor-8320-thread-1-processing-n:127.0.0.1:60592_solr x:collection1_shard1_replica_t2 s:shard1 c:collection1 r:core_node5) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrCore [collection1_shard1_replica_t2] Registered new searcher Searcher@355d14bc[collection1_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2488423 INFO (searcherExecutor-8319-thread-1-processing-n:127.0.0.1:60592_solr x:collection1_shard2_replica_t6 s:shard2 c:collection1 r:core_node8) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrCore [collection1_shard2_replica_t6] Registered new searcher Searcher@5ea774ff[collection1_shard2_replica_t6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2488423 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1577824058152583168
[junit4] 2> 2488423 INFO (searcherExecutor-8318-thread-1-processing-n:127.0.0.1:60593_solr x:collection1_shard1_replica_t1 s:shard1 c:collection1 r:core_node3) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SolrCore [collection1_shard1_replica_t1] Registered new searcher Searcher@7773d003[collection1_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2488423 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1577824058152583168
[junit4] 2> 2488429 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 2488429 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 2488429 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:60592/solr/collection1_shard1_replica_t2/
[junit4] 2> 2488429 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.PeerSync PeerSync: core=collection1_shard1_replica_t2 url=http://127.0.0.1:60592/solr START replicas=[http://127.0.0.1:60593/solr/collection1_shard1_replica_t1/] nUpdates=100
[junit4] 2> 2488432 INFO (qtp1347939166-19837) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 2488536 INFO (zkCallback-3019-thread-2-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2488536 INFO (zkCallback-3019-thread-1-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2488536 INFO (zkCallback-3020-thread-1-processing-n:127.0.0.1:60592_solr) [n:127.0.0.1:60592_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2488536 INFO (zkCallback-3020-thread-2-processing-n:127.0.0.1:60592_solr) [n:127.0.0.1:60592_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2488730 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.PeerSync PeerSync: core=collection1_shard1_replica_t2 url=http://127.0.0.1:60592/solr DONE. We have no versions. sync failed.
[junit4] 2> 2488730 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 2488730 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 2488730 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 2488731 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ZkController collection1_shard1_replica_t2 stopping background replication from leader
[junit4] 2> 2488736 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:60592/solr/collection1_shard1_replica_t2/ shard1
[junit4] 2> 2488841 INFO (zkCallback-3020-thread-1-processing-n:127.0.0.1:60592_solr) [n:127.0.0.1:60592_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2488841 INFO (zkCallback-3020-thread-2-processing-n:127.0.0.1:60592_solr) [n:127.0.0.1:60592_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2488841 INFO (zkCallback-3019-thread-2-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2488841 INFO (zkCallback-3019-thread-1-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2488844 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 2488846 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t2&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1600
[junit4] 2> 2488924 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 2488924 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 2488924 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:60593/solr/collection1_shard2_replica_t4/
[junit4] 2> 2488925 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.PeerSync PeerSync: core=collection1_shard2_replica_t4 url=http://127.0.0.1:60593/solr START replicas=[http://127.0.0.1:60592/solr/collection1_shard2_replica_t6/] nUpdates=100
[junit4] 2> 2488927 INFO (qtp1750240443-19831) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 2488951 INFO (zkCallback-3019-thread-1-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2488951 INFO (zkCallback-3020-thread-2-processing-n:127.0.0.1:60592_solr) [n:127.0.0.1:60592_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2488951 INFO (zkCallback-3020-thread-1-processing-n:127.0.0.1:60592_solr) [n:127.0.0.1:60592_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2488951 INFO (zkCallback-3019-thread-2-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2489228 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.PeerSync PeerSync: core=collection1_shard2_replica_t4 url=http://127.0.0.1:60593/solr DONE. We have no versions. sync failed.
[junit4] 2> 2489228 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 2489228 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 2489228 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 2489229 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ZkController collection1_shard2_replica_t4 stopping background replication from leader
[junit4] 2> 2489233 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:60593/solr/collection1_shard2_replica_t4/ shard2
[junit4] 2> 2489337 INFO (zkCallback-3019-thread-2-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2489337 INFO (zkCallback-3019-thread-1-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2489337 INFO (zkCallback-3020-thread-2-processing-n:127.0.0.1:60592_solr) [n:127.0.0.1:60592_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2489337 INFO (zkCallback-3020-thread-1-processing-n:127.0.0.1:60592_solr) [n:127.0.0.1:60592_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2489389 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 2489391 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t4&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2142
[junit4] 2> 2489431 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.ZkController collection1_shard2_replica_t6 starting background replication from leader
[junit4] 2> 2489431 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ZkController collection1_shard1_replica_t1 starting background replication from leader
[junit4] 2> 2489431 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
[junit4] 2> 2489431 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
[junit4] 2> 2489434 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
[junit4] 2> 2489434 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
[junit4] 2> 2489434 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Replica core_node7 is leader but it's state is down, skipping replication
[junit4] 2> 2489436 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Updated masterUrl to http://127.0.0.1:60592/solr/collection1_shard1_replica_t2/
[junit4] 2> 2489437 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t6&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2190
[junit4] 2> 2489437 INFO (qtp1750240443-19823) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 2489437 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2190
[junit4] 2> 2489437 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 2489437 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 2489437 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 2489437 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 2489440 INFO (qtp1750240443-19825) [n:127.0.0.1:60592_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 2489545 INFO (zkCallback-3019-thread-2-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2489545 INFO (zkCallback-3019-thread-1-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2489545 INFO (zkCallback-3020-thread-2-processing-n:127.0.0.1:60592_solr) [n:127.0.0.1:60592_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2489545 INFO (zkCallback-3020-thread-1-processing-n:127.0.0.1:60592_solr) [n:127.0.0.1:60592_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 2490442 INFO (qtp1750240443-19825) [n:127.0.0.1:60592_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=0&collection.configName=config&maxShardsPerNode=2&name=collection1&nrtReplicas=0&action=CREATE&numShards=2&tlogReplicas=2&wt=javabin&version=2} status=0 QTime=3542
[junit4] 2> 2490442 INFO (SUITE-TestCloudRecovery-seed#[FA8D96852459F7C3]-worker) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
[junit4] 2> 2490443 INFO (SUITE-TestCloudRecovery-seed#[FA8D96852459F7C3]-worker) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
[junit4] 2> 2490473 INFO (TEST-TestCloudRecovery.leaderRecoverFromLogOnStartupTest-seed#[FA8D96852459F7C3]) [ ] o.a.s.SolrTestCaseJ4 ###Starting leaderRecoverFromLogOnStartupTest
[junit4] 2> 2490481 INFO (qtp1750240443-19828) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t6] webapp=/solr path=/update params={update.distrib=FROMLEADER&_version_=-1577824060305309696&distrib.from=http://127.0.0.1:60593/solr/collection1_shard2_replica_t4/&wt=javabin&version=2}{deleteByQuery=*:* (-1577824060305309696)} 0 0
[junit4] 2> 2490485 INFO (qtp1347939166-19835) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t1] webapp=/solr path=/update params={update.distrib=FROMLEADER&_version_=-1577824060311601152&distrib.from=http://127.0.0.1:60592/solr/collection1_shard1_replica_t2/&wt=javabin&version=2}{deleteByQuery=*:* (-1577824060311601152)} 0 0
[junit4] 2> 2490485 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t2] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://127.0.0.1:60593/solr/collection1_shard2_replica_t4/&wt=javabin&version=2}{deleteByQuery=*:* (-1577824060311601152)} 0 3
[junit4] 2> 2490485 INFO (qtp1347939166-19839) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t4] webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1577824060305309696)} 0 9
[junit4] 2> 2490487 INFO (qtp1750240443-19830) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1577824060316844032,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 2490488 INFO (qtp1750240443-19830) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 2490489 INFO (qtp1750240443-19830) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 2490489 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1577824060318941184,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 2490489 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 2490489 INFO (qtp1347939166-19837) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.TestInjection Start waiting for replica in sync with leader
[junit4] 2> 2490490 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 2490489 INFO (qtp1750240443-19830) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t2] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:60593/solr/collection1_shard2_replica_t4/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 2
[junit4] 2> 2490490 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t4] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:60593/solr/collection1_shard2_replica_t4/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 2490490 INFO (qtp1750240443-19831) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.TestInjection Start waiting for replica in sync with leader
[junit4] 2> 2490492 INFO (qtp1750240443-19830) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2490492 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2490919 INFO (OverseerCollectionConfigSetProcessor-98614003505561605-127.0.0.1:60593_solr-n_0000000000) [n:127.0.0.1:60593_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 2490995 INFO (qtp1347939166-19839) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2490995 INFO (qtp1750240443-19828) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2491498 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2491498 INFO (qtp1750240443-19830) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2492002 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2492005 INFO (qtp1750240443-19825) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2492443 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Last replication failed, so I'll force replication
[junit4] 2> 2492443 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Updated masterUrl to http://127.0.0.1:60593/solr/collection1_shard2_replica_t4/
[junit4] 2> 2492444 INFO (qtp1750240443-19825) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 2492444 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 2492444 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 2492444 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 2492444 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 2492444 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 2492444 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 2492444 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 2492444 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 2492444 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 2492444 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 2492444 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 2492444 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@e2aa160 commitCommandVersion:0
[junit4] 2> 2492464 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.SolrIndexSearcher Opening [Searcher@4d2ca10[collection1_shard2_replica_t6] main]
[junit4] 2> 2492465 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 2492466 INFO (searcherExecutor-8319-thread-1-processing-n:127.0.0.1:60592_solr x:collection1_shard2_replica_t6 s:shard2 c:collection1 r:core_node8) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrCore [collection1_shard2_replica_t6] Registered new searcher Searcher@4d2ca10[collection1_shard2_replica_t6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 2492512 INFO (qtp1347939166-19835) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2492512 INFO (qtp1750240443-19830) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2493015 INFO (qtp1750240443-19823) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2493015 INFO (qtp1347939166-19839) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2493518 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2493518 INFO (qtp1347939166-19835) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2494028 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2494028 INFO (qtp1750240443-19823) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2494531 INFO (qtp1347939166-19838) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2494535 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2495033 INFO (qtp1347939166-19835) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2495037 INFO (qtp1750240443-19825) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2495435 INFO (qtp1347939166-19838) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 2495435 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 2495435 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 2495436 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 2495436 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's version: 1504730284090
[junit4] 2> 2495436 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 2495437 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 2495437 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 2495437 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 2495437 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 2495536 INFO (qtp1347939166-19835) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2495540 INFO (qtp1750240443-19828) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2496039 INFO (qtp1347939166-19839) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2496046 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2496542 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2496552 INFO (qtp1750240443-19830) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2497044 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2497056 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2497547 INFO (qtp1347939166-19838) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2497558 INFO (qtp1750240443-19825) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2498049 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2498064 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2498446 INFO (qtp1750240443-19828) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 2498446 INFO (qtp1347939166-19838) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 2498446 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 2498446 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 2498446 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 2498446 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 2498446 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 2498446 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's version: 1504730284090
[junit4] 2> 2498446 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 2498446 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 2498557 INFO (qtp1347939166-19835) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2498567 INFO (qtp1750240443-19825) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2499060 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2499069 INFO (qtp1750240443-19828) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2499562 INFO (qtp1347939166-19839) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2499572 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2500070 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2500079 INFO (qtp1750240443-19823) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2500573 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2500592 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2501076 INFO (qtp1347939166-19835) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2501095 INFO (qtp1750240443-19823) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2501434 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 2501434 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 2501434 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 2501435 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's generation: 2
[junit4] 2> 2501435 INFO (indexFetcher-8341-thread-1) [n:127.0.0.1:60592_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.IndexFetcher Slave's version: 1504730284090
[junit4] 2> 2501447 INFO (qtp1750240443-19830) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 2501448 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 2501448 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 2501448 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 2501448 INFO (indexFetcher-8340-thread-1) [n:127.0.0.1:60593_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 2501584 INFO (qtp1347939166-19826) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2501610 INFO (qtp1750240443-19829) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2502088 INFO (qtp1347939166-19838) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2502123 INFO (qtp1750240443-19830) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2502599 INFO (qtp1347939166-19839) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2502626 INFO (qtp1750240443-19823) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2503111 INFO (qtp1347939166-19834) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2503138 INFO (qtp1750240443-19827) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2503621 INFO (qtp1347939166-19835) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2503650 INFO (qtp1750240443-19823) [n:127.0.0.1:60592_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 2504133 INF
[...truncated too long message...]
ateExecutor-3042-thread-2-processing-n:127.0.0.1:60593_solr x:collection1_shard2_replica_t4 s:shard2 c:collection1 r:core_node7) [n:127.0.0.1:60593_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.DefaultSolrCoreState Skipping recovery because Solr is shutdown
[junit4] 2> 2663056 INFO (jetty-closer-3009-thread-1) [ ] o.a.s.c.Overseer Overseer (id=98614003505561617-127.0.0.1:60593_solr-n_0000000004) closing
[junit4] 2> 2663056 INFO (OverseerStateUpdate-98614003505561617-127.0.0.1:60593_solr-n_0000000004) [n:127.0.0.1:60593_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:60593_solr
[junit4] 2> 2663057 WARN (zkCallback-3045-thread-2-processing-n:127.0.0.1:60593_solr) [n:127.0.0.1:60593_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 2663058 INFO (jetty-closer-3009-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@154e994a{/solr,null,UNAVAILABLE}
[junit4] 2> 2663059 ERROR (SUITE-TestCloudRecovery-seed#[FA8D96852459F7C3]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2663059 INFO (SUITE-TestCloudRecovery-seed#[FA8D96852459F7C3]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60590 60590
[junit4] 2> 2663069 INFO (Thread-3916) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60590 60590
[junit4] 2> 2663327 WARN (Thread-3916) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 7 /solr/aliases.json
[junit4] 2> 7 /solr/clusterprops.json
[junit4] 2> 6 /solr/security.json
[junit4] 2> 6 /solr/configs/config
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 151 /solr/collections/collection1/state.json
[junit4] 2> 7 /solr/clusterstate.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 7 /solr/collections
[junit4] 2> 3 /solr/overseer/queue
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/overseer/collection-queue-work
[junit4] 2> 2 /solr/overseer/queue-work
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_FA8D96852459F7C3-001
[junit4] 2> Sep 06, 2017 8:40:54 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {id=PostingsFormat(name=LuceneVarGapDocFreqInterval)}, docValues:{_version_=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1199, maxMBSortInHeap=7.178653014763332, sim=RandomSimilarity(queryNorm=true): {}, locale=sq-AL, timezone=America/Argentina/Rio_Gallegos
[junit4] 2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=3,threads=1,free=161124568,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [StatsReloadRaceTest, ZkStateReaderTest, HLLSerializationTest, TestPKIAuthenticationPlugin, TestCloudSearcherWarming, ShardSplitTest, TestHdfsUpdateLog, TestHdfsCloudBackupRestore, TestIBSimilarityFactory, AutoScalingHandlerTest, TestJavabinTupleStreamParser, CoreMergeIndexesAdminHandlerTest, TlogReplayBufferedWhileIndexingTest, FastVectorHighlighterTest, TestTolerantUpdateProcessorCloud, TestFoldingMultitermQuery, TestMissingGroups, TestSmileRequest, TestZkAclsWithHadoopAuth, SyncSliceTest, CheckHdfsIndexTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestCloudDeleteByQuery, DistributedSpellCheckComponentTest, TestReplicationHandlerBackup, SolrGangliaReporterTest, TestSolrConfigHandlerConcurrent, ResponseHeaderTest, CurrencyFieldTypeTest, TestQuerySenderNoQuery, TestBulkSchemaAPI, TestCSVResponseWriter, HdfsRestartWhileUpdatingTest, TestSearcherReuse, SuggesterTest, HdfsCollectionsAPIDistributedZkTest, IndexBasedSpellCheckerTest, TestFunctionQuery, AsyncCallRequestStatusResponseTest, TestFreeTextSuggestions, CdcrReplicationHandlerTest, DefaultValueUpdateProcessorTest, DirectUpdateHandlerTest, TestPhraseSuggestions, DocExpirationUpdateProcessorFactoryTest, TestCustomStream, TestPayloadCheckQParserPlugin, TestScoreJoinQPScore, TestFieldCollectionResource, DistanceFunctionTest, ClusterStateTest, CollectionsAPISolrJTest, TestLeaderElectionZkExpiry, ReturnFieldsTest, AssignBackwardCompatibilityTest, TestSolrCloudWithKerberosAlt, AlternateDirectoryTest, MoveReplicaHDFSUlogDirTest, TestCustomSort, TestHashQParserPlugin, TestOmitPositions, TestReplicationHandler, TestCloudSchemaless, TestManagedSchemaAPI, TestSchemaNameResource, TestPointFields, AddBlockUpdateTest, BooleanFieldTest, UUIDUpdateProcessorFallbackTest, RequiredFieldsTest, TestLegacyFieldCache, PKIAuthenticationIntegrationTest, XmlUpdateRequestHandlerTest, SolrXmlInZkTest, DistributedExpandComponentTest, TestBackupRepositoryFactory, HdfsLockFactoryTest, SolrIndexSplitterTest, HdfsRecoverLeaseTest, JavabinLoaderTest, OverseerTest, SearchHandlerTest, DirectoryFactoryTest, TestUtils, DocValuesMissingTest, TestPrepRecovery, TestNonDefinedSimilarityFactory, TestTrieFacet, BlockJoinFacetRandomTest, TestBinaryField, StressHdfsTest, DirectUpdateHandlerOptimizeTest, AnalysisAfterCoreReloadTest, TestNamedUpdateProcessors, SolrCoreMetricManagerTest, TestSolrDeletionPolicy2, TestSolrQueryResponse, SpatialHeatmapFacetsTest, RankQueryTest, NotRequiredUniqueKeyTest, TestLegacyTerms, TestHighlightDedupGrouping, TestFuzzyAnalyzedSuggestions, TestNoOpRegenerator, DistributedQueryComponentOptimizationTest, WordBreakSolrSpellCheckerTest, DeleteShardTest, TestRandomDVFaceting, TestQueryWrapperFilter, TestMacros, BinaryUpdateRequestHandlerTest, OutputWriterTest, ParsingFieldUpdateProcessorsTest, HighlighterTest, HdfsThreadLeakTest, TestFilteredDocIdSet, TestOrdValues, TestJettySolrRunner, TestJsonFacets, TestRecoveryHdfs, TestCursorMarkWithoutUniqueKey, TestXmlQParserPlugin, DocValuesTest, TestExpandComponent, DistributedFacetPivotWhiteBoxTest, DateMathParserTest, QueryElevationComponentTest, TestRandomRequestDistribution, TestDynamicFieldCollectionResource, SpellCheckCollatorWithCollapseTest, ClassificationUpdateProcessorTest, TestStandardQParsers, TestDeleteCollectionOnDownNodes, OverseerCollectionConfigSetProcessorTest, TestSolrCoreProperties, TestPseudoReturnFields, DistribCursorPagingTest, TestExtendedDismaxParser, TestImplicitCoreProperties, BlockCacheTest, LargeFieldTest, TestFieldCacheSort, DeleteNodeTest, TestDocBasedVersionConstraints, TestManagedResourceStorage, TestNumericRangeQuery32, TestTolerantSearch, SimpleFacetsTest, TestInPlaceUpdatesStandalone, SolrJmxReporterCloudTest, MissingSegmentRecoveryTest, JvmMetricsTest, SolrShardReporterTest, TestSizeLimitedDistributedMap, OverseerRolesTest, TestImpersonationWithHadoopAuth, LeaderInitiatedRecoveryOnShardRestartTest, TestSha256AuthenticationProvider, TestStressReorder, CdcrBootstrapTest, SortByFunctionTest, TestTestInjection, ClassificationUpdateProcessorFactoryTest, RecoveryZkTest, ManagedSchemaRoundRobinCloudTest, TestClassicSimilarityFactory, ConfigureRecoveryStrategyTest, TestLegacyFieldReuse, HardAutoCommitTest, MetricsHandlerTest, TestSchemalessBufferedUpdates, GraphQueryTest, CoreAdminCreateDiscoverTest, PathHierarchyTokenizerFactoryTest, TestLMDirichletSimilarityFactory, ActionThrottleTest, BasicDistributedZkTest, ShowFileRequestHandlerTest, SimplePostToolTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, SpellCheckCollatorTest, SolrRequestParserTest, SuggesterTSTTest, TestStressLucene, TestTrie, SuggesterWFSTTest, NoCacheHeaderTest, TestAtomicUpdateErrorCases, QueryEqualityTest, SolrInfoBeanTest, DisMaxRequestHandlerTest, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, RequestHandlersTest, TestIndexingPerformance, FieldAnalysisRequestHandlerTest, IndexSchemaTest, TestCollationField, MBeansHandlerTest, HighlighterConfigTest, SOLR749Test, TestQuerySenderListener, ResponseLogComponentTest, BadComponentTest, TestStressRecovery, TestXIncludeConfig, TestDFRSimilarityFactory, TestFastWriter, TimeZoneUtilsTest, TestFastOutputStream, ScriptEngineTest, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest, BaseCdcrDistributedZkTest, CdcrRequestHandlerTest, CdcrVersionReplicationTest, ChaosMonkeySafeLeaderWithPullReplicasTest, ConfigSetsAPITest, CreateCollectionCleanupTest, DeleteInactiveReplicaTest, DistribJoinFromCollectionTest, DistributedVersionInfoTest, LeaderElectionContextKeyTest, LeaderFailoverAfterPartitionTest, LeaderFailureAfterFreshStartTest, MultiThreadedOCPTest, RemoteQueryErrorTest, ReplaceNodeTest, RestartWhileUpdatingTest, SSLMigrationTest, SharedFSAutoReplicaFailoverTest, SharedFSAutoReplicaFailoverUtilsTest, SimpleCollectionCreateDeleteTest, SolrCLIZkUtilsTest, SolrCloudExampleTest, TestCloudRecovery]
[junit4] Completed [548/731 (1!)] on J0 in 177.15s, 2 tests, 1 failure <<< FAILURES!
[...truncated 46827 lines...]
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/build.xml:810: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/build.xml:690: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/build.xml:678: Source checkout is modified!!! Offending files:
* solr/licenses/commons-math3-3.6.1.jar.sha1
Total time: 92 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
[JENKINS] Lucene-Solr-7.x-MacOSX (64bit/jdk1.8.0) - Build # 164 -
Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-MacOSX/164/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC
2 tests failed.
FAILED: org.apache.solr.cloud.CustomCollectionTest.testCustomCollectionsAPI
Error Message:
Error from server at http://127.0.0.1:60072/solr: ADDREPLICA failed to create replica
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:60072/solr: ADDREPLICA failed to create replica
at __randomizedtesting.SeedInfo.seed([9FE8A28912BCCBEA:F5092CE22F267D92]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:627)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:253)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:242)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1121)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:862)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:793)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:178)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:195)
at org.apache.solr.cloud.CustomCollectionTest.testCustomCollectionsAPI(CustomCollectionTest.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.TestTlogReplica.testRecovery
Error Message:
Can not find doc 8 in https://127.0.0.1:52496/solr
Stack Trace:
java.lang.AssertionError: Can not find doc 8 in https://127.0.0.1:52496/solr
at __randomizedtesting.SeedInfo.seed([9FE8A28912BCCBEA:5E18DB253FEC014D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:868)
at org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:589)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 11169 lines...]
[junit4] Suite: org.apache.solr.cloud.TestTlogReplica
[junit4] 2> 0 INFO (SUITE-TestTlogReplica-seed#[9FE8A28912BCCBEA]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_9FE8A28912BCCBEA-001/init-core-data-001
[junit4] 2> 595 INFO (SUITE-TestTlogReplica-seed#[9FE8A28912BCCBEA]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 899 INFO (SUITE-TestTlogReplica-seed#[9FE8A28912BCCBEA]-worker) [ ] o.e.j.u.log Logging initialized @7197ms
[junit4] 2> 924 INFO (SUITE-TestTlogReplica-seed#[9FE8A28912BCCBEA]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
[junit4] 2> 1549 INFO (SUITE-TestTlogReplica-seed#[9FE8A28912BCCBEA]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_9FE8A28912BCCBEA-001/tempDir-001
[junit4] 2> 1587 INFO (SUITE-TestTlogReplica-seed#[9FE8A28912BCCBEA]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1591 INFO (Thread-1) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1591 INFO (Thread-1) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1937 INFO (SUITE-TestTlogReplica-seed#[9FE8A28912BCCBEA]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:52494
[junit4] 2> 2226 ERROR (Thread-1) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2685 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x15e642cda400000, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 3768 INFO (jetty-launcher-1-thread-1) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 3775 INFO (jetty-launcher-1-thread-2) [ ] o.e.j.s.Server jetty-9.3.20.v20170531
[junit4] 2> 3921 INFO (jetty-launcher-1-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4727fc1e{/solr,null,AVAILABLE}
[junit4] 2> 3924 INFO (jetty-launcher-1-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5e4ad353{/solr,null,AVAILABLE}
[junit4] 2> 4007 INFO (jetty-launcher-1-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4e171509{SSL,[ssl, http/1.1]}{127.0.0.1:52496}
[junit4] 2> 4008 INFO (jetty-launcher-1-thread-1) [ ] o.e.j.s.Server Started @10312ms
[junit4] 2> 4008 INFO (jetty-launcher-1-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52496}
[junit4] 2> 4010 INFO (jetty-launcher-1-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4a91f0aa{SSL,[ssl, http/1.1]}{127.0.0.1:52497}
[junit4] 2> 4021 INFO (jetty-launcher-1-thread-2) [ ] o.e.j.s.Server Started @10325ms
[junit4] 2> 4025 INFO (jetty-launcher-1-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=52497}
[junit4] 2> 4210 ERROR (jetty-launcher-1-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 4211 INFO (jetty-launcher-1-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 4220 ERROR (jetty-launcher-1-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 4226 INFO (jetty-launcher-1-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 7.1.0
[junit4] 2> 4232 INFO (jetty-launcher-1-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 4232 INFO (jetty-launcher-1-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 4233 INFO (jetty-launcher-1-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 4232 INFO (jetty-launcher-1-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null, Default config dir: null
[junit4] 2> 4252 INFO (jetty-launcher-1-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-09T01:06:56.982Z
[junit4] 2> 4253 INFO (jetty-launcher-1-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2017-09-09T01:06:56.985Z
[junit4] 2> 4436 INFO (jetty-launcher-1-thread-2) [ ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
[junit4] 2> 4484 INFO (jetty-launcher-1-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 4490 INFO (jetty-launcher-1-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 6842 INFO (jetty-launcher-1-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52494/solr
[junit4] 2> 6842 INFO (jetty-launcher-1-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:52494/solr
[junit4] 2> 7634 INFO (jetty-launcher-1-thread-2) [n:127.0.0.1:52497_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 7635 INFO (jetty-launcher-1-thread-1) [n:127.0.0.1:52496_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 7638 INFO (jetty-launcher-1-thread-2) [n:127.0.0.1:52497_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:52497_solr
[junit4] 2> 7642 INFO (jetty-launcher-1-thread-1) [n:127.0.0.1:52496_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52496_solr
[junit4] 2> 7649 INFO (zkCallback-12-thread-1-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 7650 INFO (zkCallback-13-thread-1-processing-n:127.0.0.1:52496_solr) [n:127.0.0.1:52496_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 7653 INFO (jetty-launcher-1-thread-2) [n:127.0.0.1:52497_solr ] o.a.s.c.Overseer Overseer (id=98626385651695621-127.0.0.1:52497_solr-n_0000000000) starting
[junit4] 2> 8358 INFO (jetty-launcher-1-thread-2) [n:127.0.0.1:52497_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:52497_solr
[junit4] 2> 8366 INFO (zkCallback-12-thread-1-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 8367 INFO (zkCallback-13-thread-1-processing-n:127.0.0.1:52496_solr) [n:127.0.0.1:52496_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 9259 INFO (jetty-launcher-1-thread-2) [n:127.0.0.1:52497_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52497.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3139eb6d
[junit4] 2> 9263 INFO (jetty-launcher-1-thread-2) [n:127.0.0.1:52497_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52497.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3139eb6d
[junit4] 2> 9264 INFO (jetty-launcher-1-thread-2) [n:127.0.0.1:52497_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52497.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3139eb6d
[junit4] 2> 9334 INFO (jetty-launcher-1-thread-1) [n:127.0.0.1:52496_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52496.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3139eb6d
[junit4] 2> 9359 INFO (jetty-launcher-1-thread-1) [n:127.0.0.1:52496_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52496.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3139eb6d
[junit4] 2> 9359 INFO (jetty-launcher-1-thread-1) [n:127.0.0.1:52496_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52496.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3139eb6d
[junit4] 2> 9764 INFO (jetty-launcher-1-thread-2) [n:127.0.0.1:52497_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_9FE8A28912BCCBEA-001/tempDir-001/node2/.
[junit4] 2> 9772 INFO (jetty-launcher-1-thread-1) [n:127.0.0.1:52496_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_9FE8A28912BCCBEA-001/tempDir-001/node1/.
[junit4] 2> 10271 INFO (SUITE-TestTlogReplica-seed#[9FE8A28912BCCBEA]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 10273 INFO (SUITE-TestTlogReplica-seed#[9FE8A28912BCCBEA]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:52494/solr ready
[junit4] 2> 10281 INFO (SUITE-TestTlogReplica-seed#[9FE8A28912BCCBEA]-worker) [ ] o.a.s.c.TestTlogReplica Using legacyCloud?: false
[junit4] 2> 12031 INFO (qtp111205268-35) [n:127.0.0.1:52497_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterprop with params val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 12045 INFO (qtp111205268-35) [n:127.0.0.1:52497_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={val=false&name=legacyCloud&action=CLUSTERPROP&wt=javabin&version=2} status=0 QTime=237
[junit4] 2> 12321 INFO (TEST-TestTlogReplica.testRealTimeGet-seed#[9FE8A28912BCCBEA]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testRealTimeGet
[junit4] 2> 12345 INFO (qtp111205268-36) [n:127.0.0.1:52497_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=2&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_real_time_get&nrtReplicas=2&action=CREATE&numShards=1&tlogReplicas=1&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 12395 INFO (OverseerThreadFactory-10-thread-1-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.CreateCollectionCmd Create collection tlog_replica_test_real_time_get
[junit4] 2> 12406 WARN (OverseerThreadFactory-10-thread-1-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.CreateCollectionCmd Specified number of replicas of 3 on collection tlog_replica_test_real_time_get is higher than the number of Solr instances currently live or live and part of your createNodeSet(2). It's unusual to run two replica of the same slice on the same Solr-instance.
[junit4] 2> 12549 INFO (OverseerStateUpdate-98626385651695621-127.0.0.1:52497_solr-n_0000000000) [n:127.0.0.1:52497_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_real_time_get",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"tlog_replica_test_real_time_get_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:52496/solr",
[junit4] 2> "type":"NRT"}
[junit4] 2> 12563 INFO (OverseerStateUpdate-98626385651695621-127.0.0.1:52497_solr-n_0000000000) [n:127.0.0.1:52497_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_real_time_get",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"tlog_replica_test_real_time_get_shard1_replica_n3",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:52497/solr",
[junit4] 2> "type":"NRT"}
[junit4] 2> 12573 INFO (OverseerStateUpdate-98626385651695621-127.0.0.1:52497_solr-n_0000000000) [n:127.0.0.1:52497_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_real_time_get",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"tlog_replica_test_real_time_get_shard1_replica_t4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:52496/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 12753 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_n3&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 12760 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 12770 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_n1&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 12774 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_t4&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 12774 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 12907 INFO (zkCallback-12-thread-1-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 12908 INFO (zkCallback-12-thread-2-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 12908 INFO (zkCallback-13-thread-1-processing-n:127.0.0.1:52496_solr) [n:127.0.0.1:52496_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 13888 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 13888 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 13909 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 14035 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.s.IndexSchema [tlog_replica_test_real_time_get_shard1_replica_n3] Schema name=minimal
[junit4] 2> 14042 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.s.IndexSchema [tlog_replica_test_real_time_get_shard1_replica_t4] Schema name=minimal
[junit4] 2> 14050 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.s.IndexSchema [tlog_replica_test_real_time_get_shard1_replica_n1] Schema name=minimal
[junit4] 2> 14219 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 14219 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 14220 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 14221 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_real_time_get_shard1_replica_t4' using configuration from collection tlog_replica_test_real_time_get, trusted=true
[junit4] 2> 14222 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_real_time_get_shard1_replica_n1' using configuration from collection tlog_replica_test_real_time_get, trusted=true
[junit4] 2> 14223 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_real_time_get_shard1_replica_n3' using configuration from collection tlog_replica_test_real_time_get, trusted=true
[junit4] 2> 14338 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52496.solr.core.tlog_replica_test_real_time_get.shard1.replica_n1' (registry 'solr.core.tlog_replica_test_real_time_get.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3139eb6d
[junit4] 2> 14339 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52497.solr.core.tlog_replica_test_real_time_get.shard1.replica_n3' (registry 'solr.core.tlog_replica_test_real_time_get.shard1.replica_n3') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3139eb6d
[junit4] 2> 14339 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52496.solr.core.tlog_replica_test_real_time_get.shard1.replica_t4' (registry 'solr.core.tlog_replica_test_real_time_get.shard1.replica_t4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3139eb6d
[junit4] 2> 14358 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 14362 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 14364 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 14375 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.SolrCore [[tlog_replica_test_real_time_get_shard1_replica_t4] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_9FE8A28912BCCBEA-001/tempDir-001/node1/tlog_replica_test_real_time_get_shard1_replica_t4], dataDir=[/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_9FE8A28912BCCBEA-001/tempDir-001/node1/./tlog_replica_test_real_time_get_shard1_replica_t4/data/]
[junit4] 2> 14375 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.c.SolrCore [[tlog_replica_test_real_time_get_shard1_replica_n3] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_9FE8A28912BCCBEA-001/tempDir-001/node2/tlog_replica_test_real_time_get_shard1_replica_n3], dataDir=[/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_9FE8A28912BCCBEA-001/tempDir-001/node2/./tlog_replica_test_real_time_get_shard1_replica_n3/data/]
[junit4] 2> 14375 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.SolrCore [[tlog_replica_test_real_time_get_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_9FE8A28912BCCBEA-001/tempDir-001/node1/tlog_replica_test_real_time_get_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_9FE8A28912BCCBEA-001/tempDir-001/node1/./tlog_replica_test_real_time_get_shard1_replica_n1/data/]
[junit4] 2> 17799 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 17799 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 17803 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 17803 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 17806 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 17809 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 17844 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 17845 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 17853 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 17853 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 17848 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 17854 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 17967 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.s.SolrIndexSearcher Opening [Searcher@35d0336[tlog_replica_test_real_time_get_shard1_replica_t4] main]
[junit4] 2> 17967 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1d107e5c[tlog_replica_test_real_time_get_shard1_replica_n1] main]
[junit4] 2> 17967 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.s.SolrIndexSearcher Opening [Searcher@1809fe0a[tlog_replica_test_real_time_get_shard1_replica_n3] main]
[junit4] 2> 18047 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 18048 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 18056 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 18152 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 18153 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 18153 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 18291 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 18292 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 18291 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 19101 INFO (searcherExecutor-17-thread-1-processing-n:127.0.0.1:52497_solr x:tlog_replica_test_real_time_get_shard1_replica_n3 s:shard1 c:tlog_replica_test_real_time_get r:core_node5) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_n3] Registered new searcher Searcher@1809fe0a[tlog_replica_test_real_time_get_shard1_replica_n3] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 19114 INFO (searcherExecutor-15-thread-1-processing-n:127.0.0.1:52496_solr x:tlog_replica_test_real_time_get_shard1_replica_t4 s:shard1 c:tlog_replica_test_real_time_get r:core_node6) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_t4] Registered new searcher Searcher@35d0336[tlog_replica_test_real_time_get_shard1_replica_t4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 19117 INFO (searcherExecutor-16-thread-1-processing-n:127.0.0.1:52496_solr x:tlog_replica_test_real_time_get_shard1_replica_n1 s:shard1 c:tlog_replica_test_real_time_get r:core_node2) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_n1] Registered new searcher Searcher@1d107e5c[tlog_replica_test_real_time_get_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 19149 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578022188505628672
[junit4] 2> 19153 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578022188509822976
[junit4] 2> 19154 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578022188510871552
[junit4] 2> 19190 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=3 found=2 timeoutin=9997ms
[junit4] 2> 19205 INFO (zkCallback-12-thread-2-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 19205 INFO (zkCallback-12-thread-1-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 19206 INFO (zkCallback-13-thread-1-processing-n:127.0.0.1:52496_solr) [n:127.0.0.1:52496_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 19698 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 19698 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 19699 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/
[junit4] 2> 19711 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_real_time_get_shard1_replica_t4 url=https://127.0.0.1:52496/solr START replicas=[https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_n1/, https://127.0.0.1:52497/solr/tlog_replica_test_real_time_get_shard1_replica_n3/] nUpdates=100
[junit4] 2> 19737 INFO (qtp1877440554-27) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=14
[junit4] 2> 19737 INFO (qtp111205268-34) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n3] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=14
[junit4] 2> 20014 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.PeerSync PeerSync: core=tlog_replica_test_real_time_get_shard1_replica_t4 url=https://127.0.0.1:52496/solr DONE. We have no versions. sync failed.
[junit4] 2> 20017 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 20017 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 20017 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 20018 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ZkController tlog_replica_test_real_time_get_shard1_replica_t4 stopping background replication from leader
[junit4] 2> 20066 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/ shard1
[junit4] 2> 20182 INFO (zkCallback-13-thread-1-processing-n:127.0.0.1:52496_solr) [n:127.0.0.1:52496_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 20182 INFO (zkCallback-12-thread-2-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 20182 INFO (zkCallback-12-thread-1-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 20195 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_n3&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=7445
[junit4] 2> 20224 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 20229 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node6&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_t4&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=7456
[junit4] 2> 20233 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=conf&newCollection=true&name=tlog_replica_test_real_time_get_shard1_replica_n1&action=CREATE&numShards=1&collection=tlog_replica_test_real_time_get&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=7462
[junit4] 2> 20262 INFO (qtp111205268-36) [n:127.0.0.1:52497_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 20360 INFO (zkCallback-12-thread-1-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 20360 INFO (zkCallback-12-thread-2-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 20360 INFO (zkCallback-13-thread-1-processing-n:127.0.0.1:52496_solr) [n:127.0.0.1:52496_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_real_time_get/state.json] for collection [tlog_replica_test_real_time_get] has occurred - updating... (live nodes size: [2])
[junit4] 2> 20412 INFO (OverseerCollectionConfigSetProcessor-98626385651695621-127.0.0.1:52497_solr-n_0000000000) [n:127.0.0.1:52497_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 21265 INFO (qtp111205268-36) [n:127.0.0.1:52497_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&replicationFactor=2&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_real_time_get&nrtReplicas=2&action=CREATE&numShards=1&tlogReplicas=1&wt=javabin&version=2} status=0 QTime=8919
[junit4] 2> 22083 INFO (qtp111205268-35) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n3] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/&wt=javabin&version=2}{add=[0 (1578022190935179264)]} 0 11
[junit4] 2> 22085 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/&wt=javabin&version=2}{add=[0 (1578022190935179264)]} 0 6
[junit4] 2> 22087 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_n1/&wt=javabin&version=2}{add=[0 (1578022190935179264)]} 0 632
[junit4] 2> 22089 INFO (qtp1877440554-27) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[0]} 0 779
[junit4] 2> 22168 INFO (qtp111205268-37) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n3] webapp=/solr path=/get params={qt=/get&_stateVer_=tlog_replica_test_real_time_get:5&ids=0&wt=javabin&version=2} status=0 QTime=52
[junit4] 2> 22210 INFO (qtp1877440554-25) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&ids=0&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 22213 INFO (qtp111205268-38) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n3] webapp=/solr path=/get params={qt=/get&ids=0&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 22219 INFO (qtp1877440554-26) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:52497/solr/tlog_replica_test_real_time_get_shard1_replica_n3/|https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_n1/|https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/
[junit4] 2> 22228 INFO (qtp111205268-36) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n3] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1504919234963&ids=0&isShard=true&shard.url=https://127.0.0.1:52497/solr/tlog_replica_test_real_time_get_shard1_replica_n3/|https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_n1/|https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 22229 INFO (qtp1877440554-26) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/get params={qt=/get&ids=0&wt=javabin&version=2} status=0 QTime=14
[junit4] 2> 22256 INFO (qtp111205268-37) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n3] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/&wt=javabin&version=2}{add=[1 (1578022191747825664)]} 0 5
[junit4] 2> 22259 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/&wt=javabin&version=2}{add=[1 (1578022191747825664)]} 0 8
[junit4] 2> 22260 INFO (qtp1877440554-27) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=https://127.0.0.1:52497/solr/tlog_replica_test_real_time_get_shard1_replica_n3/&wt=javabin&version=2}{add=[1 (1578022191747825664)]} 0 19
[junit4] 2> 22261 INFO (qtp111205268-35) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n3] webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 28
[junit4] 2> 22264 INFO (qtp111205268-38) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n3] webapp=/solr path=/get params={qt=/get&_stateVer_=tlog_replica_test_real_time_get:5&ids=1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 22267 INFO (qtp1877440554-25) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&ids=1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 22271 INFO (qtp111205268-33) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n3] webapp=/solr path=/get params={qt=/get&ids=1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 22274 INFO (qtp1877440554-26) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/|https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_n1/|https://127.0.0.1:52497/solr/tlog_replica_test_real_time_get_shard1_replica_n3/
[junit4] 2> 22287 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1504919235022&ids=1&isShard=true&shard.url=https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/|https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_n1/|https://127.0.0.1:52497/solr/tlog_replica_test_real_time_get_shard1_replica_n3/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=1
[junit4] 2> 22288 INFO (qtp1877440554-26) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/get params={qt=/get&ids=1&wt=javabin&version=2} status=0 QTime=14
[junit4] 2> 22299 INFO (qtp111205268-34) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n3] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/&wt=javabin&version=2}{add=[2 (1578022191801303040)]} 0 4
[junit4] 2> 22300 INFO (qtp1877440554-25) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/&wt=javabin&version=2}{add=[2 (1578022191801303040)]} 0 6
[junit4] 2> 22302 INFO (qtp1877440554-27) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.u.p.LogUpdateProcessorFactory [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/update params={wt=javabin&version=2}{add=[2 (1578022191801303040)]} 0 11
[junit4] 2> 22305 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n3] webapp=/solr path=/get params={qt=/get&_stateVer_=tlog_replica_test_real_time_get:5&ids=2&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 22309 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&ids=2&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 22314 INFO (qtp111205268-36) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n3] webapp=/solr path=/get params={qt=/get&ids=2&wt=javabin&version=2} status=0 QTime=1
[junit4] 2> 22319 INFO (qtp1877440554-29) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_n1/|https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/|https://127.0.0.1:52497/solr/tlog_replica_test_real_time_get_shard1_replica_n3/
[junit4] 2> 22323 INFO (qtp1877440554-23) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1504919235067&ids=2&isShard=true&shard.url=https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_n1/|https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/|https://127.0.0.1:52497/solr/tlog_replica_test_real_time_get_shard1_replica_n3/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 22326 INFO (qtp1877440554-29) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/get params={qt=/get&ids=2&wt=javabin&version=2} status=0 QTime=7
[junit4] 2> 22329 INFO (qtp1877440554-30) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node2 x:tlog_replica_test_real_time_get_shard1_replica_n1] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n1] webapp=/solr path=/get params={qt=/get&ids=0&ids=1&ids=2&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 22334 INFO (qtp111205268-37) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n3] webapp=/solr path=/get params={qt=/get&ids=0&ids=1&ids=2&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 22337 INFO (qtp1877440554-26) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.h.c.RealTimeGetComponent LOOKUP_SLICE:shard1=https://127.0.0.1:52497/solr/tlog_replica_test_real_time_get_shard1_replica_n3/|https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/|https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_n1/
[junit4] 2> 22340 INFO (qtp111205268-35) [n:127.0.0.1:52497_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node5 x:tlog_replica_test_real_time_get_shard1_replica_n3] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_n3] webapp=/solr path=/get params={distrib=false&qt=/get&omitHeader=true&shards.purpose=1&NOW=1504919235085&ids=0,1,2&isShard=true&shard.url=https://127.0.0.1:52497/solr/tlog_replica_test_real_time_get_shard1_replica_n3/|https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_t4/|https://127.0.0.1:52496/solr/tlog_replica_test_real_time_get_shard1_replica_n1/&wt=javabin&version=2&shards.qt=/get} status=0 QTime=0
[junit4] 2> 22342 INFO (qtp1877440554-26) [n:127.0.0.1:52496_solr c:tlog_replica_test_real_time_get s:shard1 r:core_node6 x:tlog_replica_test_real_time_get_shard1_replica_t4] o.a.s.c.S.Request [tlog_replica_test_real_time_get_shard1_replica_t4] webapp=/solr path=/get params={qt=/get&ids=0&ids=1&ids=2&wt=javabin&version=2} status=0 QTime=4
[junit4] 2> 22344 INFO (TEST-TestTlogReplica.testRealTimeGet-seed#[9FE8A28912BCCBEA]) [ ] o.a.s.c.TestTlogReplica tearDown deleting collection
[junit4] 2> 22349 INFO (qtp111205268-38) [n:127.0.0.1:52497_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=tlog_replica_test_real_time_get&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 22363 INFO (OverseerThreadFactory-10-thread-2-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.OverseerCollectionMessageHandler Executing Collection Cmd : action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true
[junit4] 2> 22369 INFO (qtp1877440554-25) [n:127.0.0.1:52496_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_real_time_get.shard1.replica_t4, tag=null
[junit4] 2> 22371 INFO (qtp1877440554-25) [n:127.0.0.1:52496_solr ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@400c4d3: rootName = solr_52496, domain = solr.core.tlog_replica_test_real_time_get.shard1.replica_t4, service url = null, agent id = null] for registry solr.core.tlog_replica_test_real_time_get.shard1.replica_t4 / com.codahale.metrics.MetricRegistry@1d76f5f
[junit4] 2> 22371 INFO (qtp111205268-33) [n:127.0.0.1:52497_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_real_time_get.shard1.replica_n3, tag=null
[junit4] 2> 22371 INFO (qtp111205268-33) [n:127.0.0.1:52497_solr ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3d4c55c: rootName = solr_52497, domain = solr.core.tlog_replica_test_real_time_get.shard1.replica_n3, service url = null, agent id = null] for registry solr.core.tlog_replica_test_real_time_get.shard1.replica_n3 / com.codahale.metrics.MetricRegistry@12258613
[junit4] 2> 22408 INFO (qtp111205268-33) [n:127.0.0.1:52497_solr ] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_n3] CLOSING SolrCore org.apache.solr.core.SolrCore@1284ccec
[junit4] 2> 22408 INFO (qtp111205268-33) [n:127.0.0.1:52497_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_real_time_get.shard1.replica_n3, tag=310693100
[junit4] 2> 22408 INFO (qtp111205268-33) [n:127.0.0.1:52497_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.tlog_replica_test_real_time_get.shard1.leader, tag=310693100
[junit4] 2> 22414 INFO (qtp111205268-33) [n:127.0.0.1:52497_solr ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
[junit4] 2> 22414 INFO (qtp111205268-33) [n:127.0.0.1:52497_solr ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@556e05af commitCommandVersion:0
[junit4] 2> 22425 INFO (qtp1877440554-25) [n:127.0.0.1:52496_solr ] o.a.s.c.ZkController tlog_replica_test_real_time_get_shard1_replica_t4 stopping background replication from leader
[junit4] 2> 22425 INFO (qtp1877440554-27) [n:127.0.0.1:52496_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_real_time_get.shard1.replica_n1, tag=null
[junit4] 2> 22427 INFO (qtp1877440554-25) [n:127.0.0.1:52496_solr ] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_t4] CLOSING SolrCore org.apache.solr.core.SolrCore@7f55ac60
[junit4] 2> 22427 INFO (qtp1877440554-27) [n:127.0.0.1:52496_solr ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@20ee5f0d: rootName = solr_52496, domain = solr.core.tlog_replica_test_real_time_get.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.tlog_replica_test_real_time_get.shard1.replica_n1 / com.codahale.metrics.MetricRegistry@314be08a
[junit4] 2> 22465 INFO (qtp1877440554-27) [n:127.0.0.1:52496_solr ] o.a.s.c.SolrCore [tlog_replica_test_real_time_get_shard1_replica_n1] CLOSING SolrCore org.apache.solr.core.SolrCore@37a9ecdf
[junit4] 2> 22466 INFO (qtp1877440554-27) [n:127.0.0.1:52496_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_real_time_get.shard1.replica_n1, tag=933883103
[junit4] 2> 22466 INFO (qtp1877440554-27) [n:127.0.0.1:52496_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.tlog_replica_test_real_time_get.shard1.leader, tag=933883103
[junit4] 2> 22466 INFO (qtp1877440554-27) [n:127.0.0.1:52496_solr ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
[junit4] 2> 22466 INFO (qtp1877440554-27) [n:127.0.0.1:52496_solr ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@2faef1ad commitCommandVersion:0
[junit4] 2> 22478 INFO (qtp1877440554-25) [n:127.0.0.1:52496_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tlog_replica_test_real_time_get.shard1.replica_t4, tag=2136321120
[junit4] 2> 22478 INFO (qtp1877440554-25) [n:127.0.0.1:52496_solr ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.tlog_replica_test_real_time_get.shard1.leader, tag=2136321120
[junit4] 2> 22479 INFO (qtp1877440554-25) [n:127.0.0.1:52496_solr ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter close.
[junit4] 2> 22479 INFO (qtp1877440554-25) [n:127.0.0.1:52496_solr ] o.a.s.u.SolrIndexWriter Calling setCommitData with IW:org.apache.solr.update.SolrIndexWriter@500262bf commitCommandVersion:0
[junit4] 2> 23312 INFO (qtp1877440554-27) [n:127.0.0.1:52496_solr ] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
[junit4] 2> 23314 INFO (qtp111205268-33) [n:127.0.0.1:52497_solr ] o.a.s.c.ShardLeaderElectionContextBase No version found for ephemeral leader parent node, won't remove previous leader registration.
[junit4] 2> 23316 INFO (qtp111205268-33) [n:127.0.0.1:52497_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=tlog_replica_test_real_time_get_shard1_replica_n3&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=949
[junit4] 2> 23318 WARN (zkCallback-13-thread-1-processing-n:127.0.0.1:52496_solr) [n:127.0.0.1:52496_solr ] o.a.s.c.LeaderElector Our node is no longer in line to be leader
[junit4] 2> 23318 WARN (zkCallback-12-thread-2-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.LeaderElector Our node is no longer in line to be leader
[junit4] 2> 23320 INFO (qtp1877440554-27) [n:127.0.0.1:52496_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=tlog_replica_test_real_time_get_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=952
[junit4] 2> 23323 INFO (qtp1877440554-25) [n:127.0.0.1:52496_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=tlog_replica_test_real_time_get_shard1_replica_t4&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=955
[junit4] 2> 24062 INFO (qtp111205268-38) [n:127.0.0.1:52497_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=tlog_replica_test_real_time_get&action=DELETE&wt=javabin&version=2} status=0 QTime=1713
[junit4] 2> 24166 INFO (TEST-TestTlogReplica.testRealTimeGet-seed#[9FE8A28912BCCBEA]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testRealTimeGet
[junit4] 2> 24219 INFO (TEST-TestTlogReplica.testKillTlogReplica-seed#[9FE8A28912BCCBEA]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testKillTlogReplica
[junit4] 2> 24221 INFO (qtp111205268-34) [n:127.0.0.1:52497_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&replicationFactor=0&collection.configName=conf&maxShardsPerNode=100&name=tlog_replica_test_kill_tlog_replica&nrtReplicas=0&action=CREATE&numShards=1&tlogReplicas=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 24240 INFO (OverseerThreadFactory-10-thread-3-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.CreateCollectionCmd Create collection tlog_replica_test_kill_tlog_replica
[junit4] 2> 24245 INFO (OverseerCollectionConfigSetProcessor-98626385651695621-127.0.0.1:52497_solr-n_0000000000) [n:127.0.0.1:52497_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 24502 INFO (OverseerStateUpdate-98626385651695621-127.0.0.1:52497_solr-n_0000000000) [n:127.0.0.1:52497_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_kill_tlog_replica",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"tlog_replica_test_kill_tlog_replica_shard1_replica_t1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:52496/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 24514 INFO (OverseerStateUpdate-98626385651695621-127.0.0.1:52497_solr-n_0000000000) [n:127.0.0.1:52497_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"tlog_replica_test_kill_tlog_replica",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"tlog_replica_test_kill_tlog_replica_shard1_replica_t2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"https://127.0.0.1:52497/solr",
[junit4] 2> "type":"TLOG"}
[junit4] 2> 24726 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=tlog_replica_test_kill_tlog_replica_shard1_replica_t1&action=CREATE&numShards=1&collection=tlog_replica_test_kill_tlog_replica&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 24738 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=tlog_replica_test_kill_tlog_replica_shard1_replica_t2&action=CREATE&numShards=1&collection=tlog_replica_test_kill_tlog_replica&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 24890 INFO (zkCallback-13-thread-1-processing-n:127.0.0.1:52496_solr) [n:127.0.0.1:52496_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_tlog_replica/state.json] for collection [tlog_replica_test_kill_tlog_replica] has occurred - updating... (live nodes size: [2])
[junit4] 2> 24890 INFO (zkCallback-12-thread-1-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_tlog_replica/state.json] for collection [tlog_replica_test_kill_tlog_replica] has occurred - updating... (live nodes size: [2])
[junit4] 2> 24890 INFO (zkCallback-12-thread-2-processing-n:127.0.0.1:52497_solr) [n:127.0.0.1:52497_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_tlog_replica/state.json] for collection [tlog_replica_test_kill_tlog_replica] has occurred - updating... (live nodes size: [2])
[junit4] 2> 24891 INFO (zkCallback-13-thread-2-processing-n:127.0.0.1:52496_solr) [n:127.0.0.1:52496_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tlog_replica_test_kill_tlog_replica/state.json] for collection [tlog_replica_test_kill_tlog_replica] has occurred - updating... (live nodes size: [2])
[junit4] 2> 25795 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 25796 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.1.0
[junit4] 2> 25829 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.s.IndexSchema [tlog_replica_test_kill_tlog_replica_shard1_replica_t1] Schema name=minimal
[junit4] 2> 25844 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 25844 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_kill_tlog_replica_shard1_replica_t1' using configuration from collection tlog_replica_test_kill_tlog_replica, trusted=true
[junit4] 2> 25845 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52496.solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t1' (registry 'solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3139eb6d
[junit4] 2> 25846 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 25847 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.c.SolrCore [[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_9FE8A28912BCCBEA-001/tempDir-001/node1/tlog_replica_test_kill_tlog_replica_shard1_replica_t1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_9FE8A28912BCCBEA-001/tempDir-001/node1/./tlog_replica_test_kill_tlog_replica_shard1_replica_t1/data/]
[junit4] 2> 25850 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.s.IndexSchema [tlog_replica_test_kill_tlog_replica_shard1_replica_t2] Schema name=minimal
[junit4] 2> 25863 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 25863 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.CoreContainer Creating SolrCore 'tlog_replica_test_kill_tlog_replica_shard1_replica_t2' using configuration from collection tlog_replica_test_kill_tlog_replica, trusted=true
[junit4] 2> 25864 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_52497.solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t2' (registry 'solr.core.tlog_replica_test_kill_tlog_replica.shard1.replica_t2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@3139eb6d
[junit4] 2> 25864 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 25865 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.SolrCore [[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_9FE8A28912BCCBEA-001/tempDir-001/node2/tlog_replica_test_kill_tlog_replica_shard1_replica_t2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.TestTlogReplica_9FE8A28912BCCBEA-001/tempDir-001/node2/./tlog_replica_test_kill_tlog_replica_shard1_replica_t2/data/]
[junit4] 2> 25987 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 25987 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 26000 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 26001 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 26004 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@67a8ac07[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] main]
[junit4] 2> 26006 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 26006 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 26006 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 26006 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 26008 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 26008 INFO (qtp111205268-31) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1578022195697811456
[junit4] 2> 26011 INFO (searcherExecutor-31-thread-1-processing-n:127.0.0.1:52497_solr x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2 s:shard1 c:tlog_replica_test_kill_tlog_replica r:core_node4) [n:127.0.0.1:52497_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node4 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t2] o.a.s.c.SolrCore [tlog_replica_test_kill_tlog_replica_shard1_replica_t2] Registered new searcher Searcher@67a8ac07[tlog_replica_test_kill_tlog_replica_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 26012 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 26013 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 26016 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5ab734aa[tlog_replica_test_kill_tlog_replica_shard1_replica_t1] main]
[junit4] 2> 26022 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 26023 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 26024 INFO (qtp1877440554-28) [n:127.0.0.1:52496_solr c:tlog_replica_test_kill_tlog_replica s:shard1 r:core_node3 x:tlog_replica_test_kill_tlog_replica_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000
[junit4] 2> 26024 INFO (qtp1877440554-28) [n:127.0.0.1:52496
[...truncated too long message...]
c:implicitcoll s:c r:core_node24 x:implicitcoll_c_replica_n22] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.implicitcoll.c.leader, tag=807224253
[junit4] 2> 2164506 INFO (jetty-closer-3507-thread-4) [ ] o.a.s.c.Overseer Overseer (id=98626525528981514-127.0.0.1:60071_solr-n_0000000000) closing
[junit4] 2> 2164506 INFO (OverseerStateUpdate-98626525528981514-127.0.0.1:60071_solr-n_0000000000) [n:127.0.0.1:60071_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:60071_solr
[junit4] 2> 2164509 INFO (zkCallback-3526-thread-1-processing-n:127.0.0.1:60071_solr) [n:127.0.0.1:60071_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 2164515 WARN (zkCallback-3526-thread-2-processing-n:127.0.0.1:60071_solr) [n:127.0.0.1:60071_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 2164515 INFO (jetty-closer-3507-thread-4) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2d45df10{/solr,null,UNAVAILABLE}
[junit4] 2> 2166009 WARN (zkCallback-3527-thread-4-processing-n:127.0.0.1:60073_solr) [n:127.0.0.1:60073_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 2166010 INFO (jetty-closer-3507-thread-3) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@f52fce1{/solr,null,UNAVAILABLE}
[junit4] 2> 2166013 ERROR (SUITE-CustomCollectionTest-seed#[9FE8A28912BCCBEA]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2166014 INFO (SUITE-CustomCollectionTest-seed#[9FE8A28912BCCBEA]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60069 60069
[junit4] 2> 2166038 INFO (Thread-4086) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60069 60069
[junit4] 2> 2166248 WARN (Thread-4086) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 15 /solr/configs/conf
[junit4] 2> 9 /solr/clusterprops.json
[junit4] 2> 9 /solr/aliases.json
[junit4] 2> 4 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 44 /solr/collections/implicitcoll/state.json
[junit4] 2> 44 /solr/collections/withShardField/state.json
[junit4] 2> 35 /solr/collections/routeFieldColl/state.json
[junit4] 2> 25 /solr/collections/testCreateShardRepFactor/state.json
[junit4] 2> 9 /solr/clusterstate.json
[junit4] 2> 2 /solr/collections/implicitcoll/leader_elect/b/election/98626525528981514-core_node17-n_0000000000
[junit4] 2> 2 /solr/overseer_elect/election/98626525528981514-127.0.0.1:60071_solr-n_0000000000
[junit4] 2> 2 /solr/collections/withShardField/leader_elect/d/election/98626525528981513-core_node31-n_0000000001
[junit4] 2> 2 /solr/collections/implicitcoll/leader_elect/c/election/98626525528981515-core_node21-n_0000000000
[junit4] 2> 2 /solr/collections/implicitcoll/leader_elect/a/election/98626525528981514-core_node9-n_0000000000
[junit4] 2> 2 /solr/collections/withShardField/leader_elect/a/election/98626525528981514-core_node5-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 28 /solr/collections
[junit4] 2> 9 /solr/live_nodes
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.CustomCollectionTest_9FE8A28912BCCBEA-001
[junit4] 2> Sep 09, 2017 1:42:58 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=false): {}, locale=en-NZ, timezone=Europe/Lisbon
[junit4] 2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=3,threads=1,free=190264664,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [DistributedFacetPivotLargeTest, TestSearcherReuse, TestSolrCoreSnapshots, TestSizeLimitedDistributedMap, ReturnFieldsTest, ClusterStateTest, SolrGangliaReporterTest, TestScoreJoinQPScore, HLLSerializationTest, TestSolrDeletionPolicy2, TestFoldingMultitermQuery, DeleteShardTest, TestReloadAndDeleteDocs, DefaultValueUpdateProcessorTest, HdfsDirectoryFactoryTest, TestSmileRequest, TestHashPartitioner, TestRandomDVFaceting, TestGroupingSearch, TestStressRecovery, OverseerRolesTest, TestCoreContainer, TestDeleteCollectionOnDownNodes, CoreAdminHandlerTest, TestSolrCloudWithKerberosAlt, BinaryUpdateRequestHandlerTest, HdfsRecoverLeaseTest, OverseerCollectionConfigSetProcessorTest, HdfsThreadLeakTest, ActionThrottleTest, UUIDFieldTest, TestReversedWildcardFilterFactory, TestTestInjection, ConfigureRecoveryStrategyTest, TestStressReorder, TestLegacyNumericRangeQueryBuilder, TestExpandComponent, HighlighterConfigTest, CursorPagingTest, TlogReplayBufferedWhileIndexingTest, MoveReplicaHDFSUlogDirTest, TestFieldCacheSort, TestNumericTokenStream, HdfsRestartWhileUpdatingTest, TestPointFields, TestRequestStatusCollectionAPI, TestNonDefinedSimilarityFactory, SolrIndexSplitterTest, DocValuesTest, TestStressVersions, ChaosMonkeySafeLeaderWithPullReplicasTest, NumericFieldsTest, SolrSlf4jReporterTest, QueryParsingTest, TestReplicationHandler, TestNamedUpdateProcessors, MultiThreadedOCPTest, RestartWhileUpdatingTest, TestExtendedDismaxParser, RAMDirectoryFactoryTest, TestSchemalessBufferedUpdates, DirectoryFactoryTest, TestMaxTokenLenTokenizer, HdfsBasicDistributedZk2Test, UUIDUpdateProcessorFallbackTest, SearchHandlerTest, TestPHPSerializedResponseWriter, WordBreakSolrSpellCheckerTest, TestOmitPositions, PathHierarchyTokenizerFactoryTest, HighlighterTest, TestSolrXml, CSVRequestHandlerTest, TestSort, TestPivotHelperCode, TestReplicaProperties, TestSchemaNameResource, CdcrVersionReplicationTest, TestNoOpRegenerator, TestManagedStopFilterFactory, UninvertDocValuesMergePolicyTest, TestRandomRequestDistribution, RecoveryZkTest, JvmMetricsTest, PeerSyncWithIndexFingerprintCachingTest, TestFieldCacheVsDocValues, BasicDistributedZkTest, TestFunctionQuery, SolrTestCaseJ4Test, TestLFUCache, TestCustomStream, SolrShardReporterTest, TestNumericRangeQuery32, TestAddFieldRealTimeGet, DirectUpdateHandlerOptimizeTest, TestLazyCores, TestFieldCollectionResource, CollectionsAPISolrJTest, CollectionReloadTest, TestTolerantSearch, TestSolrCoreParser, TestAtomicUpdateErrorCases, DeleteInactiveReplicaTest, SSLMigrationTest, TestExactStatsCache, BufferStoreTest, TestStressCloudBlindAtomicUpdates, AddBlockUpdateTest, TestRawResponseWriter, TestRecoveryHdfs, HardAutoCommitTest, TestBulkSchemaConcurrent, TestCollectionAPI, SliceStateTest, HdfsChaosMonkeyNothingIsSafeTest, TestFieldCache, DistributedExpandComponentTest, TestCollectionAPIs, SmileWriterTest, ShardRoutingTest, UniqFieldsUpdateProcessorFactoryTest, TestImplicitCoreProperties, TestDFRSimilarityFactory, IndexBasedSpellCheckerTest, TestStressLiveNodes, ResponseHeaderTest, ClassificationUpdateProcessorFactoryTest, UnloadDistributedZkTest, LeaderFailureAfterFreshStartTest, StressHdfsTest, SuggestComponentTest, TestNumericTerms64, TestConfigSetsAPI, CircularListTest, TestCloudDeleteByQuery, SyncSliceTest, RecoveryAfterSoftCommitTest, TestSurroundQueryParser, TestPKIAuthenticationPlugin, BigEndianAscendingWordSerializerTest, SimplePostToolTest, TestSQLHandlerNonCloud, DisMaxRequestHandlerTest, SolrMetricReporterTest, AtomicUpdateProcessorFactoryTest, TestBlendedInfixSuggestions, SoftAutoCommitTest, TestQuerySenderNoQuery, TestDistributedSearch, DistributedVersionInfoTest, TestJavabinTupleStreamParser, TestImpersonationWithHadoopAuth, TestWriterPerf, DirectSolrConnectionTest, VersionInfoTest, SolrXmlInZkTest, PKIAuthenticationIntegrationTest, TestIndexingPerformance, TestBackupRepositoryFactory, ReplaceNodeTest, BJQParserTest, TestMiniSolrCloudClusterSSL, OverseerTaskQueueTest, QueryElevationComponentTest, LargeFieldTest, BlockCacheTest, DistributedQueryComponentOptimizationTest, SpatialHeatmapFacetsTest, CoreMergeIndexesAdminHandlerTest, TestCloudPseudoReturnFields, TestCustomSort, CoreAdminCreateDiscoverTest, TestCloudSchemaless, RankQueryTest, TestClassicSimilarityFactory, SpatialFilterTest, TestTolerantUpdateProcessorCloud, LeaderElectionIntegrationTest, LeaderElectionTest, TestDistributedGrouping, TestFaceting, DistributedSpellCheckComponentTest, ZkControllerTest, TestSolr4Spatial, SolrCmdDistributorTest, BasicFunctionalityTest, DirectUpdateHandlerTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SignatureUpdateProcessorFactoryTest, SuggesterFSTTest, SuggesterTest, SuggesterWFSTTest, NoCacheHeaderTest, TestPseudoReturnFields, TestUpdate, FieldMutatingUpdateProcessorTest, TestRemoteStreaming, DistanceFunctionTest, SolrInfoBeanTest, CacheHeaderTest, LukeRequestHandlerTest, PrimitiveFieldTypeTest, TermsComponentTest, RegexBoostProcessorTest, TestJmxIntegration, TestComponentsName, SearchHandlerTest, TestSearchPerf, OutputWriterTest, ExternalFileFieldSortTest, NotRequiredUniqueKeyTest, TestPhraseSuggestions, TestCodecSupport, TestPerFieldSimilarity, TestLMDirichletSimilarityFactory, TestIBSimilarityFactory, TimeZoneUtilsTest, ScriptEngineTest, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest, AssignTest, CdcrReplicationDistributedZkTest, CdcrReplicationHandlerTest, CleanupOldIndexTest, CollectionsAPIAsyncDistributedZkTest, CustomCollectionTest]
[junit4] Completed [509/731 (2!)] on J0 in 30.30s, 4 tests, 1 error <<< FAILURES!
[...truncated 47055 lines...]
[JENKINS] Lucene-Solr-7.x-MacOSX (64bit/jdk1.8.0) - Build # 163 -
Still Failing!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-MacOSX/163/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC
81 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.analytics.NoFacetCloudTest
Error Message:
Error starting up MiniSolrCloudCluster
Stack Trace:
java.lang.Exception: Error starting up MiniSolrCloudCluster
at __randomizedtesting.SeedInfo.seed([2A014917FCE8CFBA]:0)
at org.apache.solr.cloud.MiniSolrCloudCluster.checkForExceptions(MiniSolrCloudCluster.java:507)
at org.apache.solr.cloud.MiniSolrCloudCluster.<init>(MiniSolrCloudCluster.java:251)
at org.apache.solr.cloud.SolrCloudTestCase$Builder.configure(SolrCloudTestCase.java:190)
at org.apache.solr.analytics.AbstractAnalyticsStatsCloudTest.setupCluster(AbstractAnalyticsStatsCloudTest.java:73)
at org.apache.solr.analytics.NoFacetCloudTest.populate(NoFacetCloudTest.java:62)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:874)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Suppressed: java.lang.RuntimeException: java.lang.NullPointerException
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1600)
at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1659)
at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1316)
at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1145)
at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:384)
at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:247)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: java.lang.NullPointerException
at org.apache.solr.util.configuration.providers.EnvSSLCredentialProvider.getCredential(EnvSSLCredentialProvider.java:57)
at org.apache.solr.util.configuration.providers.AbstractSSLCredentialProvider.getCredential(AbstractSSLCredentialProvider.java:53)
at org.apache.solr.util.configuration.SSLConfigurations.getPassword(SSLConfigurations.java:123)
at org.apache.solr.util.configuration.SSLConfigurations.getClientKeyStorePassword(SSLConfigurations.java:109)
at org.apache.solr.util.configuration.SSLConfigurations.init(SSLConfigurations.java:62)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:148)
at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1596)
... 16 more
Suppressed: java.lang.RuntimeException: java.lang.NullPointerException
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1600)
at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1659)
at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1316)
at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1145)
at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:384)
at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:247)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: java.lang.NullPointerException
at org.apache.solr.util.configuration.providers.EnvSSLCredentialProvider.getCredential(EnvSSLCredentialProvider.java:57)
at org.apache.solr.util.configuration.providers.AbstractSSLCredentialProvider.getCredential(AbstractSSLCredentialProvider.java:53)
at org.apache.solr.util.configuration.SSLConfigurations.getPassword(SSLConfigurations.java:123)
at org.apache.solr.util.configuration.SSLConfigurations.getClientKeyStorePassword(SSLConfigurations.java:109)
at org.apache.solr.util.configuration.SSLConfigurations.init(SSLConfigurations.java:62)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:148)
at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1596)
... 16 more
Suppressed: java.lang.RuntimeException: java.lang.NullPointerException
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1600)
at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1659)
at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1316)
at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1145)
at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:384)
at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:247)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: java.lang.NullPointerException
at org.apache.solr.util.configuration.providers.EnvSSLCredentialProvider.getCredential(EnvSSLCredentialProvider.java:57)
at org.apache.solr.util.configuration.providers.AbstractSSLCredentialProvider.getCredential(AbstractSSLCredentialProvider.java:53)
at org.apache.solr.util.configuration.SSLConfigurations.getPassword(SSLConfigurations.java:123)
at org.apache.solr.util.configuration.SSLConfigurations.getClientKeyStorePassword(SSLConfigurations.java:109)
at org.apache.solr.util.configuration.SSLConfigurations.init(SSLConfigurations.java:62)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:148)
at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1596)
... 16 more
Suppressed: java.lang.RuntimeException: java.lang.NullPointerException
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1600)
at org.eclipse.jetty.servlet.ServletHandler.setFilterMappings(ServletHandler.java:1659)
at org.eclipse.jetty.servlet.ServletHandler.addFilterMapping(ServletHandler.java:1316)
at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:1145)
at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:448)
at org.apache.solr.client.solrj.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:306)
at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:179)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:69)
at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:394)
at org.apache.solr.client.solrj.embedded.JettySolrRunner.start(JettySolrRunner.java:367)
at org.apache.solr.cloud.MiniSolrCloudCluster.startJettySolrRunner(MiniSolrCloudCluster.java:384)
at org.apache.solr.cloud.MiniSolrCloudCluster.lambda$new$0(MiniSolrCloudCluster.java:247)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: java.lang.NullPointerException
at org.apache.solr.util.configuration.providers.EnvSSLCredentialProvider.getCredential(EnvSSLCredentialProvider.java:57)
at org.apache.solr.util.configuration.providers.AbstractSSLCredentialProvider.getCredential(AbstractSSLCredentialProvider.java:53)
at org.apache.solr.util.configuration.SSLConfigurations.getPassword(SSLConfigurations.java:123)
at org.apache.solr.util.configuration.SSLConfigurations.getClientKeyStorePassword(SSLConfigurations.java:109)
at org.apache.solr.util.configuration.SSLConfigurations.init(SSLConfigurations.java:62)
at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:148)
at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
at org.eclipse.jetty.servlet.ServletHandler.updateMappings(ServletHandler.java:1596)
... 16 more
FAILED: junit.framework.TestSuite.org.apache.solr.analytics.NoFacetCloudTest
Error Message:
36 threads leaked from SUITE scope at org.apache.solr.analytics.NoFacetCloudTest: 1) Thread[id=66, name=qtp123030152-66, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=71, name=qtp1949380581-71-acceptor-0@34657965-ServerConnector@12772944{HTTP/1.1,[http/1.1]}{127.0.0.1:52238}, state=RUNNABLE, group=TGRP-NoFacetCloudTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=78, name=qtp1949380581-78, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=93, name=org.eclipse.jetty.server.session.HashSessionManager@3a0e8b47Timer, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=95, name=org.eclipse.jetty.server.session.HashSessionManager@5c0484d0Timer, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=73, name=qtp2072002467-73, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=85, name=qtp1949380581-85, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 8) Thread[id=79, name=qtp1949380581-79, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 9) Thread[id=75, name=qtp2072002467-75, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 10) Thread[id=81, name=qtp1949380581-81, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 11) Thread[id=76, name=qtp2072002467-76, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 12) Thread[id=70, name=qtp728895240-70-acceptor-0@185934e3-ServerConnector@14a0d172{HTTP/1.1,[http/1.1]}{127.0.0.1:52239}, state=RUNNABLE, group=TGRP-NoFacetCloudTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 13) Thread[id=90, name=qtp728895240-90, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 14) Thread[id=62, name=qtp1949380581-62, state=RUNNABLE, group=TGRP-NoFacetCloudTest] at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 15) Thread[id=88, name=qtp728895240-88, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 16) Thread[id=67, name=qtp123030152-67, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 17) Thread[id=69, name=qtp2072002467-69-acceptor-0@739a8b5f-ServerConnector@3b8c1fa5{HTTP/1.1,[http/1.1]}{127.0.0.1:52237}, state=RUNNABLE, group=TGRP-NoFacetCloudTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 18) Thread[id=74, name=qtp2072002467-74, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 19) Thread[id=91, name=qtp728895240-91, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 20) Thread[id=92, name=org.eclipse.jetty.server.session.HashSessionManager@2cbcb95cTimer, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 21) Thread[id=83, name=qtp1949380581-83, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 22) Thread[id=61, name=qtp123030152-61, state=RUNNABLE, group=TGRP-NoFacetCloudTest] at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 23) Thread[id=77, name=qtp2072002467-77, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 24) Thread[id=72, name=qtp123030152-72, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 25) Thread[id=87, name=qtp728895240-87, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 26) Thread[id=65, name=qtp728895240-65, state=RUNNABLE, group=TGRP-NoFacetCloudTest] at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 27) Thread[id=68, name=qtp123030152-68, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 28) Thread[id=82, name=qtp2072002467-82, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 29) Thread[id=94, name=org.eclipse.jetty.server.session.HashSessionManager@1303d777Timer, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 30) Thread[id=89, name=qtp728895240-89, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 31) Thread[id=80, name=qtp123030152-80, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 32) Thread[id=86, name=qtp728895240-86, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 33) Thread[id=60, name=qtp2072002467-60, state=RUNNABLE, group=TGRP-NoFacetCloudTest] at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 34) Thread[id=63, name=qtp123030152-63-acceptor-0@52c3c06e-ServerConnector@45154204{HTTP/1.1,[http/1.1]}{127.0.0.1:52236}, state=RUNNABLE, group=TGRP-NoFacetCloudTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 35) Thread[id=64, name=qtp123030152-64, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748) 36) Thread[id=84, name=qtp1949380581-84, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 36 threads leaked from SUITE scope at org.apache.solr.analytics.NoFacetCloudTest:
1) Thread[id=66, name=qtp123030152-66, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
2) Thread[id=71, name=qtp1949380581-71-acceptor-0@34657965-ServerConnector@12772944{HTTP/1.1,[http/1.1]}{127.0.0.1:52238}, state=RUNNABLE, group=TGRP-NoFacetCloudTest]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
3) Thread[id=78, name=qtp1949380581-78, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
4) Thread[id=93, name=org.eclipse.jetty.server.session.HashSessionManager@3a0e8b47Timer, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
5) Thread[id=95, name=org.eclipse.jetty.server.session.HashSessionManager@5c0484d0Timer, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
6) Thread[id=73, name=qtp2072002467-73, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
7) Thread[id=85, name=qtp1949380581-85, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
8) Thread[id=79, name=qtp1949380581-79, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
9) Thread[id=75, name=qtp2072002467-75, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
10) Thread[id=81, name=qtp1949380581-81, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
11) Thread[id=76, name=qtp2072002467-76, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
12) Thread[id=70, name=qtp728895240-70-acceptor-0@185934e3-ServerConnector@14a0d172{HTTP/1.1,[http/1.1]}{127.0.0.1:52239}, state=RUNNABLE, group=TGRP-NoFacetCloudTest]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
13) Thread[id=90, name=qtp728895240-90, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
14) Thread[id=62, name=qtp1949380581-62, state=RUNNABLE, group=TGRP-NoFacetCloudTest]
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
15) Thread[id=88, name=qtp728895240-88, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
16) Thread[id=67, name=qtp123030152-67, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
17) Thread[id=69, name=qtp2072002467-69-acceptor-0@739a8b5f-ServerConnector@3b8c1fa5{HTTP/1.1,[http/1.1]}{127.0.0.1:52237}, state=RUNNABLE, group=TGRP-NoFacetCloudTest]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
18) Thread[id=74, name=qtp2072002467-74, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
19) Thread[id=91, name=qtp728895240-91, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
20) Thread[id=92, name=org.eclipse.jetty.server.session.HashSessionManager@2cbcb95cTimer, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
21) Thread[id=83, name=qtp1949380581-83, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
22) Thread[id=61, name=qtp123030152-61, state=RUNNABLE, group=TGRP-NoFacetCloudTest]
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
23) Thread[id=77, name=qtp2072002467-77, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
24) Thread[id=72, name=qtp123030152-72, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
25) Thread[id=87, name=qtp728895240-87, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
26) Thread[id=65, name=qtp728895240-65, state=RUNNABLE, group=TGRP-NoFacetCloudTest]
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
27) Thread[id=68, name=qtp123030152-68, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
28) Thread[id=82, name=qtp2072002467-82, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
29) Thread[id=94, name=org.eclipse.jetty.server.session.HashSessionManager@1303d777Timer, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
30) Thread[id=89, name=qtp728895240-89, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
31) Thread[id=80, name=qtp123030152-80, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
32) Thread[id=86, name=qtp728895240-86, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
33) Thread[id=60, name=qtp2072002467-60, state=RUNNABLE, group=TGRP-NoFacetCloudTest]
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
34) Thread[id=63, name=qtp123030152-63-acceptor-0@52c3c06e-ServerConnector@45154204{HTTP/1.1,[http/1.1]}{127.0.0.1:52236}, state=RUNNABLE, group=TGRP-NoFacetCloudTest]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:371)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:601)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
35) Thread[id=64, name=qtp123030152-64, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
36) Thread[id=84, name=qtp1949380581-84, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([2A014917FCE8CFBA]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.analytics.NoFacetCloudTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=65, name=qtp728895240-65, state=RUNNABLE, group=TGRP-NoFacetCloudTest] at sun.nio.ch.KQueueArrayWrapper.interrupt(Native Method) at sun.nio.ch.KQueueArrayWrapper.interrupt(KQueueArrayWrapper.java:203) at sun.nio.ch.KQueueSelectorImpl.wakeup(KQueueSelectorImpl.java:257) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:116) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=93, name=org.eclipse.jetty.server.session.HashSessionManager@3a0e8b47Timer, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=95, name=org.eclipse.jetty.server.session.HashSessionManager@5c0484d0Timer, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=94, name=org.eclipse.jetty.server.session.HashSessionManager@1303d777Timer, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=92, name=org.eclipse.jetty.server.session.HashSessionManager@2cbcb95cTimer, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=61, name=qtp123030152-61, state=RUNNABLE, group=TGRP-NoFacetCloudTest] at sun.nio.ch.KQueueArrayWrapper.interrupt(Native Method) at sun.nio.ch.KQueueArrayWrapper.interrupt(KQueueArrayWrapper.java:203) at sun.nio.ch.KQueueSelectorImpl.wakeup(KQueueSelectorImpl.java:257) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:116) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=62, name=qtp1949380581-62, state=RUNNABLE, group=TGRP-NoFacetCloudTest] at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.KQueueSelectorImpl.updateSelectedKeys(KQueueSelectorImpl.java:181) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) 8) Thread[id=60, name=qtp2072002467-60, state=RUNNABLE, group=TGRP-NoFacetCloudTest] at sun.nio.ch.IOUtil.drain(Native Method) at sun.nio.ch.KQueueSelectorImpl.updateSelectedKeys(KQueueSelectorImpl.java:181) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:122) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100) at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=65, name=qtp728895240-65, state=RUNNABLE, group=TGRP-NoFacetCloudTest]
at sun.nio.ch.KQueueArrayWrapper.interrupt(Native Method)
at sun.nio.ch.KQueueArrayWrapper.interrupt(KQueueArrayWrapper.java:203)
at sun.nio.ch.KQueueSelectorImpl.wakeup(KQueueSelectorImpl.java:257)
at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:116)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
2) Thread[id=93, name=org.eclipse.jetty.server.session.HashSessionManager@3a0e8b47Timer, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
3) Thread[id=95, name=org.eclipse.jetty.server.session.HashSessionManager@5c0484d0Timer, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
4) Thread[id=94, name=org.eclipse.jetty.server.session.HashSessionManager@1303d777Timer, state=TIMED_WAITING, group=TGRP-NoFacetCloudTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.Scheduled
[...truncated too long message...]
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
[junit4] > at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 6) Thread[id=180, name=qtp1060073388-180, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
[junit4] > at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 7) Thread[id=174, name=qtp1060073388-174, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
[junit4] > at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
[junit4] > at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
[junit4] > at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
[junit4] > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
[junit4] > at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
[junit4] > at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
[junit4] > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
[junit4] > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
[junit4] > at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 8) Thread[id=179, name=qtp1060073388-179, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
[junit4] > at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 9) Thread[id=181, name=qtp1060073388-181, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
[junit4] > at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:563)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([2B259AD14C8BA4C4]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=182, name=org.eclipse.jetty.server.session.HashSessionManager@743300aaTimer, state=TIMED_WAITING, group=TGRP-TestLTROnSolrCloud]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
[junit4] > at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 2) Thread[id=174, name=qtp1060073388-174, state=RUNNABLE, group=TGRP-TestLTROnSolrCloud]
[junit4] > at sun.nio.ch.KQueueArrayWrapper.interrupt(Native Method)
[junit4] > at sun.nio.ch.KQueueArrayWrapper.interrupt(KQueueArrayWrapper.java:203)
[junit4] > at sun.nio.ch.KQueueSelectorImpl.wakeup(KQueueSelectorImpl.java:257)
[junit4] > at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
[junit4] > at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
[junit4] > at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:116)
[junit4] > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
[junit4] > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
[junit4] > at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:243)
[junit4] > at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:191)
[junit4] > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:249)
[junit4] > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
[junit4] > at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.execute(ExecuteProduceConsume.java:100)
[junit4] > at org.eclipse.jetty.io.ManagedSelector.run(ManagedSelector.java:147)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([2B259AD14C8BA4C4]:0)
[junit4] Completed [33/33 (26!)] on J1 in 26.17s, 1 test, 3 errors <<< FAILURES!
[...truncated 38836 lines...]
-documentation-lint:
[jtidy] Checking for broken html (such as invalid tags)...
[delete] Deleting directory /Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/lucene/build/jtidy_tmp
[echo] Checking for broken links...
[exec]
[exec] Crawl/parse...
[exec]
[exec] Verify...
[echo] Checking for malformed docs...
[exec]
[exec] /Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build/docs/solr-core/overview-summary.html
[exec] missing description: org.apache.solr.util.configuration.providers
[exec]
[exec] Missing javadocs were found!
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/build.xml:810: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/build.xml:101: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build.xml:669: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/solr/build.xml:685: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-7.x-MacOSX/lucene/common-build.xml:2532: exec returned: 1
Total time: 38 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any