You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2018/01/25 02:31:13 UTC
[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4406
- Still Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4406/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
3 tests failed.
FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([F686562254A27B7F:4C8A61AD0B4AAD30]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNull(Assert.java:551)
at org.junit.Assert.assertNull(Assert.java:562)
at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger(TriggerIntegrationTest.java:1575)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeAddedTriggerRestoreState
Error Message:
The trigger did not fire at all
Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
at __randomizedtesting.SeedInfo.seed([F686562254A27B7F:7EBBDF5D6E629AD2]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeAddedTriggerRestoreState(TriggerIntegrationTest.java:426)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventFromRestoredState
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([F686562254A27B7F:F6B0E2E27DAFDF15]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventFromRestoredState(TriggerIntegrationTest.java:760)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 11992 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_F686562254A27B7F-001/init-core-data-001
[junit4] 2> 603661 WARN (SUITE-TriggerIntegrationTest-seed#[F686562254A27B7F]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
[junit4] 2> 603661 INFO (SUITE-TriggerIntegrationTest-seed#[F686562254A27B7F]-worker) [ ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 603664 INFO (SUITE-TriggerIntegrationTest-seed#[F686562254A27B7F]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
[junit4] 2> 603664 INFO (SUITE-TriggerIntegrationTest-seed#[F686562254A27B7F]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 603665 INFO (SUITE-TriggerIntegrationTest-seed#[F686562254A27B7F]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_F686562254A27B7F-001/tempDir-001
[junit4] 2> 603665 INFO (SUITE-TriggerIntegrationTest-seed#[F686562254A27B7F]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 603665 INFO (Thread-1640) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 603665 INFO (Thread-1640) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 603667 ERROR (Thread-1640) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 603775 INFO (SUITE-TriggerIntegrationTest-seed#[F686562254A27B7F]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:59977
[junit4] 2> 603794 INFO (zkConnectionManagerCallback-3171-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 603805 INFO (jetty-launcher-3168-thread-1) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 603805 INFO (jetty-launcher-3168-thread-2) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 603808 INFO (jetty-launcher-3168-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 603808 INFO (jetty-launcher-3168-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 603808 INFO (jetty-launcher-3168-thread-1) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 603809 INFO (jetty-launcher-3168-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@79f54b2e{/solr,null,AVAILABLE}
[junit4] 2> 603810 INFO (jetty-launcher-3168-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 603810 INFO (jetty-launcher-3168-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 603810 INFO (jetty-launcher-3168-thread-2) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 603811 INFO (jetty-launcher-3168-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5e51a043{/solr,null,AVAILABLE}
[junit4] 2> 603813 INFO (jetty-launcher-3168-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4f808113{HTTP/1.1,[http/1.1]}{127.0.0.1:59980}
[junit4] 2> 603814 INFO (jetty-launcher-3168-thread-2) [ ] o.e.j.s.Server Started @607214ms
[junit4] 2> 603814 INFO (jetty-launcher-3168-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@2e602617{HTTP/1.1,[http/1.1]}{127.0.0.1:59979}
[junit4] 2> 603814 INFO (jetty-launcher-3168-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59980}
[junit4] 2> 603814 INFO (jetty-launcher-3168-thread-1) [ ] o.e.j.s.Server Started @607215ms
[junit4] 2> 603814 INFO (jetty-launcher-3168-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59979}
[junit4] 2> 603814 ERROR (jetty-launcher-3168-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 603814 ERROR (jetty-launcher-3168-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 603814 INFO (jetty-launcher-3168-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 603814 INFO (jetty-launcher-3168-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 603814 INFO (jetty-launcher-3168-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 603814 INFO (jetty-launcher-3168-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 603814 INFO (jetty-launcher-3168-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-25T01:13:54.766Z
[junit4] 2> 603815 INFO (jetty-launcher-3168-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 603815 INFO (jetty-launcher-3168-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 603815 INFO (jetty-launcher-3168-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-25T01:13:54.767Z
[junit4] 2> 603818 INFO (zkConnectionManagerCallback-3173-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 603819 INFO (zkConnectionManagerCallback-3175-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 603820 INFO (jetty-launcher-3168-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 603820 INFO (jetty-launcher-3168-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 603843 INFO (jetty-launcher-3168-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59977/solr
[junit4] 2> 603843 INFO (jetty-launcher-3168-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59977/solr
[junit4] 2> 603844 INFO (zkConnectionManagerCallback-3181-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 603845 INFO (zkConnectionManagerCallback-3183-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 603847 INFO (zkConnectionManagerCallback-3186-thread-1-processing-n:127.0.0.1:59979_solr) [n:127.0.0.1:59979_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 603847 INFO (zkConnectionManagerCallback-3187-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 603986 INFO (jetty-launcher-3168-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 603987 INFO (jetty-launcher-3168-thread-1) [n:127.0.0.1:59979_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 603988 INFO (jetty-launcher-3168-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59980_solr
[junit4] 2> 603988 INFO (jetty-launcher-3168-thread-1) [n:127.0.0.1:59979_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59979_solr
[junit4] 2> 603989 INFO (jetty-launcher-3168-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.Overseer Overseer (id=99407812009525254-127.0.0.1:59980_solr-n_0000000000) starting
[junit4] 2> 604004 INFO (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 604004 INFO (zkCallback-3184-thread-1-processing-n:127.0.0.1:59979_solr) [n:127.0.0.1:59979_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 604018 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 604022 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 604025 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:59979_solr]
[junit4] 2> 604026 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
[junit4] 2> 604026 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 604026 INFO (jetty-launcher-3168-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59980_solr
[junit4] 2> 604031 INFO (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 604032 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 604033 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 604033 DEBUG (ScheduledTrigger-3063-thread-1) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 604034 INFO (zkCallback-3184-thread-1-processing-n:127.0.0.1:59979_solr) [n:127.0.0.1:59979_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 604104 INFO (jetty-launcher-3168-thread-1) [n:127.0.0.1:59979_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59979.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4087a45
[junit4] 2> 604106 INFO (jetty-launcher-3168-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59980.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4087a45
[junit4] 2> 604115 INFO (jetty-launcher-3168-thread-1) [n:127.0.0.1:59979_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59979.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4087a45
[junit4] 2> 604115 INFO (jetty-launcher-3168-thread-1) [n:127.0.0.1:59979_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59979.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4087a45
[junit4] 2> 604115 INFO (jetty-launcher-3168-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59980.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4087a45
[junit4] 2> 604115 INFO (jetty-launcher-3168-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59980.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4087a45
[junit4] 2> 604116 INFO (jetty-launcher-3168-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_F686562254A27B7F-001/tempDir-001/node2/.
[junit4] 2> 604116 INFO (jetty-launcher-3168-thread-1) [n:127.0.0.1:59979_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_F686562254A27B7F-001/tempDir-001/node1/.
[junit4] 2> 604157 INFO (zkConnectionManagerCallback-3193-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 604159 INFO (zkConnectionManagerCallback-3197-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 604164 INFO (SUITE-TriggerIntegrationTest-seed#[F686562254A27B7F]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 604166 INFO (SUITE-TriggerIntegrationTest-seed#[F686562254A27B7F]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59977/solr ready
[junit4] 2> 604183 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
[junit4] 2> 604184 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 2
[junit4] 2> 604184 DEBUG (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 604184 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 604187 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 604188 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 604188 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 604192 INFO (qtp741019546-8534) [n:127.0.0.1:59979_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 604193 INFO (qtp1724343164-8535) [n:127.0.0.1:59980_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 604194 DEBUG (qtp741019546-8530) [n:127.0.0.1:59979_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 604194 INFO (qtp741019546-8530) [n:127.0.0.1:59979_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=3
[junit4] 2> 604195 DEBUG (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 604202 DEBUG (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:59979_solr, 127.0.0.1:59980_solr]
[junit4] 2> 604202 DEBUG (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
[junit4] 2> 604202 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 604204 INFO (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
[junit4] 2> 604205 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 604206 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 604206 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 604206 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 604206 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 604206 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2baf7a51{/solr,null,AVAILABLE}
[junit4] 2> 604206 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@6529ce70{HTTP/1.1,[http/1.1]}{127.0.0.1:59992}
[junit4] 2> 604206 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.e.j.s.Server Started @607608ms
[junit4] 2> 604206 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59992}
[junit4] 2> 604207 ERROR (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 604207 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 604207 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 604207 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 604207 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-25T01:13:55.159Z
[junit4] 2> 604208 INFO (zkConnectionManagerCallback-3199-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 604209 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 604209 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 604209 DEBUG (ScheduledTrigger-3063-thread-1) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 604209 DEBUG (ScheduledTrigger-3063-thread-1) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 604221 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59977/solr
[junit4] 2> 604222 INFO (zkConnectionManagerCallback-3203-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 604225 INFO (zkConnectionManagerCallback-3205-thread-1-processing-n:127.0.0.1:59992_solr) [n:127.0.0.1:59992_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 604234 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [n:127.0.0.1:59992_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 604237 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [n:127.0.0.1:59992_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 604241 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [n:127.0.0.1:59992_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59992_solr
[junit4] 2> 604243 INFO (zkCallback-3196-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 604243 INFO (zkCallback-3204-thread-1-processing-n:127.0.0.1:59992_solr) [n:127.0.0.1:59992_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 604243 INFO (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 604244 INFO (zkCallback-3184-thread-1-processing-n:127.0.0.1:59979_solr) [n:127.0.0.1:59979_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 604304 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [n:127.0.0.1:59992_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59992.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4087a45
[junit4] 2> 604311 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [n:127.0.0.1:59992_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59992.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4087a45
[junit4] 2> 604311 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [n:127.0.0.1:59992_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59992.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4087a45
[junit4] 2> 604312 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [n:127.0.0.1:59992_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_F686562254A27B7F-001/tempDir-001/node3/.
[junit4] 2> 605220 DEBUG (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 605220 DEBUG (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 605220 DEBUG (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:59992_solr at time 543355143497183
[junit4] 2> 606230 DEBUG (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 606230 DEBUG (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 607240 DEBUG (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 607240 DEBUG (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 607240 DEBUG (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:59992_solr] added at times [543355143497183], now=543357163918953
[junit4] 2> 607241 DEBUG (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"1ee2dba50a5dfT4385hg8u3das2d44y1gmansoy",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":543355143497183,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[543355143497183],
[junit4] 2> "nodeNames":["127.0.0.1:59992_solr"]}}
[junit4] 2> 607241 DEBUG (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_trigger]
[junit4] 2> 607243 INFO (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
[junit4] 2> 607244 DEBUG (AutoscalingActionExecutor-3064-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"1ee2dba50a5dfT4385hg8u3das2d44y1gmansoy",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":543355143497183,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[543355143497183],
[junit4] 2> "_enqueue_time_":543357165107735,
[junit4] 2> "nodeNames":["127.0.0.1:59992_solr"]}}
[junit4] 2> 607248 DEBUG (AutoscalingActionExecutor-3064-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 5000ms
[junit4] 2> 607248 DEBUG (AutoscalingActionExecutor-3064-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.ScheduledTriggers -- processing took 4 ms for event id=1ee2dba50a5dfT4385hg8u3das2d44y1gmansoy
[junit4] 2> 607251 INFO (qtp741019546-8534) [n:127.0.0.1:59979_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 607255 INFO (qtp1724343164-8540) [n:127.0.0.1:59980_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 607259 INFO (qtp1539384674-8592) [n:127.0.0.1:59992_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
[junit4] 2> 607260 DEBUG (qtp741019546-8532) [n:127.0.0.1:59979_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 607262 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=15
[junit4] 2> 607264 DEBUG (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 607265 DEBUG (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:59979_solr, 127.0.0.1:59980_solr, 127.0.0.1:59992_solr]
[junit4] 2> 607265 DEBUG (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
[junit4] 2> 607265 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 607266 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 607267 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 609268 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[F686562254A27B7F]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTrigger
[junit4] 2> 609308 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
[junit4] 2> 609309 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 5
[junit4] 2> 609310 DEBUG (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
[junit4] 2> 609310 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 609312 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 609313 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 609313 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
[junit4] 2> 609314 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 609316 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 609316 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 609316 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 609316 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1f9e346c{/solr,null,AVAILABLE}
[junit4] 2> 609316 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@e6dc6df{HTTP/1.1,[http/1.1]}{127.0.0.1:59999}
[junit4] 2> 609317 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.e.j.s.Server Started @612718ms
[junit4] 2> 609317 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59999}
[junit4] 2> 609317 ERROR (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 609317 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 609317 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 609317 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 609317 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-25T01:14:00.269Z
[junit4] 2> 609319 INFO (zkConnectionManagerCallback-3209-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 609320 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 609335 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59977/solr
[junit4] 2> 609336 INFO (zkConnectionManagerCallback-3213-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 609339 INFO (zkConnectionManagerCallback-3215-thread-1-processing-n:127.0.0.1:59999_solr) [n:127.0.0.1:59999_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 609351 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [n:127.0.0.1:59999_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 609355 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [n:127.0.0.1:59999_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 609359 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [n:127.0.0.1:59999_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59999_solr
[junit4] 2> 609361 INFO (zkCallback-3196-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 609361 INFO (zkCallback-3204-thread-1-processing-n:127.0.0.1:59992_solr) [n:127.0.0.1:59992_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 609361 INFO (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 609361 INFO (zkCallback-3184-thread-1-processing-n:127.0.0.1:59979_solr) [n:127.0.0.1:59979_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 609362 INFO (zkCallback-3214-thread-1-processing-n:127.0.0.1:59999_solr) [n:127.0.0.1:59999_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 609447 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [n:127.0.0.1:59999_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59999.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4087a45
[junit4] 2> 609454 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [n:127.0.0.1:59999_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59999.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4087a45
[junit4] 2> 609455 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [n:127.0.0.1:59999_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59999.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4087a45
[junit4] 2> 609456 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [n:127.0.0.1:59999_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_F686562254A27B7F-001/tempDir-001/node4/.
[junit4] 2> 609520 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 609521 INFO (qtp1724343164-8544) [n:127.0.0.1:59980_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 609523 INFO (qtp1539384674-8596) [n:127.0.0.1:59992_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 609524 INFO (qtp1096899079-8621) [n:127.0.0.1:59999_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 609525 DEBUG (qtp1724343164-8540) [n:127.0.0.1:59980_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 609525 INFO (qtp1724343164-8540) [n:127.0.0.1:59980_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=7
[junit4] 2> 609526 DEBUG (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
[junit4] 2> 609526 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@e6dc6df{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 609526 DEBUG (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:59979_solr, 127.0.0.1:59980_solr, 127.0.0.1:59992_solr, 127.0.0.1:59999_solr]
[junit4] 2> 609526 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=800029809
[junit4] 2> 609526 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
[junit4] 2> 609527 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 609527 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6c9cd645: rootName = solr_59999, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@399eeac1
[junit4] 2> 609529 INFO (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
[junit4] 2> 609529 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 609533 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
[junit4] 2> 609533 DEBUG (ScheduledTrigger-3063-thread-3) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 4
[junit4] 2> 609536 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 609536 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@26f01fc4: rootName = solr_59999, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@51f838a2
[junit4] 2> 609540 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 609540 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1849861c: rootName = solr_59999, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@15600968
[junit4] 2> 609542 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:59999_solr
[junit4] 2> 609543 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 609543 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 609544 INFO (zkCallback-3204-thread-1-processing-n:127.0.0.1:59992_solr) [n:127.0.0.1:59992_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 609544 INFO (zkCallback-3214-thread-1-processing-n:127.0.0.1:59999_solr) [n:127.0.0.1:59999_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 609544 INFO (zkCallback-3196-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 609544 INFO (zkCallback-3184-thread-1-processing-n:127.0.0.1:59979_solr) [n:127.0.0.1:59979_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 609544 INFO (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 609545 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1f9e346c{/solr,null,UNAVAILABLE}
[junit4] 2> 609546 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 610535 DEBUG (ScheduledTrigger-3063-thread-3) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 3
[junit4] 2> 610535 DEBUG (ScheduledTrigger-3063-thread-3) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:59999_solr
[junit4] 2> 611063 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 611067 INFO (qtp1724343164-8542) [n:127.0.0.1:59980_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 611070 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 611070 DEBUG (qtp741019546-8536) [n:127.0.0.1:59979_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 611072 INFO (qtp741019546-8536) [n:127.0.0.1:59979_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=12
[junit4] 2> 611073 DEBUG (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
[junit4] 2> 611074 DEBUG (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:59979_solr, 127.0.0.1:59980_solr, 127.0.0.1:59992_solr]
[junit4] 2> 611074 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
[junit4] 2> 611076 INFO (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
[junit4] 2> 611077 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:59999_solr
[junit4] 2> 611079 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 611079 DEBUG (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 3
[junit4] 2> 611080 DEBUG (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:59999_solr]
[junit4] 2> 611080 DEBUG (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"1ee2f1772cb4eT4385hg8u3das2d44y1gmansp0",
[junit4] 2> "source":"node_lost_restore_trigger",
[junit4] 2> "eventTime":543361000983374,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[543361000983374],
[junit4] 2> "nodeNames":["127.0.0.1:59999_solr"]}}
[junit4] 2> 611080 DEBUG (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_lost_restore_trigger]
[junit4] 2> 611080 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
[junit4] 2> 611081 INFO (ScheduledTrigger-3063-thread-2) [n:127.0.0.1:59980_solr ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
[junit4] 2> 611082 DEBUG (AutoscalingActionExecutor-3064-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"1ee2f1772cb4eT4385hg8u3das2d44y1gmansp0",
[junit4] 2> "source":"node_lost_restore_trigger",
[junit4] 2> "eventTime":543361000983374,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[543361000983374],
[junit4] 2> "_enqueue_time_":543361003529429,
[junit4] 2> "nodeNames":["127.0.0.1:59999_solr"]}}
[junit4] 2> 611082 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[F686562254A27B7F]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTriggerRestoreState
[junit4] 2> 611089 DEBUG (AutoscalingActionExecutor-3064-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_restore_trigger after 5000ms
[junit4] 2> 611090 DEBUG (AutoscalingActionExecutor-3064-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.ScheduledTriggers -- processing took 8 ms for event id=1ee2f1772cb4eT4385hg8u3das2d44y1gmansp0
[junit4] 2> 611124 INFO (TEST-TriggerIntegrationTest.testMetricTrigger-seed#[F686562254A27B7F]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testMetricTrigger
[junit4] 2> 611125 INFO (TEST-TriggerIntegrationTest.testMetricTrigger-seed#[F686562254A27B7F]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 8
[junit4] 2> 611126 DEBUG (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
[junit4] 2> 611126 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
[junit4] 2> 611130 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 611131 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 611132 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
[junit4] 2> 611134 INFO (zkConnectionManagerCallback-3219-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 611137 INFO (qtp741019546-8534) [n:127.0.0.1:59979_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf&name=testMetricTrigger&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 611141 INFO (OverseerThreadFactory-3065-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.c.CreateCollectionCmd Create collection testMetricTrigger
[junit4] 2> 611272 INFO (OverseerStateUpdate-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testMetricTrigger",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"testMetricTrigger_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:59979/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 611280 INFO (OverseerStateUpdate-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testMetricTrigger",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"testMetricTrigger_shard2_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:59992/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 611504 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testMetricTrigger_shard1_replica_n1&action=CREATE&numShards=2&collection=testMetricTrigger&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 611504 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=testMetricTrigger_shard2_replica_n2&action=CREATE&numShards=2&collection=testMetricTrigger&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 611506 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 611506 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 611622 INFO (zkCallback-3184-thread-1-processing-n:127.0.0.1:59979_solr) [n:127.0.0.1:59979_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 611622 INFO (zkCallback-3204-thread-1-processing-n:127.0.0.1:59992_solr) [n:127.0.0.1:59992_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 611623 INFO (zkCallback-3204-thread-2-processing-n:127.0.0.1:59992_solr) [n:127.0.0.1:59992_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 611623 INFO (zkCallback-3184-thread-2-processing-n:127.0.0.1:59979_solr) [n:127.0.0.1:59979_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 612572 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 612572 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 612589 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.s.IndexSchema [testMetricTrigger_shard1_replica_n1] Schema name=minimal
[junit4] 2> 612589 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.s.IndexSchema [testMetricTrigger_shard2_replica_n2] Schema name=minimal
[junit4] 2> 612596 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 612596 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 612596 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testMetricTrigger_shard2_replica_n2' using configuration from collection testMetricTrigger, trusted=true
[junit4] 2> 612596 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testMetricTrigger_shard1_replica_n1' using configuration from collection testMetricTrigger, trusted=true
[junit4] 2> 612597 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59992.solr.core.testMetricTrigger.shard2.replica_n2' (registry 'solr.core.testMetricTrigger.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4087a45
[junit4] 2> 612599 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 612599 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.SolrCore [[testMetricTrigger_shard2_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_F686562254A27B7F-001/tempDir-001/node3/testMetricTrigger_shard2_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_F686562254A27B7F-001/tempDir-001/node3/./testMetricTrigger_shard2_replica_n2/data/]
[junit4] 2> 612597 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59979.solr.core.testMetricTrigger.shard1.replica_n1' (registry 'solr.core.testMetricTrigger.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4087a45
[junit4] 2> 612599 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 612599 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.SolrCore [[testMetricTrigger_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_F686562254A27B7F-001/tempDir-001/node1/testMetricTrigger_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_F686562254A27B7F-001/tempDir-001/node1/./testMetricTrigger_shard1_replica_n1/data/]
[junit4] 2> 612664 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 612664 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 612664 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 612664 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 612665 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 612665 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 612665 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 612665 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 612666 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@74576b9a[testMetricTrigger_shard2_replica_n2] main]
[junit4] 2> 612666 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@efb2be1[testMetricTrigger_shard1_replica_n1] main]
[junit4] 2> 612667 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 612667 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 612668 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 612668 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 612668 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 612668 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 612668 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590525001591685120
[junit4] 2> 612668 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590525001591685120
[junit4] 2> 612670 INFO (searcherExecutor-3083-thread-1-processing-n:127.0.0.1:59992_solr x:testMetricTrigger_shard2_replica_n2 s:shard2 c:testMetricTrigger r:core_node4) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.SolrCore [testMetricTrigger_shard2_replica_n2] Registered new searcher Searcher@74576b9a[testMetricTrigger_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 612670 INFO (searcherExecutor-3084-thread-1-processing-n:127.0.0.1:59979_solr x:testMetricTrigger_shard1_replica_n1 s:shard1 c:testMetricTrigger r:core_node3) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.SolrCore [testMetricTrigger_shard1_replica_n1] Registered new searcher Searcher@efb2be1[testMetricTrigger_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 612676 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 612677 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 612677 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:59992/solr/testMetricTrigger_shard2_replica_n2/
[junit4] 2> 612677 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 612677 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 612677 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 612677 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:59979/solr/testMetricTrigger_shard1_replica_n1/
[junit4] 2> 612677 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.SyncStrategy http://127.0.0.1:59992/solr/testMetricTrigger_shard2_replica_n2/ has no replicas
[junit4] 2> 612677 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 612677 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:59979/solr/testMetricTrigger_shard1_replica_n1/ has no replicas
[junit4] 2> 612677 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 612677 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 612682 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:59979/solr/testMetricTrigger_shard1_replica_n1/ shard1
[junit4] 2> 612682 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:59992/solr/testMetricTrigger_shard2_replica_n2/ shard2
[junit4] 2> 612786 INFO (zkCallback-3184-thread-2-processing-n:127.0.0.1:59979_solr) [n:127.0.0.1:59979_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 612786 INFO (zkCallback-3204-thread-2-processing-n:127.0.0.1:59992_solr) [n:127.0.0.1:59992_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 612787 INFO (zkCallback-3204-thread-1-processing-n:127.0.0.1:59992_solr) [n:127.0.0.1:59992_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 612786 INFO (zkCallback-3184-thread-1-processing-n:127.0.0.1:59979_solr) [n:127.0.0.1:59979_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 612809 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 612809 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 612814 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testMetricTrigger_shard1_replica_n1&action=CREATE&numShards=2&collection=testMetricTrigger&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1309
[junit4] 2> 612814 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=testMetricTrigger_shard2_replica_n2&action=CREATE&numShards=2&collection=testMetricTrigger&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1309
[junit4] 2> 612821 INFO (qtp741019546-8534) [n:127.0.0.1:59979_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 612927 INFO (zkCallback-3184-thread-1-processing-n:127.0.0.1:59979_solr) [n:127.0.0.1:59979_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 612927 INFO (zkCallback-3204-thread-2-processing-n:127.0.0.1:59992_solr) [n:127.0.0.1:59992_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 612928 INFO (zkCallback-3184-thread-2-processing-n:127.0.0.1:59979_solr) [n:127.0.0.1:59979_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 612927 INFO (zkCallback-3204-thread-1-processing-n:127.0.0.1:59992_solr) [n:127.0.0.1:59992_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 613153 INFO (OverseerCollectionConfigSetProcessor-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 613823 INFO (qtp741019546-8534) [n:127.0.0.1:59979_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf&name=testMetricTrigger&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2686
[junit4] 2> 613832 INFO (qtp741019546-8534) [n:127.0.0.1:59979_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.testMetricTrigger.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
[junit4] 2> 613834 INFO (qtp741019546-8532) [n:127.0.0.1:59979_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
[junit4] 2> 613836 INFO (qtp1724343164-8540) [n:127.0.0.1:59980_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 613837 INFO (qtp1539384674-8596) [n:127.0.0.1:59992_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.testMetricTrigger.shard2.replica_n2:INDEX.sizeInBytes} status=0 QTime=0
[junit4] 2> 613839 INFO (qtp1539384674-8594) [n:127.0.0.1:59992_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
[junit4] 2> 613839 DEBUG (qtp741019546-8536) [n:127.0.0.1:59979_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 613841 INFO (qtp741019546-8536) [n:127.0.0.1:59979_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=11
[junit4] 2> 613842 DEBUG (zkCallback-3185-thread-1-processing-n:127.0.0.1:59980_solr) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
[junit4] 2> 613854 INFO (qtp741019546-8534) [n:127.0.0.1:59979_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.testMetricTrigger.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=0
[junit4] 2> 613854 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
[junit4] 2> 613855 INFO (qtp741019546-8536) [n:127.0.0.1:59979_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 613858 INFO (qtp1724343164-8540) [n:127.0.0.1:59980_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 613858 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_0000000000) [n:127.0.0.1:59980_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 613859 DEBUG (OverseerAutoScalingTriggerThread-99407812009525254-127.0.0.1:59980_solr-n_00000000
[...truncated too long message...]
eption.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$create$7(SolrZkClient.java:364)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:364)
[junit4] 2> at org.apache.solr.cloud.ZkController.lambda$registerLiveNodesListener$2(ZkController.java:879)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.lambda$refreshLiveNodes$1(ZkStateReader.java:704)
[junit4] 2> at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.refreshLiveNodes(ZkStateReader.java:703)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.access$900(ZkStateReader.java:72)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader$LiveNodeWatcher.refreshAndWatch(ZkStateReader.java:1090)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader$LiveNodeWatcher.process(ZkStateReader.java:1085)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 782696 WARN (zkCallback-3312-thread-1-processing-n:127.0.0.1:60359_solr) [n:127.0.0.1:60359_solr ] o.a.s.c.ZkController Unable to register nodeLost path for 127.0.0.1:60582_solr
[junit4] 2> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /autoscaling/nodeLost/127.0.0.1:60582_solr
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4] 2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4] 2> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.lambda$create$7(SolrZkClient.java:364)
[junit4] 2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient.create(SolrZkClient.java:364)
[junit4] 2> at org.apache.solr.cloud.ZkController.lambda$registerLiveNodesListener$2(ZkController.java:879)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.lambda$refreshLiveNodes$1(ZkStateReader.java:704)
[junit4] 2> at java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4649)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.refreshLiveNodes(ZkStateReader.java:703)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader.access$900(ZkStateReader.java:72)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader$LiveNodeWatcher.refreshAndWatch(ZkStateReader.java:1090)
[junit4] 2> at org.apache.solr.common.cloud.ZkStateReader$LiveNodeWatcher.process(ZkStateReader.java:1085)
[junit4] 2> at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268)
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 782696 INFO (jetty-closer-3169-thread-8) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1d416b0c{/solr,null,UNAVAILABLE}
[junit4] 2> 782696 INFO (jetty-closer-3169-thread-8) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 782799 INFO (jetty-closer-3169-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6b66d96f{/solr,null,UNAVAILABLE}
[junit4] 2> 782799 INFO (jetty-closer-3169-thread-2) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 782800 ERROR (SUITE-TriggerIntegrationTest-seed#[F686562254A27B7F]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 782801 INFO (SUITE-TriggerIntegrationTest-seed#[F686562254A27B7F]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59977 59977
[junit4] 2> 782818 INFO (Thread-1640) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59977 59977
[junit4] 2> 782958 WARN (Thread-1640) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 21 /solr/clusterprops.json
[junit4] 2> 21 /solr/aliases.json
[junit4] 2> 20 /solr/security.json
[junit4] 2> 7 /solr/configs/conf
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 153 /solr/collections/collection1/state.json
[junit4] 2> 21 /solr/clusterstate.json
[junit4] 2> 13 /solr/collections/testMetricTrigger/state.json
[junit4] 2> 5 /solr/autoscaling.json
[junit4] 2> 3 /solr/collections/collection1/leader_elect/shard1/election/99407812009525269-core_node3-n_0000000000
[junit4] 2> 2 /solr/overseer_elect/election/99407812009525272-127.0.0.1:60167_solr-n_0000000006
[junit4] 2> 2 /solr/collections/collection1/leader_elect/shard1/election/99407812009525272-core_node12-n_0000000005
[junit4] 2> 2 /solr/collections/collection1/leader_elect/shard1/election/99407812009525275-core_node6-n_0000000002
[junit4] 2> 2 /solr/overseer_elect/election/99407812009525290-127.0.0.1:60359_solr-n_0000000012
[junit4] 2> 2 /solr/overseer_elect/election/99407812009525296-127.0.0.1:60400_solr-n_0000000014
[junit4] 2> 2 /solr/overseer_elect/election/99407812009525259-127.0.0.1:59992_solr-n_0000000002
[junit4] 2> 2 /solr/overseer_elect/election/99407812009525281-127.0.0.1:60232_solr-n_0000000009
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 21 /solr/live_nodes
[junit4] 2> 21 /solr/collections
[junit4] 2> 5 /solr/overseer/queue-work
[junit4] 2> 5 /solr/autoscaling/events/.auto_add_replicas
[junit4] 2> 5 /solr/overseer/queue
[junit4] 2> 5 /solr/overseer/collection-queue-work
[junit4] 2> 2 /solr/autoscaling/events/node_lost_triggerMR
[junit4] 2> 2 /solr/autoscaling/events/node_added_triggerCTOOR
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_F686562254A27B7F-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {_version_=PostingsFormat(name=LuceneVarGapDocFreqInterval), id=PostingsFormat(name=Asserting), x_s=PostingsFormat(name=Memory)}, docValues:{_version_=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=1938, maxMBSortInHeap=6.566608758383209, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@e866029), locale=el-CY, timezone=Asia/Aqtau
[junit4] 2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=3,threads=1,free=83964976,total=322912256
[junit4] 2> NOTE: All tests run in this JVM: [ChangedSchemaMergeTest, TestCollapseQParserPlugin, TestReqParamsAPI, TestSha256AuthenticationProvider, SolrMetricManagerTest, TestZkAclsWithHadoopAuth, ConnectionManagerTest, V2StandaloneTest, MetricsConfigTest, BigEndianAscendingWordSerializerTest, TestSystemCollAutoCreate, TestClassicSimilarityFactory, TestGraphTermsQParserPlugin, CollectionsAPISolrJTest, TestExactStatsCache, TestGeoJSONResponseWriter, TestReplicationHandler, BlockCacheTest, TestSQLHandler, TestShortCircuitedRequests, TestEmbeddedSolrServerSchemaAPI, BlockJoinFacetRandomTest, TestNumericRangeQuery64, TestIBSimilarityFactory, TestSystemIdResolver, TestEmbeddedSolrServerAdminHandler, TestCoreContainer, SpellCheckCollatorWithCollapseTest, RulesTest, TestExecutePlanAction, DirectUpdateHandlerOptimizeTest, TestElisionMultitermQuery, TestDistributedStatsComponentCardinality, TestReload, LukeRequestHandlerTest, TestQueryUtils, TermVectorComponentDistributedTest, ConfigureRecoveryStrategyTest, ClassificationUpdateProcessorFactoryTest, TestMultiWordSynonyms, BaseCdcrDistributedZkTest, TermsComponentTest, LegacyCloudClusterPropTest, ForceLeaderTest, DistribCursorPagingTest, TestOnReconnectListenerSupport, BasicAuthIntegrationTest, SolrInfoBeanTest, TestSkipOverseerOperations, SolrPluginUtilsTest, OpenCloseCoreStressTest, TestLeaderElectionZkExpiry, TolerantUpdateProcessorTest, TestRandomFlRTGCloud, BinaryUpdateRequestHandlerTest, TestDistribDocBasedVersion, TestSlowCompositeReaderWrapper, TestCoreAdminApis, TestGroupingSearch, TestHdfsBackupRestoreCore, StatsComponentTest, TestChildDocTransformer, AddReplicaTest, TestLegacyNumericUtils, TestFieldSortValues, TriggerIntegrationTest]
[junit4] Completed [135/768 (1!)] on J0 in 179.44s, 14 tests, 3 failures <<< FAILURES!
[...truncated 50278 lines...]
[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-9) - Build # 4415 -
Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4415/
Java: 64bit/jdk-9 -XX:-UseCompressedOops -XX:+UseSerialGC
3 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.analytics.OverallAnalyticsTest
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.analytics.OverallAnalyticsTest: 1) Thread[id=66, name=qtp1315797791-66, state=TIMED_WAITING, group=TGRP-OverallAnalyticsTest] at java.base@9/jdk.internal.misc.Unsafe.park(Native Method) at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192) at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308) at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.base@9/java.lang.Thread.run(Thread.java:844)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.analytics.OverallAnalyticsTest:
1) Thread[id=66, name=qtp1315797791-66, state=TIMED_WAITING, group=TGRP-OverallAnalyticsTest]
at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.base@9/java.lang.Thread.run(Thread.java:844)
at __randomizedtesting.SeedInfo.seed([52209ED3420A94E7]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.analytics.OverallAnalyticsTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=66, name=qtp1315797791-66, state=TIMED_WAITING, group=TGRP-OverallAnalyticsTest] at java.base@9/jdk.internal.misc.Unsafe.park(Native Method) at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192) at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308) at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.base@9/java.lang.Thread.run(Thread.java:844)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=66, name=qtp1315797791-66, state=TIMED_WAITING, group=TGRP-OverallAnalyticsTest]
at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.base@9/java.lang.Thread.run(Thread.java:844)
at __randomizedtesting.SeedInfo.seed([52209ED3420A94E7]:0)
FAILED: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate
Error Message:
{srt=[CapturedEvent{timestamp=1044782017928525, stage=AFTER_ACTION, actionName='compute', event={ "id":"3b638852f4573T6lejbeoqu6skdmmwimmhol185", "source":"search_rate_trigger", "eventTime":1044778799023475, "eventType":"SEARCHRATE", "properties":{ "node":{ "127.0.0.1:10016_solr":250.0, "127.0.0.1:10013_solr":250.0}, "__start__":1, "replica":[ "{\"core_node2\":{\n \"core\":\"collection1_shard1_replica_n2\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10016_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}", "{\"core_node1\":{\n \"core\":\"collection1_shard1_replica_n1\",\n \"leader\":\"true\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10013_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}"], "collection":{"collection1":500.0}, "shard":{"collection1":{"shard1":500.0}}, "_enqueue_time_":1044781914036225, "requestedOps":[ "Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}", "Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}", "Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}"]}}, context={properties.operations=[{class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10013_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10016_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10016_solr, params.type=NRT}], properties.BEFORE_ACTION=[compute], source=search_rate_trigger, properties.AFTER_ACTION=[compute]}, config={ "trigger":"search_rate_trigger", "stage":[ "FAILED", "SUCCEEDED"], "afterAction":[ "compute", "execute", "test"], "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener", "beforeAction":[]}, message='null'}, CapturedEvent{timestamp=1044784243661125, stage=AFTER_ACTION, actionName='execute', event={ "id":"3b638852f4573T6lejbeoqu6skdmmwimmhol185", "source":"search_rate_trigger", "eventTime":1044778799023475, "eventType":"SEARCHRATE", "properties":{ "node":{ "127.0.0.1:10016_solr":250.0, "127.0.0.1:10013_solr":250.0}, "__start__":1, "replica":[ "{\"core_node2\":{\n \"core\":\"collection1_shard1_replica_n2\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10016_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}", "{\"core_node1\":{\n \"core\":\"collection1_shard1_replica_n1\",\n \"leader\":\"true\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10013_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}"], "collection":{"collection1":500.0}, "shard":{"collection1":{"shard1":500.0}}, "_enqueue_time_":1044781914036225, "requestedOps":[ "Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}", "Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}", "Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}"]}}, context={properties.operations=[{class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.async=search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0, params.waitForFinalState=true, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10013_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.async=search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0, params.waitForFinalState=true, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10016_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.async=search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0, params.waitForFinalState=true, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10016_solr, params.type=NRT}], properties.BEFORE_ACTION=[compute, execute], properties.responses=[{status={state=completed,msg=found [search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0] in completed tasks},success=}, {status={state=completed,msg=found [search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0] in completed tasks},success=}, {status={state=completed,msg=found [search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0] in completed tasks},success=}], source=search_rate_trigger, properties.AFTER_ACTION=[compute, execute]}, config={ "trigger":"search_rate_trigger", "stage":[ "FAILED", "SUCCEEDED"], "afterAction":[ "compute", "execute", "test"], "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener", "beforeAction":[]}, message='null'}, CapturedEvent{timestamp=1044784252198475, stage=AFTER_ACTION, actionName='test', event={ "id":"3b638852f4573T6lejbeoqu6skdmmwimmhol185", "source":"search_rate_trigger", "eventTime":1044778799023475, "eventType":"SEARCHRATE", "properties":{ "node":{ "127.0.0.1:10016_solr":250.0, "127.0.0.1:10013_solr":250.0}, "__start__":1, "replica":[ "{\"core_node2\":{\n \"core\":\"collection1_shard1_replica_n2\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10016_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}", "{\"core_node1\":{\n \"core\":\"collection1_shard1_replica_n1\",\n \"leader\":\"true\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10013_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}"], "collection":{"collection1":500.0}, "shard":{"collection1":{"shard1":500.0}}, "_enqueue_time_":1044781914036225, "requestedOps":[ "Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}", "Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}", "Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}"]}}, context={properties.operations=[{class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.async=search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0, params.waitForFinalState=true, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10013_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.async=search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0, params.waitForFinalState=true, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10016_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.async=search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0, params.waitForFinalState=true, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10016_solr, params.type=NRT}], properties.BEFORE_ACTION=[compute, execute, test], properties.responses=[{status={state=completed,msg=found [search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0] in completed tasks},success=}, {status={state=completed,msg=found [search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0] in completed tasks},success=}, {status={state=completed,msg=found [search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0] in completed tasks},success=}], source=search_rate_trigger, properties.AFTER_ACTION=[compute, execute, test]}, config={ "trigger":"search_rate_trigger", "stage":[ "FAILED", "SUCCEEDED"], "afterAction":[ "compute", "execute", "test"], "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener", "beforeAction":[]}, message='null'}, CapturedEvent{timestamp=1044784257796925, stage=SUCCEEDED, actionName='null', event={ "id":"3b638852f4573T6lejbeoqu6skdmmwimmhol185", "source":"search_rate_trigger", "eventTime":1044778799023475, "eventType":"SEARCHRATE", "properties":{ "node":{ "127.0.0.1:10016_solr":250.0, "127.0.0.1:10013_solr":250.0}, "__start__":1, "replica":[ "{\"core_node2\":{\n \"core\":\"collection1_shard1_replica_n2\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10016_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}", "{\"core_node1\":{\n \"core\":\"collection1_shard1_replica_n1\",\n \"leader\":\"true\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10013_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}"], "collection":{"collection1":500.0}, "shard":{"collection1":{"shard1":500.0}}, "_enqueue_time_":1044781914036225, "requestedOps":[ "Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}", "Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}", "Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}"]}}, context={}, config={ "trigger":"search_rate_trigger", "stage":[ "FAILED", "SUCCEEDED"], "afterAction":[ "compute", "execute", "test"], "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener", "beforeAction":[]}, message='null'}, CapturedEvent{timestamp=1044789475113175, stage=AFTER_ACTION, actionName='compute', event={ "id":"3b6393b28634bT6lejbeoqu6skdmmwimmhol188", "source":"search_rate_trigger", "eventTime":1044781852025675, "eventType":"SEARCHRATE", "properties":{ "node":{ "127.0.0.1:10016_solr":250.0, "127.0.0.1:10013_solr":250.0}, "__start__":1, "replica":[ "{\"core_node2\":{\n \"core\":\"collection1_shard1_replica_n2\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10016_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}", "{\"core_node1\":{\n \"core\":\"collection1_shard1_replica_n1\",\n \"leader\":\"true\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10013_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}"], "collection":{"collection1":500.0}, "shard":{"collection1":{"shard1":500.0}}, "_enqueue_time_":1044789345167425, "requestedOps":[ "Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}", "Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}", "Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}"]}}, context={properties.operations=[{class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10013_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10013_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10013_solr, params.type=NRT}], properties.BEFORE_ACTION=[compute], source=search_rate_trigger, properties.AFTER_ACTION=[compute]}, config={ "trigger":"search_rate_trigger", "stage":[ "FAILED", "SUCCEEDED"], "afterAction":[ "compute", "execute", "test"], "class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener", "beforeAction":[]}, message='null'}]} expected:<4> but was:<5>
Stack Trace:
java.lang.AssertionError: {srt=[CapturedEvent{timestamp=1044782017928525, stage=AFTER_ACTION, actionName='compute', event={
"id":"3b638852f4573T6lejbeoqu6skdmmwimmhol185",
"source":"search_rate_trigger",
"eventTime":1044778799023475,
"eventType":"SEARCHRATE",
"properties":{
"node":{
"127.0.0.1:10016_solr":250.0,
"127.0.0.1:10013_solr":250.0},
"__start__":1,
"replica":[
"{\"core_node2\":{\n \"core\":\"collection1_shard1_replica_n2\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10016_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}",
"{\"core_node1\":{\n \"core\":\"collection1_shard1_replica_n1\",\n \"leader\":\"true\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10013_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}"],
"collection":{"collection1":500.0},
"shard":{"collection1":{"shard1":500.0}},
"_enqueue_time_":1044781914036225,
"requestedOps":[
"Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}",
"Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}",
"Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}"]}}, context={properties.operations=[{class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10013_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10016_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10016_solr, params.type=NRT}], properties.BEFORE_ACTION=[compute], source=search_rate_trigger, properties.AFTER_ACTION=[compute]}, config={
"trigger":"search_rate_trigger",
"stage":[
"FAILED",
"SUCCEEDED"],
"afterAction":[
"compute",
"execute",
"test"],
"class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",
"beforeAction":[]}, message='null'}, CapturedEvent{timestamp=1044784243661125, stage=AFTER_ACTION, actionName='execute', event={
"id":"3b638852f4573T6lejbeoqu6skdmmwimmhol185",
"source":"search_rate_trigger",
"eventTime":1044778799023475,
"eventType":"SEARCHRATE",
"properties":{
"node":{
"127.0.0.1:10016_solr":250.0,
"127.0.0.1:10013_solr":250.0},
"__start__":1,
"replica":[
"{\"core_node2\":{\n \"core\":\"collection1_shard1_replica_n2\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10016_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}",
"{\"core_node1\":{\n \"core\":\"collection1_shard1_replica_n1\",\n \"leader\":\"true\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10013_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}"],
"collection":{"collection1":500.0},
"shard":{"collection1":{"shard1":500.0}},
"_enqueue_time_":1044781914036225,
"requestedOps":[
"Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}",
"Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}",
"Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}"]}}, context={properties.operations=[{class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.async=search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0, params.waitForFinalState=true, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10013_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.async=search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0, params.waitForFinalState=true, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10016_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.async=search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0, params.waitForFinalState=true, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10016_solr, params.type=NRT}], properties.BEFORE_ACTION=[compute, execute], properties.responses=[{status={state=completed,msg=found [search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0] in completed tasks},success=}, {status={state=completed,msg=found [search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0] in completed tasks},success=}, {status={state=completed,msg=found [search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0] in completed tasks},success=}], source=search_rate_trigger, properties.AFTER_ACTION=[compute, execute]}, config={
"trigger":"search_rate_trigger",
"stage":[
"FAILED",
"SUCCEEDED"],
"afterAction":[
"compute",
"execute",
"test"],
"class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",
"beforeAction":[]}, message='null'}, CapturedEvent{timestamp=1044784252198475, stage=AFTER_ACTION, actionName='test', event={
"id":"3b638852f4573T6lejbeoqu6skdmmwimmhol185",
"source":"search_rate_trigger",
"eventTime":1044778799023475,
"eventType":"SEARCHRATE",
"properties":{
"node":{
"127.0.0.1:10016_solr":250.0,
"127.0.0.1:10013_solr":250.0},
"__start__":1,
"replica":[
"{\"core_node2\":{\n \"core\":\"collection1_shard1_replica_n2\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10016_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}",
"{\"core_node1\":{\n \"core\":\"collection1_shard1_replica_n1\",\n \"leader\":\"true\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10013_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}"],
"collection":{"collection1":500.0},
"shard":{"collection1":{"shard1":500.0}},
"_enqueue_time_":1044781914036225,
"requestedOps":[
"Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}",
"Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}",
"Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}"]}}, context={properties.operations=[{class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.async=search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0, params.waitForFinalState=true, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10013_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.async=search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0, params.waitForFinalState=true, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10016_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.async=search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0, params.waitForFinalState=true, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10016_solr, params.type=NRT}], properties.BEFORE_ACTION=[compute, execute, test], properties.responses=[{status={state=completed,msg=found [search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0] in completed tasks},success=}, {status={state=completed,msg=found [search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0] in completed tasks},success=}, {status={state=completed,msg=found [search_rate_trigger/3b638852f4573T6lejbeoqu6skdmmwimmhol185/0] in completed tasks},success=}], source=search_rate_trigger, properties.AFTER_ACTION=[compute, execute, test]}, config={
"trigger":"search_rate_trigger",
"stage":[
"FAILED",
"SUCCEEDED"],
"afterAction":[
"compute",
"execute",
"test"],
"class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",
"beforeAction":[]}, message='null'}, CapturedEvent{timestamp=1044784257796925, stage=SUCCEEDED, actionName='null', event={
"id":"3b638852f4573T6lejbeoqu6skdmmwimmhol185",
"source":"search_rate_trigger",
"eventTime":1044778799023475,
"eventType":"SEARCHRATE",
"properties":{
"node":{
"127.0.0.1:10016_solr":250.0,
"127.0.0.1:10013_solr":250.0},
"__start__":1,
"replica":[
"{\"core_node2\":{\n \"core\":\"collection1_shard1_replica_n2\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10016_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}",
"{\"core_node1\":{\n \"core\":\"collection1_shard1_replica_n1\",\n \"leader\":\"true\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10013_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}"],
"collection":{"collection1":500.0},
"shard":{"collection1":{"shard1":500.0}},
"_enqueue_time_":1044781914036225,
"requestedOps":[
"Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}",
"Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}",
"Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}"]}}, context={}, config={
"trigger":"search_rate_trigger",
"stage":[
"FAILED",
"SUCCEEDED"],
"afterAction":[
"compute",
"execute",
"test"],
"class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",
"beforeAction":[]}, message='null'}, CapturedEvent{timestamp=1044789475113175, stage=AFTER_ACTION, actionName='compute', event={
"id":"3b6393b28634bT6lejbeoqu6skdmmwimmhol188",
"source":"search_rate_trigger",
"eventTime":1044781852025675,
"eventType":"SEARCHRATE",
"properties":{
"node":{
"127.0.0.1:10016_solr":250.0,
"127.0.0.1:10013_solr":250.0},
"__start__":1,
"replica":[
"{\"core_node2\":{\n \"core\":\"collection1_shard1_replica_n2\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10016_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}",
"{\"core_node1\":{\n \"core\":\"collection1_shard1_replica_n1\",\n \"leader\":\"true\",\n \"INDEX.sizeInBytes\":1000000000,\n \"rate\":250.0,\n \"node_name\":\"127.0.0.1:10013_solr\",\n \"state\":\"active\",\n \"type\":\"NRT\",\n \"QUERY./select.requestTimes:1minRate\":250.0,\n \"shard\":\"shard1\",\n \"collection\":\"collection1\"}}"],
"collection":{"collection1":500.0},
"shard":{"collection1":{"shard1":500.0}},
"_enqueue_time_":1044789345167425,
"requestedOps":[
"Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}",
"Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}",
"Op{action=ADDREPLICA, hints={COLL_SHARD={\n \"first\":\"collection1\",\n \"second\":\"shard1\"}}}"]}}, context={properties.operations=[{class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10013_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10013_solr, params.type=NRT}, {class=org.apache.solr.client.solrj.request.CollectionAdminRequest$AddReplica, method=GET, params.action=ADDREPLICA, params.collection=collection1, params.shard=shard1, params.node=127.0.0.1:10013_solr, params.type=NRT}], properties.BEFORE_ACTION=[compute], source=search_rate_trigger, properties.AFTER_ACTION=[compute]}, config={
"trigger":"search_rate_trigger",
"stage":[
"FAILED",
"SUCCEEDED"],
"afterAction":[
"compute",
"execute",
"test"],
"class":"org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerListener",
"beforeAction":[]}, message='null'}]} expected:<4> but was:<5>
at __randomizedtesting.SeedInfo.seed([D1BC188EC38B7886:8CF406070C4DDEC9]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate(TestTriggerIntegration.java:1183)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
Build Log:
[...truncated 13540 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_D1BC188EC38B7886-001/init-core-data-001
[junit4] 2> 3085570 INFO (SUITE-TestTriggerIntegration-seed#[D1BC188EC38B7886]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 3085571 INFO (SUITE-TestTriggerIntegration-seed#[D1BC188EC38B7886]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0) w/ MAC_OS_X supressed clientAuth
[junit4] 2> 3085571 INFO (SUITE-TestTriggerIntegration-seed#[D1BC188EC38B7886]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 3085572 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 3085572 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
[junit4] 2> 3085572 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10012_solr, 127.0.0.1:10011_solr]
[junit4] 2> 3085572 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
[junit4] 2> 3085573 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
[junit4] 2> 3085573 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
[junit4] 2> 3085573 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 3085573 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3085573 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 3085573 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 0
[junit4] 2> 3085573 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 3085573 DEBUG (ScheduledTrigger-12513-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3085574 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 3085574 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3085574 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 3085574 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1247)
[junit4] 2> at java.base/java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:317)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> 3085575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 3085575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 3085575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10012_solr, 127.0.0.1:10011_solr]
[junit4] 2> 3085575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
[junit4] 2> 3085575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 3085575 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3085576 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 3085576 DEBUG (ScheduledTrigger-12516-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3085602 DEBUG (ScheduledTrigger-12516-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3085635 DEBUG (ScheduledTrigger-12516-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3085666 DEBUG (ScheduledTrigger-12516-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3085684 DEBUG (simCloudManagerPool-12515-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3085685 DEBUG (simCloudManagerPool-12515-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 3085685 DEBUG (simCloudManagerPool-12515-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10012_solr, 127.0.0.1:10011_solr]
[junit4] 2> 3085685 DEBUG (simCloudManagerPool-12515-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10012_solr, 127.0.0.1:10011_solr]
[junit4] 2> 3085685 DEBUG (simCloudManagerPool-12515-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 3085685 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 3085685 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 3085686 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 3085686 DEBUG (ScheduledTrigger-12516-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 3085686 DEBUG (ScheduledTrigger-12516-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3085686 DEBUG (ScheduledTrigger-12516-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10013_solr at time 1044725189097425
[junit4] 2> 3085686 DEBUG (ScheduledTrigger-12516-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3085713 DEBUG (ScheduledTrigger-12516-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3085713 DEBUG (ScheduledTrigger-12516-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 3085713 DEBUG (ScheduledTrigger-12516-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3085737 DEBUG (ScheduledTrigger-12516-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3085737 DEBUG (ScheduledTrigger-12516-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 3085737 DEBUG (ScheduledTrigger-12516-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3085737 DEBUG (ScheduledTrigger-12516-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10013_solr] added at times [1044725189097425], now=1044727746418275
[junit4] 2> 3085737 DEBUG (ScheduledTrigger-12516-thread-1) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"3b62c09c8b7d1T6lejbeoqu6skdmmwimmhol17q",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1044725189097425,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1044725189097425],
[junit4] 2> "nodeNames":["127.0.0.1:10013_solr"]}}
[junit4] 2> 3085737 DEBUG (ScheduledTrigger-12516-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger]
[junit4] 2> 3085738 DEBUG (AutoscalingActionExecutor-12517-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"3b62c09c8b7d1T6lejbeoqu6skdmmwimmhol17q",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":1044725189097425,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1044725189097425],
[junit4] 2> "_enqueue_time_":1044727763214875,
[junit4] 2> "nodeNames":["127.0.0.1:10013_solr"]}}
[junit4] 2> 3085739 DEBUG (AutoscalingActionExecutor-12517-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 3085739 DEBUG (AutoscalingActionExecutor-12517-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
[junit4] 2> 3085739 DEBUG (AutoscalingActionExecutor-12517-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 42 ms for event id=3b62c09c8b7d1T6lejbeoqu6skdmmwimmhol17q
[junit4] 2> 3085739 DEBUG (simCloudManagerPool-12515-thread-4) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3085739 DEBUG (simCloudManagerPool-12515-thread-4) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 3085739 DEBUG (simCloudManagerPool-12515-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10013_solr, 127.0.0.1:10012_solr, 127.0.0.1:10011_solr]
[junit4] 2> 3085739 DEBUG (simCloudManagerPool-12515-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10013_solr, 127.0.0.1:10012_solr, 127.0.0.1:10011_solr]
[junit4] 2> 3085740 DEBUG (simCloudManagerPool-12515-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 3085740 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 3085740 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 3085784 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTrigger
[junit4] 2> 3085784 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 3085784 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 3085784 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 3085784 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 3085784 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 3
[junit4] 2> 3085784 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 3
[junit4] 2> 3085784 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 0
[junit4] 2> 3085784 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 3085784 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 3085784 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 3085784 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 3085784 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 3085784 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 2
[junit4] 2> 3085784 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 2
[junit4] 2> 3085785 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 3085786 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_trigger
[junit4] 2> 3085786 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 3085786 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 3085788 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testCooldown
[junit4] 2> 3085788 DEBUG (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
[junit4] 2> 3085788 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 3085789 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 3085789 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 3085789 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3085789 DEBUG (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 3085789 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1247)
[junit4] 2> at java.base/java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:317)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> 3085790 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 3085790 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
[junit4] 2> 3085790 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10013_solr, 127.0.0.1:10011_solr]
[junit4] 2> 3085790 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion -1
[junit4] 2> 3085790 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
[junit4] 2> 3085790 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3085790 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
[junit4] 2> 3085790 DEBUG (ScheduledTrigger-12519-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3085819 DEBUG (ScheduledTrigger-12519-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3085850 DEBUG (ScheduledTrigger-12519-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3085879 DEBUG (ScheduledTrigger-12519-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3085900 DEBUG (ScheduledTrigger-12519-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3085900 DEBUG (simCloudManagerPool-12518-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3085900 DEBUG (simCloudManagerPool-12518-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
[junit4] 2> 3085901 DEBUG (simCloudManagerPool-12518-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10013_solr, 127.0.0.1:10011_solr]
[junit4] 2> 3085901 DEBUG (simCloudManagerPool-12518-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 3085901 DEBUG (simCloudManagerPool-12518-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10013_solr, 127.0.0.1:10011_solr]
[junit4] 2> 3085901 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
[junit4] 2> 3085901 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 3085901 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
[junit4] 2> 3085901 DEBUG (ScheduledTrigger-12519-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
[junit4] 2> 3085902 DEBUG (ScheduledTrigger-12519-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 3085903 DEBUG (simCloudManagerPool-12518-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3085903 DEBUG (simCloudManagerPool-12518-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
[junit4] 2> 3085903 DEBUG (simCloudManagerPool-12518-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:10013_solr, 127.0.0.1:10011_solr]
[junit4] 2> 3085903 DEBUG (simCloudManagerPool-12518-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 3085903 DEBUG (simCloudManagerPool-12518-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10013_solr, 127.0.0.1:10011_solr]
[junit4] 2> 3085903 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
[junit4] 2> 3085903 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
[junit4] 2> 3085930 DEBUG (ScheduledTrigger-12519-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3085930 DEBUG (ScheduledTrigger-12519-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
[junit4] 2> 3085930 DEBUG (ScheduledTrigger-12519-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3085930 DEBUG (ScheduledTrigger-12519-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10014_solr at time 1044737419320175
[junit4] 2> 3085960 DEBUG (ScheduledTrigger-12519-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3085960 DEBUG (ScheduledTrigger-12519-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
[junit4] 2> 3085961 DEBUG (ScheduledTrigger-12519-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3085961 DEBUG (ScheduledTrigger-12519-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger firing registered processor for nodes: [127.0.0.1:10014_solr] added at times [1044737419320175], now=1044738930187625
[junit4] 2> 3085961 DEBUG (ScheduledTrigger-12519-thread-4) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"3b62ee2c31b6fT6lejbeoqu6skdmmwimmhol17t",
[junit4] 2> "source":"node_added_cooldown_trigger",
[junit4] 2> "eventTime":1044737419320175,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1044737419320175],
[junit4] 2> "nodeNames":["127.0.0.1:10014_solr"]}}
[junit4] 2> 3085961 DEBUG (ScheduledTrigger-12519-thread-4) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_cooldown_trigger]
[junit4] 2> 3085962 DEBUG (AutoscalingActionExecutor-12520-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"3b62ee2c31b6fT6lejbeoqu6skdmmwimmhol17t",
[junit4] 2> "source":"node_added_cooldown_trigger",
[junit4] 2> "eventTime":1044737419320175,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1044737419320175],
[junit4] 2> "_enqueue_time_":1044738952048925,
[junit4] 2> "nodeNames":["127.0.0.1:10014_solr"]}}
[junit4] 2> 3085963 DEBUG (AutoscalingActionExecutor-12520-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 3085963 DEBUG (AutoscalingActionExecutor-12520-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_cooldown_trigger after 100ms
[junit4] 2> 3085963 DEBUG (AutoscalingActionExecutor-12520-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 63 ms for event id=3b62ee2c31b6fT6lejbeoqu6skdmmwimmhol17t
[junit4] 2> 3086073 DEBUG (ScheduledTrigger-12519-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 3086073 DEBUG (ScheduledTrigger-12519-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
[junit4] 2> 3086074 DEBUG (ScheduledTrigger-12519-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 3086074 DEBUG (ScheduledTrigger-12519-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10015_solr at time 1044744583579625
[junit4] 2> 3086105 DEBUG (ScheduledTrigger-12519-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 3086105 DEBUG (ScheduledTrigger-12519-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_cooldown_trigger
[junit4] 2> 3086105 DEBUG (ScheduledTrigger-12519-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 3086106 DEBUG (ScheduledTrigger-12519-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_cooldown_trigger firing registered processor for nodes: [127.0.0.1:10015_solr] added at times [1044744583579625], now=1044746176894025
[junit4] 2> 3086106 DEBUG (ScheduledTrigger-12519-thread-3) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"3b6308dc907e9T6lejbeoqu6skdmmwimmhol17w",
[junit4] 2> "source":"node_added_cooldown_trigger",
[junit4] 2> "eventTime":1044744583579625,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1044744583579625],
[junit4] 2> "nodeNames":["127.0.0.1:10015_solr"]}}
[junit4] 2> 3086106 DEBUG (ScheduledTrigger-12519-thread-3) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_cooldown_trigger]
[junit4] 2> 3086106 DEBUG (AutoscalingActionExecutor-12520-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"3b6308dc907e9T6lejbeoqu6skdmmwimmhol17w",
[junit4] 2> "source":"node_added_cooldown_trigger",
[junit4] 2> "eventTime":1044744583579625,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1044744583579625],
[junit4] 2> "_enqueue_time_":1044746187114025,
[junit4] 2> "nodeNames":["127.0.0.1:10015_solr"]}}
[junit4] 2> 3086107 DEBUG (AutoscalingActionExecutor-12520-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 3086107 DEBUG (AutoscalingActionExecutor-12520-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_cooldown_trigger after 100ms
[junit4] 2> 3086107 DEBUG (AutoscalingActionExecutor-12520-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 41 ms for event id=3b6308dc907e9T6lejbeoqu6skdmmwimmhol17w
[junit4] 2> 3086161 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testCooldown
[junit4] 2> 3086161 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 3086161 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 3086161 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 3086162 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 3086162 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 4
[junit4] 2> 3086162 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 4
[junit4] 2> 3086162 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 0
[junit4] 2> 3086162 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 3086162 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 3086162 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 3086162 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 3086162 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 3086162 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 2
[junit4] 2> 3086162 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 4
[junit4] 2> 3086162 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 3086162 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_added_cooldown_trigger
[junit4] 2> 3086162 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 2
[junit4] 2> 3086162 INFO (TEST-TestTriggerIntegration.testCooldown-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 2
[junit4] 2> 3086165 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTrigger
[junit4] 2> 3086165 DEBUG (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
[junit4] 2> 3086165 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
[junit4] 2> 3086165 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 3086165 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 3086166 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3086166 DEBUG (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 3086166 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1247)
[junit4] 2> at java.base/java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:317)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:154)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> 3086167 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 3086167 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
[junit4] 2> 3086167 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10013_solr, 127.0.0.1:10011_solr]
[junit4] 2> 3086167 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion -1
[junit4] 2> 3086167 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
[junit4] 2> 3086167 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3086167 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
[junit4] 2> 3086167 DEBUG (ScheduledTrigger-12522-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3086191 DEBUG (ScheduledTrigger-12522-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3086222 DEBUG (ScheduledTrigger-12522-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3086247 DEBUG (ScheduledTrigger-12522-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3086276 DEBUG (ScheduledTrigger-12522-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3086277 DEBUG (simCloudManagerPool-12521-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3086277 DEBUG (simCloudManagerPool-12521-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
[junit4] 2> 3086277 DEBUG (simCloudManagerPool-12521-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10013_solr, 127.0.0.1:10011_solr]
[junit4] 2> 3086277 DEBUG (simCloudManagerPool-12521-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10013_solr, 127.0.0.1:10011_solr]
[junit4] 2> 3086278 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
[junit4] 2> 3086278 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 3086278 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3086278 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
[junit4] 2> 3086278 DEBUG (ScheduledTrigger-12522-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 3086278 DEBUG (ScheduledTrigger-12522-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10011_solr
[junit4] 2> 3086305 DEBUG (ScheduledTrigger-12522-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 3086305 DEBUG (ScheduledTrigger-12522-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10011_solr
[junit4] 2> 3086305 DEBUG (ScheduledTrigger-12522-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 3086335 DEBUG (ScheduledTrigger-12522-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 3086335 DEBUG (ScheduledTrigger-12522-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 3086365 DEBUG (ScheduledTrigger-12522-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 3086365 DEBUG (ScheduledTrigger-12522-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 3086365 DEBUG (ScheduledTrigger-12522-thread-1) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10011_solr]
[junit4] 2> 3086365 DEBUG (ScheduledTrigger-12522-thread-1) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"3b632ef3e1625T6lejbeoqu6skdmmwimmhol17z",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":1044754808575525,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1044754808575525],
[junit4] 2> "nodeNames":["127.0.0.1:10011_solr"]}}
[junit4] 2> 3086366 DEBUG (ScheduledTrigger-12522-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_lost_trigger]
[junit4] 2> 3086367 DEBUG (AutoscalingActionExecutor-12523-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"3b632ef3e1625T6lejbeoqu6skdmmwimmhol17z",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":1044754808575525,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1044754808575525],
[junit4] 2> "_enqueue_time_":1044759188643375,
[junit4] 2> "nodeNames":["127.0.0.1:10011_solr"]}}
[junit4] 2> 3086367 DEBUG (AutoscalingActionExecutor-12523-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 3086367 DEBUG (AutoscalingActionExecutor-12523-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger after 100ms
[junit4] 2> 3086367 DEBUG (AutoscalingActionExecutor-12523-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 37 ms for event id=3b632ef3e1625T6lejbeoqu6skdmmwimmhol17z
[junit4] 2> 3086368 DEBUG (simCloudManagerPool-12521-thread-4) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3086368 DEBUG (simCloudManagerPool-12521-thread-4) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 12
[junit4] 2> 3086368 DEBUG (simCloudManagerPool-12521-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10013_solr]
[junit4] 2> 3086368 DEBUG (simCloudManagerPool-12521-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10013_solr]
[junit4] 2> 3086368 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 12
[junit4] 2> 3086368 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3086368 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion 12
[junit4] 2> 3086418 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTrigger
[junit4] 2> 3086418 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase
[junit4] 2>
[junit4] 2> 3086418 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2> 3086418 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ############ FINAL CLUSTER STATS ############
[junit4] 2> 3086418 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #############################################
[junit4] 2>
[junit4] 2> 3086418 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Live nodes: 1
[junit4] 2> 3086418 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Empty nodes: 1
[junit4] 2> 3086419 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Dead nodes: 1
[junit4] 2> 3086419 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## 127.0.0.1:10011_solr
[junit4] 2> 3086419 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Collections: []
[junit4] 2> 3086419 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Max replicas per node: 0
[junit4] 2> 3086419 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Min replicas per node: 0
[junit4] 2> 3086419 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## Total replicas: 0
[junit4] 2> 3086419 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Final Solr op counts ##########
[junit4] 2> 3086419 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - autoscaling 2
[junit4] 2> 3086419 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - update 2
[junit4] 2> 3086419 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ######### Autoscaling event counts ###########
[junit4] 2> 3086419 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## * Trigger: node_lost_trigger
[junit4] 2> 3086419 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - STARTED 1
[junit4] 2> 3086419 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase ## - SUCCEEDED 1
[junit4] 2> 3086421 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[D1BC188EC38B7886]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
[junit4] 2> 3086422 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 13
[junit4] 2> 3086422 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 13
[junit4] 2> 3086422 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 3086422 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3086422 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10011_solr
[junit4] 2> 3086422 INFO (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 3086422 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion 13
[junit4] 2> 3086422 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException
[junit4] 2> at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2057)
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:166)
[junit4] 2> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> 3086423 DEBUG (TEST-TestTriggerIntegration.testNodeLostTriggerRestoreState-seed#[D1BC188EC38B7886]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 3086424 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 3086424 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 14
[junit4] 2> 3086424 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10013_solr]
[junit4] 2> 3086424 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion -1
[junit4] 2> 3086424 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 14
[junit4] 2> 3086424 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3086424 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion 14
[junit4] 2> 3086425 DEBUG (ScheduledTrigger-12525-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3086448 DEBUG (ScheduledTrigger-12525-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3086479 DEBUG (ScheduledTrigger-12525-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3086508 DEBUG (ScheduledTrigger-12525-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3086533 DEBUG (ScheduledTrigger-12525-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3086534 DEBUG (simCloudManagerPool-12524-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3086534 DEBUG (simCloudManagerPool-12524-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 15
[junit4] 2> 3086535 DEBUG (simCloudManagerPool-12524-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10013_solr, 127.0.0.1:10017_solr]
[junit4] 2> 3086535 DEBUG (simCloudManagerPool-12524-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10013_solr, 127.0.0.1:10017_solr]
[junit4] 2> 3086535 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 15
[junit4] 2> 3086535 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 3086535 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3086535 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion 15
[junit4] 2> 3086535 DEBUG (ScheduledTrigger-12525-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2
[junit4] 2> 3086535 DEBUG (ScheduledTrigger-12525-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10017_solr
[junit4] 2> 3086554 DEBUG (ScheduledTrigger-12525-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3086557 DEBUG (ScheduledTrigger-12525-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2
[junit4] 2> 3086567 DEBUG (simCloudManagerPool-12524-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3086567 DEBUG (simCloudManagerPool-12524-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 16
[junit4] 2> 3086567 DEBUG (simCloudManagerPool-12524-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10013_solr]
[junit4] 2> 3086567 DEBUG (simCloudManagerPool-12524-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10013_solr]
[junit4] 2> 3086567 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 16
[junit4] 2> 3086567 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 3086567 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3086567 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 16, lastZnodeVersion 16
[junit4] 2> 3086567 DEBUG (ScheduledTrigger-12525-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 2
[junit4] 2> 3086567 DEBUG (ScheduledTrigger-12525-thread-4) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10017_solr]
[junit4] 2> 3086568 DEBUG (ScheduledTrigger-12525-thread-4) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"3b635ed61cabfT6lejbeoqu6skdmmwimmhol182",
[junit4] 2> "source":"node_lost_restore_trigger",
[junit4] 2> "eventTime":1044767662262975,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[1044767662262975],
[junit4] 2> "nodeNames":["127.0.0.1:10017_solr"]}}
[junit4] 2> 3086568 DEBUG (ScheduledTrigger-12525-thread-4) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_lost_restore_trigger]
[junit4] 2> 3086569 DEBUG (AutoscalingActionExecutor-12526-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"3b635ed61cabfT6lejbeoqu6skdmmwimmhol182",
[junit4] 2> "source":"node_lost_restore_trigger",
[junit4] 2> "eventTime":1044767662262975,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[10447
[...truncated too long message...]
ons, SimpleFacetsTest, StatsComponentTest, TestGroupingSearch, BasicFunctionalityTest, DirectUpdateHandlerTest, ShowFileRequestHandlerTest, SignatureUpdateProcessorFactoryTest, TestFoldingMultitermQuery, SuggesterTSTTest, SuggesterWFSTTest, SchemaVersionSpecificBehaviorTest, FieldMutatingUpdateProcessorTest, SortByFunctionTest, DocValuesMultiTest, DistanceFunctionTest, XsltUpdateRequestHandlerTest, TestReversedWildcardFilterFactory, RequestHandlersTest, DocumentBuilderTest, TestValueSourceCache, PathHierarchyTokenizerFactoryTest, FieldAnalysisRequestHandlerTest, IndexSchemaRuntimeFieldTest, SolrPluginUtilsTest, TestCSVResponseWriter, TestAnalyzedSuggestions, TestPartialUpdateDeduplication, TestComponentsName, TestLFUCache, TimeZoneUtilsTest, ResourceLoaderTest, OpenExchangeRatesOrgProviderTest, TestFastLRUCache, TestSystemIdResolver, PrimUtilsTest, TestSuggestSpellingConverter, SpellingQueryConverterTest, DOMUtilTest, RAMDirectoryFactoryTest, TestSolrJ, TestLRUCache, ZkNodePropsTest, SystemInfoHandlerTest, UUIDFieldTest, DistributedMLTComponentTest, SolrTestCaseJ4Test, TestCrossCoreJoin, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestHighlightDedupGrouping, TestLuceneIndexBackCompat, TestEmbeddedSolrServerAdminHandler, AddReplicaTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, DeleteInactiveReplicaTest, DeleteNodeTest, DeleteReplicaTest, DistribJoinFromCollectionTest, HealthCheckHandlerTest, LIRRollingUpdatesTest, MissingSegmentRecoveryTest, MultiThreadedOCPTest, ReplaceNodeTest, SolrXmlInZkTest, TestAuthenticationFramework, TestCloudDeleteByQuery, TestCloudInspectUtil, TestCloudJSONFacetJoinDomain, TestCloudPseudoReturnFields, TestClusterProperties, TestConfigSetsAPIZkFailure, TestDeleteCollectionOnDownNodes, TestLeaderElectionWithEmptyReplica, TestLeaderInitiatedRecoveryThread, TestRandomFlRTGCloud, TestRandomRequestDistribution, TestSSLRandomization, TestSolrCloudWithDelegationTokens, TestSolrCloudWithSecureImpersonation, TestStressLiveNodes, TestTolerantUpdateProcessorCloud, TlogReplayBufferedWhileIndexingTest, TriLevelCompositeIdRoutingTest, AssignTest, SimpleCollectionCreateDeleteTest, TestLocalFSCloudBackupRestore, AutoScalingHandlerTest, ExecutePlanActionTest, HdfsAutoAddReplicasIntegrationTest, SystemLogListenerTest, TestClusterStateProvider, TestDistribStateManager, TestGenericDistributedQueue, TestSimDistributedQueue, TestTriggerIntegration]
[junit4] Completed [588/773 (1!)] on J1 in 3.63s, 11 tests, 1 failure <<< FAILURES!
[...truncated 3730 lines...]
[junit4] Suite: org.apache.solr.analytics.OverallAnalyticsTest
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/contrib/solr-analytics/test/J0/temp/solr.analytics.OverallAnalyticsTest_52209ED3420A94E7-001/init-core-data-001
[junit4] 2> Jan 30, 2018 6:07:46 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
[junit4] 2> Jan 30, 2018 6:08:06 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.analytics.OverallAnalyticsTest:
[junit4] 2> 1) Thread[id=66, name=qtp1315797791-66, state=TIMED_WAITING, group=TGRP-OverallAnalyticsTest]
[junit4] 2> at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
[junit4] 2> at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
[junit4] 2> at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
[junit4] 2> at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] 2> at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] 2> at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] 2> at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] 2> at java.base@9/java.lang.Thread.run(Thread.java:844)
[junit4] 2> Jan 30, 2018 6:08:06 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=66, name=qtp1315797791-66, state=TIMED_WAITING, group=TGRP-OverallAnalyticsTest]
[junit4] 2> Jan 30, 2018 6:08:09 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> SEVERE: There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=66, name=qtp1315797791-66, state=TIMED_WAITING, group=TGRP-OverallAnalyticsTest]
[junit4] 2> at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
[junit4] 2> at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
[junit4] 2> at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
[junit4] 2> at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] 2> at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] 2> at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] 2> at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] 2> at java.base@9/java.lang.Thread.run(Thread.java:844)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {string_s=PostingsFormat(name=MockRandom), id=BlockTreeOrds(blocksize=128), string_sm=Lucene50(blocksize=128)}, docValues:{int_i=DocValuesFormat(name=Asserting), double_dm=DocValuesFormat(name=Memory), float_fm=DocValuesFormat(name=Lucene70), int_im=DocValuesFormat(name=Lucene70), date_dt=DocValuesFormat(name=Asserting), date_dtm=DocValuesFormat(name=Direct), long_lm=DocValuesFormat(name=Direct), double_d=DocValuesFormat(name=Lucene70), float_f=DocValuesFormat(name=Memory), long_l=DocValuesFormat(name=Asserting), _version_=DocValuesFormat(name=Direct), string_s=DocValuesFormat(name=Asserting), id=DocValuesFormat(name=Memory), string_sm=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1406, maxMBSortInHeap=5.042074920574478, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@1aafae87), locale=mgo, timezone=America/New_York
[junit4] 2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 9 (64-bit)/cpus=3,threads=2,free=33909272,total=98959360
[junit4] 2> NOTE: All tests run in this JVM: [LegacyNoFacetTest, CastingDoubleValueTest, BottomFunctionTest, GTFunctionTest, CastingFloatValueTest, OverallAnalyticsTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=OverallAnalyticsTest -Dtests.seed=52209ED3420A94E7 -Dtests.slow=true -Dtests.locale=mgo -Dtests.timezone=America/New_York -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J0 | OverallAnalyticsTest (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.analytics.OverallAnalyticsTest:
[junit4] > 1) Thread[id=66, name=qtp1315797791-66, state=TIMED_WAITING, group=TGRP-OverallAnalyticsTest]
[junit4] > at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
[junit4] > at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
[junit4] > at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
[junit4] > at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] > at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] > at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] > at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.base@9/java.lang.Thread.run(Thread.java:844)
[junit4] > at __randomizedtesting.SeedInfo.seed([52209ED3420A94E7]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=66, name=qtp1315797791-66, state=TIMED_WAITING, group=TGRP-OverallAnalyticsTest]
[junit4] > at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
[junit4] > at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
[junit4] > at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
[junit4] > at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] > at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] > at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] > at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.base@9/java.lang.Thread.run(Thread.java:844)
[junit4] > at __randomizedtesting.SeedInfo.seed([52209ED3420A94E7]:0)
[junit4] Completed [59/78 (1!)] on J0 in 58.02s, 1 test, 2 errors <<< FAILURES!
[...truncated 37733 lines...]
[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4414
- Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4414/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
4 tests failed.
FAILED: org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple
Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:60245/solr]
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:60245/solr]
at __randomizedtesting.SeedInfo.seed([4A8EF17A1C4BEB94:723DD5843BB83F45]:0)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple(AutoAddReplicasPlanActionTest.java:110)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:60245/solr
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
... 44 more
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:60245 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
... 48 more
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.request.SchemaTest
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.SchemaTest: 1) Thread[id=1276, name=qtp1023729175-1276, state=TIMED_WAITING, group=TGRP-SchemaTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.SchemaTest:
1) Thread[id=1276, name=qtp1023729175-1276, state=TIMED_WAITING, group=TGRP-SchemaTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([A2432F7B5D8124F8]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.request.SchemaTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=1276, name=qtp1023729175-1276, state=TIMED_WAITING, group=TGRP-SchemaTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=1276, name=qtp1023729175-1276, state=TIMED_WAITING, group=TGRP-SchemaTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([A2432F7B5D8124F8]:0)
FAILED: org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory
Error Message:
expected:<5> but was:<0>
Stack Trace:
java.lang.AssertionError: expected:<5> but was:<0>
at __randomizedtesting.SeedInfo.seed([4A8EF17A1C4BEB94:27725587A6031493]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory(AutoscalingHistoryHandlerTest.java:244)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12203 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest
[junit4] 2> 1101170 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[4A8EF17A1C4BEB94]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/init-core-data-001
[junit4] 2> 1101171 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[4A8EF17A1C4BEB94]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 1101174 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[4A8EF17A1C4BEB94]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
[junit4] 2> 1101174 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[4A8EF17A1C4BEB94]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001
[junit4] 2> 1101174 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[4A8EF17A1C4BEB94]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1101175 INFO (Thread-2008) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1101175 INFO (Thread-2008) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1101181 ERROR (Thread-2008) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1101287 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[4A8EF17A1C4BEB94]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:60241
[junit4] 2> 1101297 INFO (zkConnectionManagerCallback-3205-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1101302 INFO (jetty-launcher-3202-thread-1) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 1101302 INFO (jetty-launcher-3202-thread-3) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 1101302 INFO (jetty-launcher-3202-thread-2) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 1101303 INFO (jetty-launcher-3202-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1101303 INFO (jetty-launcher-3202-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1101303 INFO (jetty-launcher-3202-thread-2) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 1101304 INFO (jetty-launcher-3202-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1101304 INFO (jetty-launcher-3202-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1101304 INFO (jetty-launcher-3202-thread-1) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 1101304 INFO (jetty-launcher-3202-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5b675389{/solr,null,AVAILABLE}
[junit4] 2> 1101304 INFO (jetty-launcher-3202-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@8b05a09{/solr,null,AVAILABLE}
[junit4] 2> 1101305 INFO (jetty-launcher-3202-thread-3) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1101305 INFO (jetty-launcher-3202-thread-3) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1101305 INFO (jetty-launcher-3202-thread-3) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 1101306 INFO (jetty-launcher-3202-thread-3) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@72bb620f{/solr,null,AVAILABLE}
[junit4] 2> 1101308 INFO (jetty-launcher-3202-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@7088bbf5{HTTP/1.1,[http/1.1]}{127.0.0.1:60244}
[junit4] 2> 1101308 INFO (jetty-launcher-3202-thread-1) [ ] o.e.j.s.Server Started @1104340ms
[junit4] 2> 1101308 INFO (jetty-launcher-3202-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@63df3ea6{HTTP/1.1,[http/1.1]}{127.0.0.1:60243}
[junit4] 2> 1101308 INFO (jetty-launcher-3202-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60244}
[junit4] 2> 1101308 INFO (jetty-launcher-3202-thread-2) [ ] o.e.j.s.Server Started @1104340ms
[junit4] 2> 1101308 INFO (jetty-launcher-3202-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60243}
[junit4] 2> 1101308 ERROR (jetty-launcher-3202-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1101308 ERROR (jetty-launcher-3202-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1101308 INFO (jetty-launcher-3202-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 1101308 INFO (jetty-launcher-3202-thread-3) [ ] o.e.j.s.AbstractConnector Started ServerConnector@1249c3a9{HTTP/1.1,[http/1.1]}{127.0.0.1:60245}
[junit4] 2> 1101309 INFO (jetty-launcher-3202-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1101309 INFO (jetty-launcher-3202-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 1101309 INFO (jetty-launcher-3202-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1101309 INFO (jetty-launcher-3202-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1101309 INFO (jetty-launcher-3202-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-29T18:34:49.372Z
[junit4] 2> 1101309 INFO (jetty-launcher-3202-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1101309 INFO (jetty-launcher-3202-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-29T18:34:49.372Z
[junit4] 2> 1101309 INFO (jetty-launcher-3202-thread-3) [ ] o.e.j.s.Server Started @1104340ms
[junit4] 2> 1101309 INFO (jetty-launcher-3202-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=60245}
[junit4] 2> 1101309 ERROR (jetty-launcher-3202-thread-3) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1101309 INFO (jetty-launcher-3202-thread-3) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 1101309 INFO (jetty-launcher-3202-thread-3) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1101309 INFO (jetty-launcher-3202-thread-3) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1101309 INFO (jetty-launcher-3202-thread-3) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-29T18:34:49.372Z
[junit4] 2> 1101310 INFO (zkConnectionManagerCallback-3207-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1101311 INFO (zkConnectionManagerCallback-3209-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1101311 INFO (zkConnectionManagerCallback-3211-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1101311 INFO (jetty-launcher-3202-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1101311 INFO (jetty-launcher-3202-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1101312 INFO (jetty-launcher-3202-thread-3) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1101327 INFO (jetty-launcher-3202-thread-3) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60241/solr
[junit4] 2> 1101327 INFO (jetty-launcher-3202-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60241/solr
[junit4] 2> 1101328 INFO (jetty-launcher-3202-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60241/solr
[junit4] 2> 1101328 INFO (zkConnectionManagerCallback-3218-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1101328 INFO (zkConnectionManagerCallback-3220-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1101329 INFO (zkConnectionManagerCallback-3223-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1101331 INFO (zkConnectionManagerCallback-3225-thread-1-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1101331 INFO (zkConnectionManagerCallback-3227-thread-1-processing-n:127.0.0.1:60244_solr) [n:127.0.0.1:60244_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1101331 INFO (zkConnectionManagerCallback-3229-thread-1-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1101476 INFO (jetty-launcher-3202-thread-2) [n:127.0.0.1:60243_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1101476 INFO (jetty-launcher-3202-thread-1) [n:127.0.0.1:60244_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1101477 INFO (jetty-launcher-3202-thread-2) [n:127.0.0.1:60243_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:60243_solr
[junit4] 2> 1101478 INFO (jetty-launcher-3202-thread-1) [n:127.0.0.1:60244_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60244_solr
[junit4] 2> 1101478 INFO (jetty-launcher-3202-thread-2) [n:127.0.0.1:60243_solr ] o.a.s.c.Overseer Overseer (id=72114465450819593-127.0.0.1:60243_solr-n_0000000000) starting
[junit4] 2> 1101479 INFO (jetty-launcher-3202-thread-3) [n:127.0.0.1:60245_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1101480 INFO (zkCallback-3226-thread-1-processing-n:127.0.0.1:60244_solr) [n:127.0.0.1:60244_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1101480 INFO (zkCallback-3228-thread-1-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1101481 INFO (zkCallback-3224-thread-1-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1101483 INFO (jetty-launcher-3202-thread-3) [n:127.0.0.1:60245_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60245_solr
[junit4] 2> 1101484 INFO (zkCallback-3226-thread-1-processing-n:127.0.0.1:60244_solr) [n:127.0.0.1:60244_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1101485 INFO (zkCallback-3224-thread-1-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1101485 INFO (zkCallback-3228-thread-1-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1101511 INFO (jetty-launcher-3202-thread-2) [n:127.0.0.1:60243_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60243_solr
[junit4] 2> 1101522 INFO (zkCallback-3224-thread-1-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1101522 INFO (zkCallback-3226-thread-1-processing-n:127.0.0.1:60244_solr) [n:127.0.0.1:60244_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1101522 INFO (zkCallback-3228-thread-1-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1101639 INFO (jetty-launcher-3202-thread-2) [n:127.0.0.1:60243_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60243.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d70a389
[junit4] 2> 1101657 INFO (jetty-launcher-3202-thread-2) [n:127.0.0.1:60243_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60243.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d70a389
[junit4] 2> 1101657 INFO (jetty-launcher-3202-thread-2) [n:127.0.0.1:60243_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60243.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d70a389
[junit4] 2> 1101659 INFO (jetty-launcher-3202-thread-2) [n:127.0.0.1:60243_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001/node2/.
[junit4] 2> 1101660 INFO (jetty-launcher-3202-thread-3) [n:127.0.0.1:60245_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60245.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d70a389
[junit4] 2> 1101695 INFO (jetty-launcher-3202-thread-3) [n:127.0.0.1:60245_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60245.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d70a389
[junit4] 2> 1101695 INFO (jetty-launcher-3202-thread-3) [n:127.0.0.1:60245_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60245.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d70a389
[junit4] 2> 1101697 INFO (jetty-launcher-3202-thread-3) [n:127.0.0.1:60245_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001/node3/.
[junit4] 2> 1101701 INFO (jetty-launcher-3202-thread-1) [n:127.0.0.1:60244_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60244.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d70a389
[junit4] 2> 1101719 INFO (jetty-launcher-3202-thread-1) [n:127.0.0.1:60244_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60244.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d70a389
[junit4] 2> 1101719 INFO (jetty-launcher-3202-thread-1) [n:127.0.0.1:60244_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60244.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d70a389
[junit4] 2> 1101722 INFO (jetty-launcher-3202-thread-1) [n:127.0.0.1:60244_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001/node1/.
[junit4] 2> 1101837 INFO (zkConnectionManagerCallback-3237-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1101841 INFO (zkConnectionManagerCallback-3241-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1101842 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[4A8EF17A1C4BEB94]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 1101844 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[4A8EF17A1C4BEB94]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:60241/solr ready
[junit4] 2> 1101880 INFO (TEST-AutoAddReplicasPlanActionTest.testSimple-seed#[4A8EF17A1C4BEB94]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testSimple
[junit4] 2> 1101884 INFO (qtp380382596-11526) [n:127.0.0.1:60243_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&maxShardsPerNode=2&autoAddReplicas=true&name=testSimple1&nrtReplicas=2&action=CREATE&numShards=2&createNodeSet=127.0.0.1:60243_solr,127.0.0.1:60245_solr&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1101887 INFO (OverseerThreadFactory-4211-thread-1-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.a.c.CreateCollectionCmd Create collection testSimple1
[junit4] 2> 1102005 INFO (OverseerStateUpdate-72114465450819593-127.0.0.1:60243_solr-n_0000000000) [n:127.0.0.1:60243_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testSimple1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"testSimple1_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:60243/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1102009 INFO (OverseerStateUpdate-72114465450819593-127.0.0.1:60243_solr-n_0000000000) [n:127.0.0.1:60243_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testSimple1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"testSimple1_shard1_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:60245/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1102013 INFO (OverseerStateUpdate-72114465450819593-127.0.0.1:60243_solr-n_0000000000) [n:127.0.0.1:60243_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testSimple1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"testSimple1_shard2_replica_n4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:60243/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1102017 INFO (OverseerStateUpdate-72114465450819593-127.0.0.1:60243_solr-n_0000000000) [n:127.0.0.1:60243_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testSimple1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"testSimple1_shard2_replica_n6",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:60245/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1102231 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n1&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1102232 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n6&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1102233 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n4&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1102233 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n2&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1102233 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1102235 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1102352 INFO (zkCallback-3228-thread-1-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1102352 INFO (zkCallback-3224-thread-1-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1102353 INFO (zkCallback-3224-thread-2-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1102353 INFO (zkCallback-3228-thread-2-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1102353 INFO (zkCallback-3228-thread-3-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1103271 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1103278 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1103278 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1103294 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1103304 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.IndexSchema [testSimple1_shard1_replica_n1] Schema name=minimal
[junit4] 2> 1103305 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.IndexSchema [testSimple1_shard2_replica_n6] Schema name=minimal
[junit4] 2> 1103311 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1103311 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard2_replica_n6' using configuration from collection testSimple1, trusted=true
[junit4] 2> 1103312 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60245.solr.core.testSimple1.shard2.replica_n6' (registry 'solr.core.testSimple1.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d70a389
[junit4] 2> 1103312 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1103312 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrCore [[testSimple1_shard2_replica_n6] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001/node3/testSimple1_shard2_replica_n6], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001/node3/./testSimple1_shard2_replica_n6/data/]
[junit4] 2> 1103312 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.IndexSchema [testSimple1_shard2_replica_n4] Schema name=minimal
[junit4] 2> 1103313 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1103313 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard2_replica_n4' using configuration from collection testSimple1, trusted=true
[junit4] 2> 1103314 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60243.solr.core.testSimple1.shard2.replica_n4' (registry 'solr.core.testSimple1.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d70a389
[junit4] 2> 1103314 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1103314 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SolrCore [[testSimple1_shard2_replica_n4] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001/node2/testSimple1_shard2_replica_n4], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001/node2/./testSimple1_shard2_replica_n4/data/]
[junit4] 2> 1103315 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1103315 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard1_replica_n1' using configuration from collection testSimple1, trusted=true
[junit4] 2> 1103318 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60243.solr.core.testSimple1.shard1.replica_n1' (registry 'solr.core.testSimple1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d70a389
[junit4] 2> 1103319 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1103319 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrCore [[testSimple1_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001/node2/testSimple1_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001/node2/./testSimple1_shard1_replica_n1/data/]
[junit4] 2> 1103323 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.IndexSchema [testSimple1_shard1_replica_n2] Schema name=minimal
[junit4] 2> 1103323 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1103323 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard1_replica_n2' using configuration from collection testSimple1, trusted=true
[junit4] 2> 1103323 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60245.solr.core.testSimple1.shard1.replica_n2' (registry 'solr.core.testSimple1.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d70a389
[junit4] 2> 1103323 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1103323 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrCore [[testSimple1_shard1_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001/node3/testSimple1_shard1_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001/node3/./testSimple1_shard1_replica_n2/data/]
[junit4] 2> 1103410 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1103410 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1103411 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1103411 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1103412 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1103412 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1103434 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1103434 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1103440 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1103440 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1103441 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1103441 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1103442 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1103442 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1103442 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1103443 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1103450 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2d718cac[testSimple1_shard1_replica_n1] main]
[junit4] 2> 1103451 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 1103451 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 1103452 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1103452 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590952870821232640
[junit4] 2> 1103453 INFO (searcherExecutor-4220-thread-1-processing-n:127.0.0.1:60243_solr x:testSimple1_shard1_replica_n1 s:shard1 c:testSimple1 r:core_node3) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrCore [testSimple1_shard1_replica_n1] Registered new searcher Searcher@2d718cac[testSimple1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1103455 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@69864ba8[testSimple1_shard2_replica_n6] main]
[junit4] 2> 1103461 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.SolrIndexSearcher Opening [Searcher@31ce4c0a[testSimple1_shard2_replica_n4] main]
[junit4] 2> 1103462 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 1103462 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 1103463 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 1103464 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1103464 INFO (searcherExecutor-4219-thread-1-processing-n:127.0.0.1:60243_solr x:testSimple1_shard2_replica_n4 s:shard2 c:testSimple1 r:core_node7) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SolrCore [testSimple1_shard2_replica_n4] Registered new searcher Searcher@31ce4c0a[testSimple1_shard2_replica_n4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1103465 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590952870834864128
[junit4] 2> 1103465 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@342451f[testSimple1_shard1_replica_n2] main]
[junit4] 2> 1103467 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 1103467 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 1103468 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1103469 INFO (searcherExecutor-4221-thread-1-processing-n:127.0.0.1:60245_solr x:testSimple1_shard1_replica_n2 s:shard1 c:testSimple1 r:core_node5) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrCore [testSimple1_shard1_replica_n2] Registered new searcher Searcher@342451f[testSimple1_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1103469 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590952870839058432
[junit4] 2> 1103470 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
[junit4] 2> 1103472 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 1103476 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1103477 INFO (searcherExecutor-4218-thread-1-processing-n:127.0.0.1:60245_solr x:testSimple1_shard2_replica_n6 s:shard2 c:testSimple1 r:core_node8) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrCore [testSimple1_shard2_replica_n6] Registered new searcher Searcher@69864ba8[testSimple1_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1103477 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590952870847447040
[junit4] 2> 1103479 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
[junit4] 2> 1103591 INFO (zkCallback-3224-thread-1-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1103591 INFO (zkCallback-3228-thread-3-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1103591 INFO (zkCallback-3228-thread-4-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1103591 INFO (zkCallback-3228-thread-2-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1103591 INFO (zkCallback-3224-thread-2-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1103980 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1103980 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1103980 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1103980 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1103980 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:60243/solr/testSimple1_shard1_replica_n1/
[junit4] 2> 1103980 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:60243/solr/testSimple1_shard2_replica_n4/
[junit4] 2> 1103981 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.PeerSync PeerSync: core=testSimple1_shard2_replica_n4 url=http://127.0.0.1:60243/solr START replicas=[http://127.0.0.1:60245/solr/testSimple1_shard2_replica_n6/] nUpdates=100
[junit4] 2> 1103981 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=testSimple1_shard1_replica_n1 url=http://127.0.0.1:60243/solr START replicas=[http://127.0.0.1:60245/solr/testSimple1_shard1_replica_n2/] nUpdates=100
[junit4] 2> 1103983 INFO (qtp239244624-11548) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.S.Request [testSimple1_shard2_replica_n6] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1103983 INFO (qtp239244624-11538) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.S.Request [testSimple1_shard1_replica_n2] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1104293 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.PeerSync PeerSync: core=testSimple1_shard2_replica_n4 url=http://127.0.0.1:60243/solr DONE. We have no versions. sync failed.
[junit4] 2> 1104293 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 1104293 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 1104293 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1104293 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=testSimple1_shard1_replica_n1 url=http://127.0.0.1:60243/solr DONE. We have no versions. sync failed.
[junit4] 2> 1104294 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 1104294 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 1104294 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1104302 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:60243/solr/testSimple1_shard1_replica_n1/ shard1
[junit4] 2> 1104302 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:60243/solr/testSimple1_shard2_replica_n4/ shard2
[junit4] 2> 1104415 INFO (zkCallback-3228-thread-2-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1104415 INFO (zkCallback-3228-thread-4-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1104415 INFO (zkCallback-3228-thread-3-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1104415 INFO (zkCallback-3224-thread-1-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1104415 INFO (zkCallback-3224-thread-2-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1104474 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1104474 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1104477 INFO (qtp380382596-11531) [n:127.0.0.1:60243_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n4&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2245
[junit4] 2> 1104477 INFO (qtp380382596-11529) [n:127.0.0.1:60243_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n1&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2246
[junit4] 2> 1104484 INFO (qtp239244624-11546) [n:127.0.0.1:60245_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n2&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2250
[junit4] 2> 1104507 INFO (qtp239244624-11528) [n:127.0.0.1:60245_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n6&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2274
[junit4] 2> 1104510 INFO (qtp380382596-11526) [n:127.0.0.1:60243_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 1104618 INFO (zkCallback-3224-thread-2-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1104618 INFO (zkCallback-3228-thread-3-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1104618 INFO (zkCallback-3228-thread-2-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1104618 INFO (zkCallback-3224-thread-1-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1104618 INFO (zkCallback-3228-thread-4-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1105515 INFO (qtp380382596-11526) [n:127.0.0.1:60243_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&collection.configName=conf&maxShardsPerNode=2&autoAddReplicas=true&name=testSimple1&nrtReplicas=2&action=CREATE&numShards=2&createNodeSet=127.0.0.1:60243_solr,127.0.0.1:60245_solr&wt=javabin&version=2} status=0 QTime=3630
[junit4] 2> 1105516 INFO (qtp380382596-11533) [n:127.0.0.1:60243_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&autoAddReplicas=false&name=testSimple2&nrtReplicas=2&action=CREATE&numShards=1&createNodeSet=127.0.0.1:60245_solr,127.0.0.1:60244_solr&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1105519 INFO (OverseerThreadFactory-4211-thread-2-processing-n:127.0.0.1:60243_solr) [n:127.0.0.1:60243_solr ] o.a.s.c.a.c.CreateCollectionCmd Create collection testSimple2
[junit4] 2> 1105520 INFO (OverseerCollectionConfigSetProcessor-72114465450819593-127.0.0.1:60243_solr-n_0000000000) [n:127.0.0.1:60243_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1105736 INFO (OverseerStateUpdate-72114465450819593-127.0.0.1:60243_solr-n_0000000000) [n:127.0.0.1:60243_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testSimple2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"testSimple2_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:60244/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1105741 INFO (OverseerStateUpdate-72114465450819593-127.0.0.1:60243_solr-n_0000000000) [n:127.0.0.1:60243_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testSimple2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"testSimple2_shard1_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:60245/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1105961 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=testSimple2_shard1_replica_n2&action=CREATE&numShards=1&collection=testSimple2&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1105964 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testSimple2_shard1_replica_n1&action=CREATE&numShards=1&collection=testSimple2&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1105965 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1106081 INFO (zkCallback-3224-thread-1-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1106081 INFO (zkCallback-3224-thread-2-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1106081 INFO (zkCallback-3226-thread-1-processing-n:127.0.0.1:60244_solr) [n:127.0.0.1:60244_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1106082 INFO (zkCallback-3226-thread-2-processing-n:127.0.0.1:60244_solr) [n:127.0.0.1:60244_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1106997 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1106997 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1107023 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.s.IndexSchema [testSimple2_shard1_replica_n1] Schema name=minimal
[junit4] 2> 1107023 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.s.IndexSchema [testSimple2_shard1_replica_n2] Schema name=minimal
[junit4] 2> 1107028 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1107028 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1107028 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testSimple2_shard1_replica_n2' using configuration from collection testSimple2, trusted=true
[junit4] 2> 1107028 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testSimple2_shard1_replica_n1' using configuration from collection testSimple2, trusted=true
[junit4] 2> 1107029 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60244.solr.core.testSimple2.shard1.replica_n1' (registry 'solr.core.testSimple2.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d70a389
[junit4] 2> 1107029 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_60245.solr.core.testSimple2.shard1.replica_n2' (registry 'solr.core.testSimple2.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@5d70a389
[junit4] 2> 1107029 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1107029 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1107029 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.SolrCore [[testSimple2_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001/node1/testSimple2_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001/node1/./testSimple2_shard1_replica_n1/data/]
[junit4] 2> 1107029 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.c.SolrCore [[testSimple2_shard1_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001/node3/testSimple2_shard1_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_4A8EF17A1C4BEB94-001/tempDir-001/node3/./testSimple2_shard1_replica_n2/data/]
[junit4] 2> 1107130 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1107130 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1107131 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1107131 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1107136 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1107136 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1107138 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1107138 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1107140 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@27d873f[testSimple2_shard1_replica_n1] main]
[junit4] 2> 1107143 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 1107144 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 1107144 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@32568dfe[testSimple2_shard1_replica_n2] main]
[junit4] 2> 1107145 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1107145 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590952874693623808
[junit4] 2> 1107146 INFO (searcherExecutor-4239-thread-1-processing-n:127.0.0.1:60244_solr x:testSimple2_shard1_replica_n1 s:shard1 c:testSimple2 r:core_node3) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.SolrCore [testSimple2_shard1_replica_n1] Registered new searcher Searcher@27d873f[testSimple2_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1107147 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 1107147 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 1107148 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1107148 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590952874696769536
[junit4] 2> 1107149 INFO (searcherExecutor-4238-thread-1-processing-n:127.0.0.1:60245_solr x:testSimple2_shard1_replica_n2 s:shard1 c:testSimple2 r:core_node4) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.c.SolrCore [testSimple2_shard1_replica_n2] Registered new searcher Searcher@32568dfe[testSimple2_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1107156 INFO (qtp239244624-11549) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.c.ZkShardTerms Failed to save terms, version is not match, retrying
[junit4] 2> 1107165 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1107165 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1107165 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:60244/solr/testSimple2_shard1_replica_n1/
[junit4] 2> 1107165 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=testSimple2_shard1_replica_n1 url=http://127.0.0.1:60244/solr START replicas=[http://127.0.0.1:60245/solr/testSimple2_shard1_replica_n2/] nUpdates=100
[junit4] 2> 1107168 INFO (qtp239244624-11538) [n:127.0.0.1:60245_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.c.S.Request [testSimple2_shard1_replica_n2] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1107270 INFO (zkCallback-3226-thread-2-processing-n:127.0.0.1:60244_solr) [n:127.0.0.1:60244_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1107270 INFO (zkCallback-3226-thread-1-processing-n:127.0.0.1:60244_solr) [n:127.0.0.1:60244_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1107270 INFO (zkCallback-3224-thread-2-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1107270 INFO (zkCallback-3224-thread-1-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1107476 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=testSimple2_shard1_replica_n1 url=http://127.0.0.1:60244/solr DONE. We have no versions. sync failed.
[junit4] 2> 1107476 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 1107476 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 1107476 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1107482 INFO (qtp1759658738-11524) [n:127.0.0.1:60244_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:60244/solr/testSimple2_shard1_replica_n1/ shard1
[junit4] 2> 1107588 INFO (zkCallback-3224-thread-1-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1107588 INFO (zkCallback-3224-thread-2-processing-n:127.0.0.1:60245_solr) [n:127.0.0.1:60245_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1107589 INFO (zkCallback-3226-thread-1-processing-n:127.0.0.1:60244_solr) [n:127.0.0.1:60244_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple
[...truncated too long message...]
0
[junit4] 2> 97145 INFO (TEST-SchemaTest.testGetDynamicFieldAccuracy-seed#[A2432F7B5D8124F8]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@14fb5951{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 97145 INFO (TEST-SchemaTest.testGetDynamicFieldAccuracy-seed#[A2432F7B5D8124F8]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=119093631
[junit4] 2> 97145 INFO (TEST-SchemaTest.testGetDynamicFieldAccuracy-seed#[A2432F7B5D8124F8]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 97145 INFO (TEST-SchemaTest.testGetDynamicFieldAccuracy-seed#[A2432F7B5D8124F8]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1f005c3b: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@1cc57ea6
[junit4] 2> 97150 INFO (TEST-SchemaTest.testGetDynamicFieldAccuracy-seed#[A2432F7B5D8124F8]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 97150 INFO (TEST-SchemaTest.testGetDynamicFieldAccuracy-seed#[A2432F7B5D8124F8]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@eb692d5: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@7d34c32f
[junit4] 2> 97153 INFO (TEST-SchemaTest.testGetDynamicFieldAccuracy-seed#[A2432F7B5D8124F8]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 97153 INFO (TEST-SchemaTest.testGetDynamicFieldAccuracy-seed#[A2432F7B5D8124F8]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@70ec5012: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@1d922bc3
[junit4] 2> 97154 INFO (coreCloseExecutor-633-thread-1) [ x:collection1] o.a.s.c.SolrCore [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@4ebb5463
[junit4] 2> 97154 INFO (coreCloseExecutor-633-thread-1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=1320899683
[junit4] 2> 97154 INFO (coreCloseExecutor-633-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@42f62c04: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@59c489dd
[junit4] 2> 97160 INFO (TEST-SchemaTest.testGetDynamicFieldAccuracy-seed#[A2432F7B5D8124F8]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7be39671{/solr,null,UNAVAILABLE}
[junit4] 2> 97160 INFO (TEST-SchemaTest.testGetDynamicFieldAccuracy-seed#[A2432F7B5D8124F8]) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 97161 INFO (TEST-SchemaTest.testGetDynamicFieldAccuracy-seed#[A2432F7B5D8124F8]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testGetDynamicFieldAccuracy
[junit4] 2> 97162 INFO (SUITE-SchemaTest-seed#[A2432F7B5D8124F8]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
[junit4] 2> Jan 29, 2018 7:15:28 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> Jan 29, 2018 7:15:48 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.SchemaTest:
[junit4] 2> 1) Thread[id=1276, name=qtp1023729175-1276, state=TIMED_WAITING, group=TGRP-SchemaTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
[junit4] 2> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] 2> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> Jan 29, 2018 7:15:48 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=1276, name=qtp1023729175-1276, state=TIMED_WAITING, group=TGRP-SchemaTest]
[junit4] 2> Jan 29, 2018 7:15:51 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> SEVERE: There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=1276, name=qtp1023729175-1276, state=TIMED_WAITING, group=TGRP-SchemaTest]
[junit4] 2> at sun.misc.Unsafe.park(Native Method)
[junit4] 2> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
[junit4] 2> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] 2> at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] 2> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> NOTE: test params are: codec=Lucene70, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@15db8fe), locale=cs, timezone=Asia/Kabul
[junit4] 2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=3,threads=2,free=62946056,total=136966144
[junit4] 2> NOTE: All tests run in this JVM: [TestToleratedUpdateError, EmpiricalDistributionEvaluatorTest, GetByIdTest, LessThanEvaluatorTest, TestCollectionStateWatchers, ExclusiveOrEvaluatorTest, TestDelegationTokenResponse, UuidEvaluatorTest, FieldAnalysisResponseTest, TestDelegationTokenRequest, SolrExampleStreamingTest, ArrayEvaluatorTest, StreamExpressionToExpessionTest, RecursiveEvaluatorTest, TestDocumentObjectBinder, NormalDistributionEvaluatorTest, ModifiableSolrParamsTest, TestJsonRecordReader, NoOpResponseParserTest, NamedListTest, TermsResponseTest, JettyWebappTest, SineEvaluatorTest, GreaterThanEvaluatorTest, SchemaTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SchemaTest -Dtests.seed=A2432F7B5D8124F8 -Dtests.slow=true -Dtests.locale=cs -Dtests.timezone=Asia/Kabul -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J0 | SchemaTest (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.SchemaTest:
[junit4] > 1) Thread[id=1276, name=qtp1023729175-1276, state=TIMED_WAITING, group=TGRP-SchemaTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([A2432F7B5D8124F8]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=1276, name=qtp1023729175-1276, state=TIMED_WAITING, group=TGRP-SchemaTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] > at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([A2432F7B5D8124F8]:0)
[junit4] Completed [53/146 (1!)] on J0 in 64.28s, 35 tests, 2 errors <<< FAILURES!
[...truncated 47506 lines...]
[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4413
- Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4413/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC
3 tests failed.
FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([1B0489124781423E:A108BE9D18699471]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNull(Assert.java:551)
at org.junit.Assert.assertNull(Assert.java:562)
at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger(TriggerIntegrationTest.java:1575)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testListeners
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([1B0489124781423E:F6C41E02FAEDDFE7]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testListeners(TriggerIntegrationTest.java:1053)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory
Error Message:
expected:<5> but was:<0>
Stack Trace:
java.lang.AssertionError: expected:<5> but was:<0>
at __randomizedtesting.SeedInfo.seed([1B0489124781423E:76F82DEFFDC9BD39]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.testHistory(AutoscalingHistoryHandlerTest.java:244)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 13586 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
[junit4] 2> 3234819 INFO (SUITE-TriggerIntegrationTest-seed#[1B0489124781423E]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_1B0489124781423E-001/init-core-data-001
[junit4] 2> 3234820 INFO (SUITE-TriggerIntegrationTest-seed#[1B0489124781423E]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 3234829 INFO (SUITE-TriggerIntegrationTest-seed#[1B0489124781423E]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
[junit4] 2> 3234830 INFO (SUITE-TriggerIntegrationTest-seed#[1B0489124781423E]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_1B0489124781423E-001/tempDir-001
[junit4] 2> 3234830 INFO (SUITE-TriggerIntegrationTest-seed#[1B0489124781423E]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 3234832 INFO (Thread-5942) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 3234832 INFO (Thread-5942) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 3234836 ERROR (Thread-5942) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 3234939 INFO (SUITE-TriggerIntegrationTest-seed#[1B0489124781423E]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:59236
[junit4] 2> 3234953 INFO (zkConnectionManagerCallback-8200-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3234959 INFO (jetty-launcher-8197-thread-2) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 3234959 INFO (jetty-launcher-8197-thread-1) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 3234960 INFO (jetty-launcher-8197-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 3234960 INFO (jetty-launcher-8197-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 3234961 INFO (jetty-launcher-8197-thread-2) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 3234961 INFO (jetty-launcher-8197-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@189b9f6a{/solr,null,AVAILABLE}
[junit4] 2> 3234961 INFO (jetty-launcher-8197-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 3234961 INFO (jetty-launcher-8197-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 3234962 INFO (jetty-launcher-8197-thread-1) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 3234962 INFO (jetty-launcher-8197-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@11d07fc8{HTTP/1.1,[http/1.1]}{127.0.0.1:59238}
[junit4] 2> 3234962 INFO (jetty-launcher-8197-thread-2) [ ] o.e.j.s.Server Started @3239487ms
[junit4] 2> 3234962 INFO (jetty-launcher-8197-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59238}
[junit4] 2> 3234962 INFO (jetty-launcher-8197-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@41bd8509{/solr,null,AVAILABLE}
[junit4] 2> 3234962 ERROR (jetty-launcher-8197-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3234963 INFO (jetty-launcher-8197-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4e7ff41b{HTTP/1.1,[http/1.1]}{127.0.0.1:59239}
[junit4] 2> 3234963 INFO (jetty-launcher-8197-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 3234963 INFO (jetty-launcher-8197-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3234963 INFO (jetty-launcher-8197-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3234963 INFO (jetty-launcher-8197-thread-1) [ ] o.e.j.s.Server Started @3239488ms
[junit4] 2> 3234963 INFO (jetty-launcher-8197-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-28T10:25:13.442Z
[junit4] 2> 3234963 INFO (jetty-launcher-8197-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59239}
[junit4] 2> 3234964 ERROR (jetty-launcher-8197-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3234964 INFO (jetty-launcher-8197-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 3234964 INFO (jetty-launcher-8197-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3234964 INFO (jetty-launcher-8197-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3234964 INFO (jetty-launcher-8197-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-28T10:25:13.443Z
[junit4] 2> 3234966 INFO (zkConnectionManagerCallback-8202-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3234966 INFO (zkConnectionManagerCallback-8204-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3234967 INFO (jetty-launcher-8197-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3234967 INFO (jetty-launcher-8197-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3234988 INFO (jetty-launcher-8197-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59236/solr
[junit4] 2> 3234988 INFO (jetty-launcher-8197-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59236/solr
[junit4] 2> 3234991 INFO (zkConnectionManagerCallback-8210-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3234991 INFO (zkConnectionManagerCallback-8212-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3234995 INFO (zkConnectionManagerCallback-8214-thread-1-processing-n:127.0.0.1:59238_solr) [n:127.0.0.1:59238_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3234995 INFO (zkConnectionManagerCallback-8216-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3235171 INFO (jetty-launcher-8197-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3235171 INFO (jetty-launcher-8197-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3235172 INFO (jetty-launcher-8197-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59238_solr
[junit4] 2> 3235173 INFO (jetty-launcher-8197-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59239_solr
[junit4] 2> 3235174 INFO (jetty-launcher-8197-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.Overseer Overseer (id=99426966786146309-127.0.0.1:59238_solr-n_0000000000) starting
[junit4] 2> 3235175 INFO (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 3235175 INFO (zkCallback-8213-thread-1-processing-n:127.0.0.1:59238_solr) [n:127.0.0.1:59238_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 3235191 DEBUG (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 3235194 DEBUG (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 3235195 DEBUG (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:59239_solr]
[junit4] 2> 3235195 DEBUG (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
[junit4] 2> 3235195 DEBUG (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 3235196 INFO (jetty-launcher-8197-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59238_solr
[junit4] 2> 3235205 INFO (zkCallback-8213-thread-1-processing-n:127.0.0.1:59238_solr) [n:127.0.0.1:59238_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 3235206 INFO (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 3235251 DEBUG (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3235252 DEBUG (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 3235252 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3235328 INFO (jetty-launcher-8197-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59239.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11cdc99c
[junit4] 2> 3235333 INFO (jetty-launcher-8197-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59238.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11cdc99c
[junit4] 2> 3235340 INFO (jetty-launcher-8197-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59239.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11cdc99c
[junit4] 2> 3235341 INFO (jetty-launcher-8197-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59239.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11cdc99c
[junit4] 2> 3235342 INFO (jetty-launcher-8197-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_1B0489124781423E-001/tempDir-001/node1/.
[junit4] 2> 3235344 INFO (jetty-launcher-8197-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59238.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11cdc99c
[junit4] 2> 3235344 INFO (jetty-launcher-8197-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59238.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11cdc99c
[junit4] 2> 3235345 INFO (jetty-launcher-8197-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_1B0489124781423E-001/tempDir-001/node2/.
[junit4] 2> 3235408 INFO (zkConnectionManagerCallback-8222-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3235413 INFO (zkConnectionManagerCallback-8226-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3235415 INFO (SUITE-TriggerIntegrationTest-seed#[1B0489124781423E]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 3235417 INFO (SUITE-TriggerIntegrationTest-seed#[1B0489124781423E]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:59236/solr ready
[junit4] 2> 3235449 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testEventFromRestoredState
[junit4] 2> 3235450 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 2
[junit4] 2> 3235452 DEBUG (zkCallback-8213-thread-1-processing-n:127.0.0.1:59238_solr) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 3235452 DEBUG (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 3235459 DEBUG (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 3235460 DEBUG (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3235461 DEBUG (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 3235474 INFO (qtp1549171901-32447) [n:127.0.0.1:59238_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3235477 INFO (qtp1520736820-32446) [n:127.0.0.1:59239_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3235478 DEBUG (qtp1549171901-32442) [n:127.0.0.1:59238_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3235479 INFO (qtp1549171901-32442) [n:127.0.0.1:59238_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=7
[junit4] 2> 3235480 DEBUG (zkCallback-8213-thread-1-processing-n:127.0.0.1:59238_solr) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 3235481 DEBUG (zkCallback-8213-thread-1-processing-n:127.0.0.1:59238_solr) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:59238_solr, 127.0.0.1:59239_solr]
[junit4] 2> 3235482 DEBUG (zkCallback-8213-thread-1-processing-n:127.0.0.1:59238_solr) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerEFRS instantiated with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
[junit4] 2> 3235482 DEBUG (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 3235485 INFO (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
[junit4] 2> 3235489 DEBUG (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 3235489 INFO (qtp1520736820-32446) [n:127.0.0.1:59239_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 3235490 DEBUG (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 3235491 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3235491 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 3235510 INFO (qtp1520736820-32446) [n:127.0.0.1:59239_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=21
[junit4] 2> 3235512 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 3235513 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 3235513 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 3235513 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 3235514 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3d0734f9{/solr,null,AVAILABLE}
[junit4] 2> 3235514 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@436305a6{HTTP/1.1,[http/1.1]}{127.0.0.1:59252}
[junit4] 2> 3235514 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.Server Started @3240039ms
[junit4] 2> 3235514 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59252}
[junit4] 2> 3235515 ERROR (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3235515 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 3235515 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3235515 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3235515 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-28T10:25:13.994Z
[junit4] 2> 3235517 INFO (zkConnectionManagerCallback-8228-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3235518 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3235531 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59236/solr
[junit4] 2> 3235533 INFO (zkConnectionManagerCallback-8232-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3235537 INFO (zkConnectionManagerCallback-8234-thread-1-processing-n:127.0.0.1:59252_solr) [n:127.0.0.1:59252_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3235550 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [n:127.0.0.1:59252_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 3235554 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [n:127.0.0.1:59252_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3235562 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [n:127.0.0.1:59252_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59252_solr
[junit4] 2> 3235573 INFO (zkCallback-8213-thread-1-processing-n:127.0.0.1:59238_solr) [n:127.0.0.1:59238_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 3235573 INFO (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 3235573 INFO (zkCallback-8225-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 3235574 INFO (zkCallback-8233-thread-1-processing-n:127.0.0.1:59252_solr) [n:127.0.0.1:59252_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 3235682 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [n:127.0.0.1:59252_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59252.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11cdc99c
[junit4] 2> 3235693 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [n:127.0.0.1:59252_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59252.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11cdc99c
[junit4] 2> 3235693 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [n:127.0.0.1:59252_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59252.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11cdc99c
[junit4] 2> 3235695 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [n:127.0.0.1:59252_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_1B0489124781423E-001/tempDir-001/node3/.
[junit4] 2> 3236506 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3236506 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3236506 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:59252_solr at time 835633993299927
[junit4] 2> 3237513 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3237513 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3237514 INFO (OverseerCollectionConfigSetProcessor-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 3238514 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3238514 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3239517 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3239517 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3240529 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3240529 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3241535 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3241535 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3242545 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3242545 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3243549 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3243549 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3244559 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3244559 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3245568 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3245568 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3246578 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3246579 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3246579 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerEFRS firing registered processor for nodes: [127.0.0.1:59252_solr] added at times [835633993299927], now=835644066160565
[junit4] 2> 3246579 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"2f801335537d7Teuf11mqxb6wwxbqcn83a7kymx",
[junit4] 2> "source":"node_added_triggerEFRS",
[junit4] 2> "eventTime":835633993299927,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[835633993299927],
[junit4] 2> "nodeNames":["127.0.0.1:59252_solr"]}}
[junit4] 2> 3246579 DEBUG (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_triggerEFRS]
[junit4] 2> 3246580 INFO (ScheduledTrigger-13016-thread-2) [n:127.0.0.1:59238_solr ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
[junit4] 2> 3246581 DEBUG (AutoscalingActionExecutor-13017-thread-1-processing-n:127.0.0.1:59238_solr) [n:127.0.0.1:59238_solr ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"2f801335537d7Teuf11mqxb6wwxbqcn83a7kymx",
[junit4] 2> "source":"node_added_triggerEFRS",
[junit4] 2> "eventTime":835633993299927,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[835633993299927],
[junit4] 2> "_enqueue_time_":835644066878696,
[junit4] 2> "nodeNames":["127.0.0.1:59252_solr"]}}
[junit4] 2> 3246582 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 3246582 DEBUG (AutoscalingActionExecutor-13017-thread-1-processing-n:127.0.0.1:59238_solr) [n:127.0.0.1:59238_solr ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_triggerEFRS after 5000ms
[junit4] 2> 3246583 DEBUG (AutoscalingActionExecutor-13017-thread-1-processing-n:127.0.0.1:59238_solr) [n:127.0.0.1:59238_solr ] o.a.s.c.a.ScheduledTriggers -- processing took 2 ms for event id=2f801335537d7Teuf11mqxb6wwxbqcn83a7kymx
[junit4] 2> 3246583 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 3246583 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 3246583 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 3246584 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@683c7214{/solr,null,AVAILABLE}
[junit4] 2> 3246584 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@27242948{HTTP/1.1,[http/1.1]}{127.0.0.1:59273}
[junit4] 2> 3246584 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.Server Started @3251109ms
[junit4] 2> 3246584 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59273}
[junit4] 2> 3246584 ERROR (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3246585 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 3246585 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3246585 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3246585 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-28T10:25:25.064Z
[junit4] 2> 3246587 INFO (zkConnectionManagerCallback-8238-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3246588 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3246598 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59236/solr
[junit4] 2> 3246599 INFO (zkConnectionManagerCallback-8242-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3246602 INFO (zkConnectionManagerCallback-8244-thread-1-processing-n:127.0.0.1:59273_solr) [n:127.0.0.1:59273_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3246615 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [n:127.0.0.1:59273_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 3246619 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [n:127.0.0.1:59273_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3246622 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [n:127.0.0.1:59273_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59273_solr
[junit4] 2> 3246624 INFO (zkCallback-8213-thread-1-processing-n:127.0.0.1:59238_solr) [n:127.0.0.1:59238_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3246625 INFO (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3246624 INFO (zkCallback-8225-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3246625 INFO (zkCallback-8233-thread-1-processing-n:127.0.0.1:59252_solr) [n:127.0.0.1:59252_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3246625 INFO (zkCallback-8243-thread-1-processing-n:127.0.0.1:59273_solr) [n:127.0.0.1:59273_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3246691 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [n:127.0.0.1:59273_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59273.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11cdc99c
[junit4] 2> 3246699 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [n:127.0.0.1:59273_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59273.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11cdc99c
[junit4] 2> 3246699 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [n:127.0.0.1:59273_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59273.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11cdc99c
[junit4] 2> 3246700 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [n:127.0.0.1:59273_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_1B0489124781423E-001/tempDir-001/node4/.
[junit4] 2> 3251591 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3251591 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 3251591 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:59273_solr at time 835649078500141
[junit4] 2> 3252593 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3252593 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 3253597 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3253598 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 3254598 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3254598 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 3255601 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3255602 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 3256602 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3256602 DEBUG (ScheduledTrigger-13016-thread-1) [n:127.0.0.1:59238_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 3256748 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@11d07fc8{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 3256748 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=386771509
[junit4] 2> 3256749 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 3256749 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@32239926: rootName = solr_59238, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@13d71d0a
[junit4] 2> 3256756 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 3256756 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@43787dcb: rootName = solr_59238, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@7a5a0d69
[junit4] 2> 3256761 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 3256761 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@39c07dbc: rootName = solr_59238, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@2237aeea
[junit4] 2> 3256763 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:59238_solr
[junit4] 2> 3256764 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 3256765 INFO (zkCallback-8225-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 3256765 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 3256765 INFO (zkCallback-8233-thread-1-processing-n:127.0.0.1:59252_solr) [n:127.0.0.1:59252_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 3256765 INFO (zkCallback-8213-thread-1-processing-n:127.0.0.1:59238_solr) [n:127.0.0.1:59238_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 3256765 INFO (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 3256765 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.c.Overseer Overseer (id=99426966786146309-127.0.0.1:59238_solr-n_0000000000) closing
[junit4] 2> 3256765 INFO (zkCallback-8243-thread-1-processing-n:127.0.0.1:59273_solr) [n:127.0.0.1:59273_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 3256766 INFO (OverseerStateUpdate-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:59238_solr
[junit4] 2> 3256767 DEBUG (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 3256767 WARN (OverseerAutoScalingTriggerThread-99426966786146309-127.0.0.1:59238_solr-n_0000000000) [n:127.0.0.1:59238_solr ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 3256768 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@189b9f6a{/solr,null,UNAVAILABLE}
[junit4] 2> 3256768 INFO (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59239_solr
[junit4] 2> 3256769 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 3256771 INFO (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.Overseer Overseer (id=99426966786146310-127.0.0.1:59239_solr-n_0000000001) starting
[junit4] 2> 3256780 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 3256781 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 3256782 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:59252_solr, 127.0.0.1:59273_solr, 127.0.0.1:59239_solr]
[junit4] 2> 3256782 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerEFRS instantiated with properties: {event=nodeAdded, waitFor=10, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
[junit4] 2> 3256783 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:59252_solr, 127.0.0.1:59273_solr, 127.0.0.1:59239_solr]
[junit4] 2> 3256783 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion -1
[junit4] 2> 3256783 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 3256783 INFO (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
[junit4] 2> 3256784 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Adding node from marker path: 127.0.0.1:59273_solr
[junit4] 2> 3256787 DEBUG (ScheduledTrigger-13037-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3256787 DEBUG (ScheduledTrigger-13037-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3256789 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 3256789 DEBUG (ScheduledTrigger-13037-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3257794 DEBUG (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3257794 DEBUG (ScheduledTrigger-13037-thread-2) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3257795 DEBUG (ScheduledTrigger-13037-thread-2) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3258799 DEBUG (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3258799 DEBUG (ScheduledTrigger-13037-thread-4) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3258799 DEBUG (ScheduledTrigger-13037-thread-4) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3259801 DEBUG (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3259801 DEBUG (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3259802 DEBUG (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3260802 DEBUG (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3260802 DEBUG (ScheduledTrigger-13037-thread-4) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3260802 DEBUG (ScheduledTrigger-13037-thread-4) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3261810 DEBUG (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3261810 DEBUG (ScheduledTrigger-13037-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerEFRS
[junit4] 2> 3261810 DEBUG (ScheduledTrigger-13037-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3261811 DEBUG (ScheduledTrigger-13037-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerEFRS firing registered processor for nodes: [127.0.0.1:59273_solr] added at times [835649078500141], now=835659298106655
[junit4] 2> 3261811 DEBUG (ScheduledTrigger-13037-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"2f804b67b1b2dTeuf11mqxb6wwxbqcn83a7kymz",
[junit4] 2> "source":"node_added_triggerEFRS",
[junit4] 2> "eventTime":835649078500141,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[835649078500141],
[junit4] 2> "nodeNames":["127.0.0.1:59273_solr"]}}
[junit4] 2> 3261811 DEBUG (ScheduledTrigger-13037-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_triggerEFRS]
[junit4] 2> 3261812 INFO (ScheduledTrigger-13037-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
[junit4] 2> 3261812 DEBUG (AutoscalingActionExecutor-13038-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"2f804b67b1b2dTeuf11mqxb6wwxbqcn83a7kymz",
[junit4] 2> "source":"node_added_triggerEFRS",
[junit4] 2> "eventTime":835649078500141,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[835649078500141],
[junit4] 2> "_enqueue_time_":835659298746882,
[junit4] 2> "nodeNames":["127.0.0.1:59273_solr"]}}
[junit4] 2> 3261813 INFO (TEST-TriggerIntegrationTest.testEventFromRestoredState-seed#[1B0489124781423E]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testEventFromRestoredState
[junit4] 2> 3261816 DEBUG (AutoscalingActionExecutor-13038-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 5000ms
[junit4] 2> 3261817 DEBUG (AutoscalingActionExecutor-13038-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_triggerEFRS after 5000ms
[junit4] 2> 3261817 DEBUG (AutoscalingActionExecutor-13038-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers -- processing took 5 ms for event id=2f804b67b1b2dTeuf11mqxb6wwxbqcn83a7kymz
[junit4] 2> 3261836 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testTriggerThrottling
[junit4] 2> 3261837 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 5
[junit4] 2> 3261838 DEBUG (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
[junit4] 2> 3261838 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 3261840 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 3261841 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3261841 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
[junit4] 2> 3261845 INFO (qtp1520736820-32446) [n:127.0.0.1:59239_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3261847 INFO (qtp96284661-32511) [n:127.0.0.1:59252_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3261848 INFO (qtp180081247-32533) [n:127.0.0.1:59273_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3261848 DEBUG (qtp96284661-32507) [n:127.0.0.1:59252_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3261849 INFO (qtp96284661-32507) [n:127.0.0.1:59252_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=5
[junit4] 2> 3261850 DEBUG (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
[junit4] 2> 3261851 DEBUG (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:59252_solr, 127.0.0.1:59273_solr, 127.0.0.1:59239_solr]
[junit4] 2> 3261851 DEBUG (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 3261851 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
[junit4] 2> 3261853 INFO (qtp1520736820-32455) [n:127.0.0.1:59239_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3261853 INFO (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
[junit4] 2> 3261854 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 3261854 INFO (qtp180081247-32537) [n:127.0.0.1:59273_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3261854 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
[junit4] 2> 3261855 DEBUG (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 3261855 DEBUG (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3261855 INFO (qtp96284661-32511) [n:127.0.0.1:59252_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3261856 DEBUG (qtp1520736820-32451) [n:127.0.0.1:59239_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3261856 INFO (qtp1520736820-32451) [n:127.0.0.1:59239_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=5
[junit4] 2> 3261857 DEBUG (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
[junit4] 2> 3261857 DEBUG (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:59252_solr, 127.0.0.1:59273_solr, 127.0.0.1:59239_solr]
[junit4] 2> 3261857 DEBUG (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 3261858 DEBUG (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:59252_solr, 127.0.0.1:59273_solr, 127.0.0.1:59239_solr]
[junit4] 2> 3261858 DEBUG (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 3261858 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
[junit4] 2> 3261860 INFO (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
[junit4] 2> 3261860 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T16:27:37-05:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 3261861 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 3261861 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
[junit4] 2> 3261861 DEBUG (ScheduledTrigger-13037-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 3261861 DEBUG (ScheduledTrigger-13037-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3261862 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 3261862 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 3261862 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 3261862 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@297a885d{/solr,null,AVAILABLE}
[junit4] 2> 3261863 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@61f82b20{HTTP/1.1,[http/1.1]}{127.0.0.1:59325}
[junit4] 2> 3261863 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.e.j.s.Server Started @3266388ms
[junit4] 2> 3261863 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=59325}
[junit4] 2> 3261863 ERROR (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3261863 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 3261863 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3261863 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3261863 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-28T10:25:40.342Z
[junit4] 2> 3261865 INFO (zkConnectionManagerCallback-8248-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3261866 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3261874 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59236/solr
[junit4] 2> 3261875 INFO (zkConnectionManagerCallback-8252-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3261878 INFO (zkConnectionManagerCallback-8254-thread-1-processing-n:127.0.0.1:59325_solr) [n:127.0.0.1:59325_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3261891 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [n:127.0.0.1:59325_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 3261895 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [n:127.0.0.1:59325_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3261898 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [n:127.0.0.1:59325_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59325_solr
[junit4] 2> 3261900 INFO (zkCallback-8225-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3261901 INFO (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3261901 INFO (zkCallback-8243-thread-1-processing-n:127.0.0.1:59273_solr) [n:127.0.0.1:59273_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3261901 INFO (zkCallback-8233-thread-1-processing-n:127.0.0.1:59252_solr) [n:127.0.0.1:59252_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3261901 INFO (zkCallback-8253-thread-1-processing-n:127.0.0.1:59325_solr) [n:127.0.0.1:59325_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3261977 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [n:127.0.0.1:59325_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59325.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11cdc99c
[junit4] 2> 3261985 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [n:127.0.0.1:59325_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59325.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11cdc99c
[junit4] 2> 3261985 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [n:127.0.0.1:59325_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_59325.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@11cdc99c
[junit4] 2> 3261986 INFO (TEST-TriggerIntegrationTest.testTriggerThrottling-seed#[1B0489124781423E]) [n:127.0.0.1:59325_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_1B0489124781423E-001/tempDir-001/node5/.
[junit4] 2> 3262860 DEBUG (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 3262860 DEBUG (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 3262860 DEBUG (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:59325_solr at time 835660347909390
[junit4] 2> 3262861 DEBUG (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 firing registered processor for nodes: [127.0.0.1:59325_solr] added at times [835660347909390], now=835660347996183
[junit4] 2> 3262861 DEBUG (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"2f8075630a50eTeuf11mqxb6wwxbqcn83a7kyn1",
[junit4] 2> "source":"node_added_trigger1",
[junit4] 2> "eventTime":835660347909390,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[835660347909390],
[junit4] 2> "nodeNames":["127.0.0.1:59325_solr"]}}
[junit4] 2> 3262861 DEBUG (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_trigger2, node_added_trigger1]
[junit4] 2> 3262862 INFO (ScheduledTrigger-13037-thread-3) [n:127.0.0.1:59239_solr ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
[junit4] 2> 3262862 DEBUG (AutoscalingActionExecutor-13038-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"2f8075630a50eTeuf11mqxb6wwxbqcn83a7kyn1",
[junit4] 2> "source":"node_added_trigger1",
[junit4] 2> "eventTime":835660347909390,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[835660347909390],
[junit4] 2> "_enqueue_time_":835660348289082,
[junit4] 2> "nodeNames":["127.0.0.1:59325_solr"]}}
[junit4] 2> 3262862 INFO (AutoscalingActionExecutor-13038-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.TriggerIntegrationTest action executed from node_added_trigger1
[junit4] 2> 3262864 DEBUG (AutoscalingActionExecutor-13038-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger2 after 5000ms
[junit4] 2> 3262864 DEBUG (AutoscalingActionExecutor-13038-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 5000ms
[junit4] 2> 3262864 DEBUG (AutoscalingActionExecutor-13038-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers -- processing took 2 ms for event id=2f8075630a50eTeuf11mqxb6wwxbqcn83a7kyn1
[junit4] 2> 3267865 DEBUG (ScheduledTrigger-13037-thread-2) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger2
[junit4] 2> 3267865 DEBUG (ScheduledTrigger-13037-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger1
[junit4] 2> 3267865 DEBUG (ScheduledTrigger-13037-thread-1) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 3267865 DEBUG (ScheduledTrigger-13037-thread-2) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 3267866 DEBUG (ScheduledTrigger-13037-thread-2) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:59325_solr at time 835665353108974
[junit4] 2> 3267866 DEBUG (ScheduledTrigger-13037-thread-2) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 firing registered processor for nodes: [127.0.0.1:59325_solr] added at times [835665353108974], now=835665353245358
[junit4] 2> 3267866 DEBUG (ScheduledTrigger-13037-thread-2) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"2f8088085edeeTeuf11mqxb6wwxbqcn83a7kyn3",
[junit4] 2> "source":"node_added_trigger2",
[junit4] 2> "eventTime":835665353108974,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[835665353108974],
[junit4] 2> "nodeNames":["127.0.0.1:59325_solr"]}}
[junit4] 2> 3267866 DEBUG (ScheduledTrigger-13037-thread-2) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_trigger2, node_added_trigger1]
[junit4] 2> 3267867 INFO (ScheduledTrigger-13037-thread-2) [n:127.0.0.1:59239_solr ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
[junit4] 2> 3267867 DEBUG (AutoscalingActionExecutor-13038-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"2f8088085edeeTeuf11mqxb6wwxbqcn83a7kyn3",
[junit4] 2> "source":"node_added_trigger2",
[junit4] 2> "eventTime":835665353108974,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[835665353108974],
[junit4] 2> "_enqueue_time_":835665353479388,
[junit4] 2> "nodeNames":["127.0.0.1:59325_solr"]}}
[junit4] 2> 3267867 INFO (AutoscalingActionExecutor-13038-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.TriggerIntegrationTest last action at 1517135141341000000 current time = 1517135146346000000
[junit4] 2> real diff: 5005000000
[junit4] 2> min diff: 4998000000
[junit4] 2> 3267867 INFO (AutoscalingActionExecutor-13038-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.TriggerIntegrationTest action executed from node_added_trigger2
[junit4] 2> 3267869 DEBUG (AutoscalingActionExecutor-13038-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger2 after 5000ms
[junit4] 2> 3267869 DEBUG (AutoscalingActionExecutor-13038-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger1 after 5000ms
[junit4] 2> 3267869 DEBUG (AutoscalingActionExecutor-13038-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.ScheduledTriggers -- processing took 2 ms for event id=2f8088085edeeTeuf11mqxb6wwxbqcn83a7kyn3
[junit4] 2> 3267870 INFO (qtp1520736820-32451) [n:127.0.0.1:59239_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3267871 INFO (qtp96284661-32511) [n:127.0.0.1:59252_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3267873 INFO (qtp180081247-32538) [n:127.0.0.1:59273_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3267875 INFO (qtp530267091-32569) [n:127.0.0.1:59325_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3267876 DEBUG (qtp180081247-32537) [n:127.0.0.1:59273_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3267876 INFO (qtp180081247-32537) [n:127.0.0.1:59273_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=8
[junit4] 2> 3267877 DEBUG (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
[junit4] 2> 3267877 DEBUG (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:59325_solr, 127.0.0.1:59252_solr, 127.0.0.1:59273_solr, 127.0.0.1:59239_solr]
[junit4] 2> 3267877 DEBUG (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger2 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 3267878 DEBUG (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:59325_solr, 127.0.0.1:59252_solr, 127.0.0.1:59273_solr, 127.0.0.1:59239_solr]
[junit4] 2> 3267878 DEBUG (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:59325_solr, 127.0.0.1:59252_solr, 127.0.0.1:59273_solr, 127.0.0.1:59239_solr]
[junit4] 2> 3267878 DEBUG (zkCallback-8215-thread-1-processing-n:127.0.0.1:59239_solr) [n:127.0.0.1:59239_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger1 instantiated with properties: {event=nodeAdded, waitFor=0, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$ThrottlingTesterAction}], enabled=true}
[junit4] 2> 3267879 DEBUG (OverseerAutoScalingTriggerThread-99426966786146310-127.0.0.1:59239_solr-n_0000000001) [n:127.0.0.1:59239_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
[junit4] 2> 3267880 INFO (qtp1520736820-32451) [n:127.0.0.1:59239_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3267882 INFO (OverseerAutoScalingTrigg
[...truncated too long message...]
5566728-core_node3-n_0000000001
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 11 /solr/live_nodes
[junit4] 2> 11 /solr/collections
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.handler.admin.AutoscalingHistoryHandlerTest_1B0489124781423E-001
[junit4] 2> Jan 28, 2018 10:31:02 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {action_s=PostingsFormat(name=Direct), event.source_s=FSTOrd50, type=PostingsFormat(name=MockRandom), before.actions_ss=Lucene50(blocksize=128), event.property._enqueue_time__ss=PostingsFormat(name=Direct), after.actions_ss=PostingsFormat(name=Direct), stage_s=PostingsFormat(name=Direct), event.id_s=FSTOrd50, event.property.eventTimes_ss=PostingsFormat(name=MockRandom), source_s=FSTOrd50, event.type_s=PostingsFormat(name=MockRandom), responses_ts=PostingsFormat(name=Direct), id=FSTOrd50, collections_ss=PostingsFormat(name=MockRandom), operations.params_ts=PostingsFormat(name=Direct), event.property.nodeNames_ss=PostingsFormat(name=MockRandom)}, docValues:{_version_=DocValuesFormat(name=Direct), event.time_l=DocValuesFormat(name=Lucene70), timestamp=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=548, maxMBSortInHeap=6.896752111836637, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@529cf6e6), locale=sr-RS, timezone=Atlantic/Reykjavik
[junit4] 2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=3,threads=1,free=200721032,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [DeleteLastCustomShardedReplicaTest, JSONWriterTest, UtilsToolTest, AliasIntegrationTest, CustomHighlightComponentTest, ZkSolrClientTest, ChaosMonkeySafeLeaderWithPullReplicasTest, TestUnifiedSolrHighlighter, AnalysisErrorHandlingTest, BlockJoinFacetDistribTest, V2ApiIntegrationTest, TestDistributedSearch, CustomCollectionTest, TestDFRSimilarityFactory, TestManagedSynonymGraphFilterFactory, TestDelegationWithHadoopAuth, PreAnalyzedUpdateProcessorTest, FieldAnalysisRequestHandlerTest, TestExactSharedStatsCache, TestCSVLoader, HdfsDirectoryTest, TestLeaderElectionWithEmptyReplica, SyncSliceTest, TestWriterPerf, DistributedIntervalFacetingTest, CollectionStateFormat2Test, TestXmlQParserPlugin, DistributedTermsComponentTest, HdfsChaosMonkeyNothingIsSafeTest, TestSearchPerf, TestStandardQParsers, TestSolrDeletionPolicy2, BlockDirectoryTest, SegmentsInfoRequestHandlerTest, SuggestComponentTest, UninvertDocValuesMergePolicyTest, TestFieldCollectionResource, ConjunctionSolrSpellCheckerTest, DeleteStatusTest, FullSolrCloudDistribCmdsTest, SpatialRPTFieldTypeTest, SolrShardReporterTest, TestApiFramework, HdfsWriteToMultipleCollectionsTest, SearchRateTriggerTest, CdcrRequestHandlerTest, NoCacheHeaderTest, OverseerStatusTest, CSVRequestHandlerTest, TestCloudNestedDocsSort, TestShardHandlerFactory, TestSolrCloudWithDelegationTokens, AnalyticsMergeStrategyTest, HdfsRecoveryZkTest, BasicFunctionalityTest, AssignTest, MoveReplicaTest, TestConfigOverlay, FullHLLTest, SolrCLIZkUtilsTest, RuleEngineTest, SolrCoreMetricManagerTest, DefaultValueUpdateProcessorTest, TestStressVersions, TestIntervalFaceting, TestLuceneIndexBackCompat, DistributedExpandComponentTest, TestSolrCloudWithHadoopAuthPlugin, TestDownShardTolerantSearch, SolrXmlInZkTest, TemplateUpdateProcessorTest, TestRTGBase, TestSchemaManager, ExternalFileFieldSortTest, TestPolicyCloud, DocumentAnalysisRequestHandlerTest, TestSchemaSimilarityResource, TestMissingGroups, TestSmileRequest, DistributedQueryComponentOptimizationTest, TestCloudDeleteByQuery, HLLUtilTest, RemoteQueryErrorTest, TestGenericDistributedQueue, PrimUtilsTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, SpellCheckCollatorTest, ShowFileRequestHandlerTest, TestFieldTypeCollectionResource, SolrCoreCheckLockOnStartupTest, QueryParsingTest, LargeFieldTest, SchemaApiFailureTest, CheckHdfsIndexTest, TestGraphMLResponseWriter, TestSQLHandlerNonCloud, TestSolrQueryResponse, RequestHandlersTest, TestTlogReplica, CoreAdminRequestStatusTest, TestJettySolrRunner, TestRecoveryHdfs, TestFileDictionaryLookup, CoreSorterTest, TestRandomDVFaceting, TestScoreJoinQPNoScore, DistributedQueueTest, SaslZkACLProviderTest, TestAuthorizationFramework, UpdateParamsTest, TestSchemaNameResource, SubstringBytesRefFilterTest, BlobRepositoryCloudTest, TestSizeLimitedDistributedMap, DOMUtilTest, TestUseDocValuesAsStored, PrimitiveFieldTypeTest, HighlighterConfigTest, SuggesterTest, TestFastLRUCache, TestDistribStateManager, LoggingHandlerTest, HdfsTlogReplayBufferedWhileIndexingTest, TestRuleBasedAuthorizationPlugin, OverseerTest, DocValuesMissingTest, URLClassifyProcessorTest, TestFieldCacheSortRandom, TestValueSourceCache, CoreAdminHandlerTest, ReplicaListTransformerTest, TestDeleteCollectionOnDownNodes, UpdateRequestProcessorFactoryTest, TestAnalyzeInfixSuggestions, DisMaxRequestHandlerTest, TestDynamicFieldCollectionResource, TestClusterStateMutator, UUIDUpdateProcessorFallbackTest, NumberUtilsTest, DistributedFacetPivotWhiteBoxTest, JvmMetricsTest, DebugComponentTest, TestPullReplicaErrorHandling, TestHashQParserPlugin, TestJsonFacetRefinement, TestStressRecovery, ChaosMonkeySafeLeaderTest, TestInPlaceUpdatesDistrib, TestLRUCache, SolrJmxReporterCloudTest, SolrSlf4jReporterTest, TestCloudPseudoReturnFields, UUIDFieldTest, DistributedSpellCheckComponentTest, HealthCheckHandlerTest, TestExpandComponent, ScriptEngineTest, TestHashPartitioner, TestUninvertingReader, TestCoreBackup, TestInfoStreamLogging, TestAddFieldRealTimeGet, AtomicUpdatesTest, TestSolrCoreParser, QueryResultKeyTest, SolrMetricReporterTest, SimpleFacetsTest, AutoAddReplicasIntegrationTest, TestMinMaxOnMultiValuedField, FileBasedSpellCheckerTest, CleanupOldIndexTest, TestTrieFacet, TestRequestForwarding, TestSubQueryTransformer, CreateRoutedAliasTest, TestCollationFieldDocValues, TestCoreDiscovery, NotRequiredUniqueKeyTest, ExecutePlanActionTest, TestLRUStatsCache, TestConfigSetProperties, TestHdfsCloudBackupRestore, TestConfig, TestLMJelinekMercerSimilarityFactory, TermVectorComponentTest, TestDefaultStatsCache, FacetPivotSmallTest, JsonLoaderTest, TestConfigsApi, ClusterStateUpdateTest, TestIndexingPerformance, TestLMDirichletSimilarityFactory, TestIndexSearcher, SimplePostToolTest, SignatureUpdateProcessorFactoryTest, TestExtendedDismaxParser, SuggesterFSTTest, TestFoldingMultitermQuery, DocValuesTest, TestStressLucene, TestTrie, SpatialFilterTest, SuggesterWFSTTest, PolyFieldTest, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, TestUpdate, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, QueryEqualityTest, DirectUpdateHandlerOptimizeTest, SortByFunctionTest, DocValuesMultiTest, DistanceFunctionTest, TestSolrDeletionPolicy1, SolrInfoBeanTest, XsltUpdateRequestHandlerTest, CacheHeaderTest, IndexBasedSpellCheckerTest, TestSurroundQueryParser, TestQueryUtils, DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, TestOmitPositions, TermsComponentTest, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, TestSolrQueryParser, RegexBoostProcessorTest, IndexSchemaTest, TestJmxIntegration, ReturnFieldsTest, TestCSVResponseWriter, TestAnalyzedSuggestions, TestPHPSerializedResponseWriter, TestComponentsName, TestLFUCache, TestQuerySenderListener, TestSolrIndexConfig, TestQuerySenderNoQuery, CopyFieldTest, ResponseLogComponentTest, BadComponentTest, MultiTermTest, SampleTest, TestDocSet, TestBinaryField, NumericFieldsTest, MinimalSchemaTest, TestElisionMultitermQuery, OutputWriterTest, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties, TestPostingsSolrHighlighter, TestLuceneMatchVersion, TestPhraseSuggestions, SpellPossibilityIteratorTest, TestCharFilters, TestCodecSupport, TestXIncludeConfig, EchoParamsTest, TestIBSimilarityFactory, TestFastWriter, TimeZoneUtilsTest, ResourceLoaderTest, TestFastOutputStream, PluginInfoTest, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, DistribDocExpirationUpdateProcessorTest, LeaderElectionContextKeyTest, LeaderFailureAfterFreshStartTest, LeaderInitiatedRecoveryOnCommitTest, MigrateRouteKeyTest, NodeMutatorTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerCollectionConfigSetProcessorTest, OverseerModifyCollectionTest, OverseerRolesTest, PeerSyncReplicationTest, SolrCloudExampleTest, TestCloudPivotFacet, TestCloudSearcherWarming, TestConfigSetsAPIZkFailure, TestExclusionRuleCollectionAccess, TestLeaderInitiatedRecoveryThread, TestLockTree, TestMiniSolrCloudClusterSSL, TestPrepRecovery, TestRandomRequestDistribution, TestSegmentSorting, TestSolrCloudWithKerberosAlt, TestStressInPlaceUpdates, CollectionReloadTest, CollectionTooManyReplicasTest, CollectionsAPIAsyncDistributedZkTest, ConcurrentDeleteAndCreateCollectionTest, HdfsCollectionsAPIDistributedZkTest, ShardSplitTest, TestClusterStateProvider, TestExecutePlanAction, TestLargeCluster, TestNodeAddedTrigger, TestPolicyCloud, TestTriggerIntegration, BaseCdcrDistributedZkTest, CdcrBidirectionalTest, CdcrReplicationDistributedZkTest, CdcrVersionReplicationTest, HdfsBasicDistributedZk2Test, HdfsBasicDistributedZkTest, HdfsChaosMonkeySafeLeaderTest, HdfsNNFailoverTest, HdfsRecoverLeaseTest, HdfsRestartWhileUpdatingTest, HdfsSyncSliceTest, HdfsUnloadDistributedZkTest, ZkStateReaderTest, ZkStateWriterTest, RulesTest, TestNRTOpen, TestReloadAndDeleteDocs, TestSolrConfigHandler, TestSolrCloudSnapshots, RequestLoggingTest, TestBlobHandler, TestRestoreCore, TestSQLHandler, V2StandaloneTest, AutoscalingHistoryHandlerTest]
[junit4] Completed [642/770 (2!)] on J1 in 20.37s, 1 test, 1 failure <<< FAILURES!
[...truncated 48595 lines...]
[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-9) - Build # 4412 -
Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4412/
Java: 64bit/jdk-9 -XX:-UseCompressedOops -XX:+UseG1GC
1 tests failed.
FAILED: org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple
Error Message:
No live SolrServers available to handle this request:[http://127.0.0.1:61001/solr]
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:61001/solr]
at __randomizedtesting.SeedInfo.seed([76984BF2D9AAB1EF:4E2B6F0CFE59653E]:0)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple(AutoAddReplicasPlanActionTest.java:110)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:61001/solr
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
... 43 more
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:61001 failed to respond
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
... 47 more
Build Log:
[...truncated 13595 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/init-core-data-001
[junit4] 2> 3198567 WARN (SUITE-AutoAddReplicasPlanActionTest-seed#[76984BF2D9AAB1EF]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=21 numCloses=21
[junit4] 2> 3198567 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[76984BF2D9AAB1EF]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 3198568 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[76984BF2D9AAB1EF]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0) w/ MAC_OS_X supressed clientAuth
[junit4] 2> 3198568 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[76984BF2D9AAB1EF]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 3198569 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[76984BF2D9AAB1EF]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 3 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001
[junit4] 2> 3198569 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[76984BF2D9AAB1EF]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 3198569 INFO (Thread-4909) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 3198569 INFO (Thread-4909) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 3198572 ERROR (Thread-4909) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 3198675 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[76984BF2D9AAB1EF]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:60999
[junit4] 2> 3198684 INFO (zkConnectionManagerCallback-6016-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3198688 INFO (jetty-launcher-6013-thread-1) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T11:27:37-10:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 3198688 INFO (jetty-launcher-6013-thread-2) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T11:27:37-10:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 3198689 INFO (jetty-launcher-6013-thread-3) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T11:27:37-10:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 3198689 INFO (jetty-launcher-6013-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 3198689 INFO (jetty-launcher-6013-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 3198689 INFO (jetty-launcher-6013-thread-1) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 3198690 INFO (jetty-launcher-6013-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 3198690 INFO (jetty-launcher-6013-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 3198690 INFO (jetty-launcher-6013-thread-2) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 3198691 INFO (jetty-launcher-6013-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@72c3e7d6{/solr,null,AVAILABLE}
[junit4] 2> 3198692 INFO (jetty-launcher-6013-thread-3) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 3198692 INFO (jetty-launcher-6013-thread-3) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 3198692 INFO (jetty-launcher-6013-thread-3) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 3198692 INFO (jetty-launcher-6013-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4a02a509{/solr,null,AVAILABLE}
[junit4] 2> 3198693 INFO (jetty-launcher-6013-thread-3) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@954bca6{/solr,null,AVAILABLE}
[junit4] 2> 3198693 INFO (jetty-launcher-6013-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@6657fdfb{HTTP/1.1,[http/1.1]}{127.0.0.1:61001}
[junit4] 2> 3198693 INFO (jetty-launcher-6013-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@62636645{HTTP/1.1,[http/1.1]}{127.0.0.1:61002}
[junit4] 2> 3198693 INFO (jetty-launcher-6013-thread-2) [ ] o.e.j.s.Server Started @3201247ms
[junit4] 2> 3198693 INFO (jetty-launcher-6013-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61002}
[junit4] 2> 3198693 ERROR (jetty-launcher-6013-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3198693 INFO (jetty-launcher-6013-thread-1) [ ] o.e.j.s.Server Started @3201247ms
[junit4] 2> 3198694 INFO (jetty-launcher-6013-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 3198694 INFO (jetty-launcher-6013-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61001}
[junit4] 2> 3198694 INFO (jetty-launcher-6013-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3198694 INFO (jetty-launcher-6013-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3198694 ERROR (jetty-launcher-6013-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3198694 INFO (jetty-launcher-6013-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-27T22:36:01.482782Z
[junit4] 2> 3198694 INFO (jetty-launcher-6013-thread-3) [ ] o.e.j.s.AbstractConnector Started ServerConnector@31a74ab5{HTTP/1.1,[http/1.1]}{127.0.0.1:61003}
[junit4] 2> 3198694 INFO (jetty-launcher-6013-thread-3) [ ] o.e.j.s.Server Started @3201248ms
[junit4] 2> 3198695 INFO (jetty-launcher-6013-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=61003}
[junit4] 2> 3198695 ERROR (jetty-launcher-6013-thread-3) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3198695 INFO (jetty-launcher-6013-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 3198695 INFO (jetty-launcher-6013-thread-3) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 3198695 INFO (jetty-launcher-6013-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3198695 INFO (jetty-launcher-6013-thread-3) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3198695 INFO (jetty-launcher-6013-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3198695 INFO (jetty-launcher-6013-thread-3) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3198695 INFO (jetty-launcher-6013-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-27T22:36:01.483466Z
[junit4] 2> 3198695 INFO (jetty-launcher-6013-thread-3) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-27T22:36:01.483512Z
[junit4] 2> 3198699 INFO (zkConnectionManagerCallback-6018-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3198699 INFO (zkConnectionManagerCallback-6022-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3198700 INFO (zkConnectionManagerCallback-6020-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3198700 INFO (jetty-launcher-6013-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3198700 INFO (jetty-launcher-6013-thread-3) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3198701 INFO (jetty-launcher-6013-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3198712 INFO (jetty-launcher-6013-thread-3) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60999/solr
[junit4] 2> 3198714 INFO (jetty-launcher-6013-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60999/solr
[junit4] 2> 3198716 INFO (zkConnectionManagerCallback-6026-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3198717 INFO (zkConnectionManagerCallback-6030-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3198718 INFO (jetty-launcher-6013-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60999/solr
[junit4] 2> 3198725 INFO (zkConnectionManagerCallback-6034-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3198726 INFO (zkConnectionManagerCallback-6036-thread-1-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3198727 INFO (zkConnectionManagerCallback-6038-thread-1-processing-n:127.0.0.1:61002_solr) [n:127.0.0.1:61002_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3198731 INFO (zkConnectionManagerCallback-6040-thread-1-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3198877 INFO (jetty-launcher-6013-thread-3) [n:127.0.0.1:61003_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3198877 INFO (jetty-launcher-6013-thread-1) [n:127.0.0.1:61001_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3198878 INFO (jetty-launcher-6013-thread-3) [n:127.0.0.1:61003_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:61003_solr
[junit4] 2> 3198879 INFO (jetty-launcher-6013-thread-1) [n:127.0.0.1:61001_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61001_solr
[junit4] 2> 3198879 INFO (jetty-launcher-6013-thread-3) [n:127.0.0.1:61003_solr ] o.a.s.c.Overseer Overseer (id=99424178101223431-127.0.0.1:61003_solr-n_0000000000) starting
[junit4] 2> 3198881 INFO (zkCallback-6035-thread-1-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 3198882 INFO (zkCallback-6039-thread-1-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 3198892 INFO (jetty-launcher-6013-thread-3) [n:127.0.0.1:61003_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61003_solr
[junit4] 2> 3198894 INFO (zkCallback-6035-thread-1-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 3198895 INFO (zkCallback-6039-thread-1-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 3198939 INFO (jetty-launcher-6013-thread-1) [n:127.0.0.1:61001_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61001.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ded7df8
[junit4] 2> 3198945 INFO (jetty-launcher-6013-thread-3) [n:127.0.0.1:61003_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61003.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ded7df8
[junit4] 2> 3198946 INFO (jetty-launcher-6013-thread-1) [n:127.0.0.1:61001_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61001.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ded7df8
[junit4] 2> 3198947 INFO (jetty-launcher-6013-thread-1) [n:127.0.0.1:61001_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61001.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ded7df8
[junit4] 2> 3198947 INFO (jetty-launcher-6013-thread-1) [n:127.0.0.1:61001_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001/node1/.
[junit4] 2> 3198951 INFO (jetty-launcher-6013-thread-2) [n:127.0.0.1:61002_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 3198952 INFO (jetty-launcher-6013-thread-3) [n:127.0.0.1:61003_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61003.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ded7df8
[junit4] 2> 3198952 INFO (jetty-launcher-6013-thread-3) [n:127.0.0.1:61003_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61003.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ded7df8
[junit4] 2> 3198953 INFO (jetty-launcher-6013-thread-3) [n:127.0.0.1:61003_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001/node3/.
[junit4] 2> 3198953 INFO (jetty-launcher-6013-thread-2) [n:127.0.0.1:61002_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3198956 INFO (jetty-launcher-6013-thread-2) [n:127.0.0.1:61002_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61002_solr
[junit4] 2> 3198957 INFO (zkCallback-6039-thread-1-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 3198958 INFO (zkCallback-6037-thread-1-processing-n:127.0.0.1:61002_solr) [n:127.0.0.1:61002_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 3198958 INFO (zkCallback-6035-thread-1-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 3199001 INFO (jetty-launcher-6013-thread-2) [n:127.0.0.1:61002_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61002.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ded7df8
[junit4] 2> 3199008 INFO (jetty-launcher-6013-thread-2) [n:127.0.0.1:61002_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61002.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ded7df8
[junit4] 2> 3199008 INFO (jetty-launcher-6013-thread-2) [n:127.0.0.1:61002_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61002.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ded7df8
[junit4] 2> 3199009 INFO (jetty-launcher-6013-thread-2) [n:127.0.0.1:61002_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001/node2/.
[junit4] 2> 3199033 INFO (zkConnectionManagerCallback-6048-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3199036 INFO (zkConnectionManagerCallback-6052-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3199038 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[76984BF2D9AAB1EF]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 3199039 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[76984BF2D9AAB1EF]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:60999/solr ready
[junit4] 2> 3199057 INFO (TEST-AutoAddReplicasPlanActionTest.testSimple-seed#[76984BF2D9AAB1EF]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testSimple
[junit4] 2> 3199060 INFO (qtp829746145-27344) [n:127.0.0.1:61001_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&maxShardsPerNode=2&autoAddReplicas=true&name=testSimple1&nrtReplicas=2&action=CREATE&numShards=2&createNodeSet=127.0.0.1:61001_solr,127.0.0.1:61003_solr&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 3199062 INFO (OverseerThreadFactory-11037-thread-1-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.a.c.CreateCollectionCmd Create collection testSimple1
[junit4] 2> 3199175 INFO (OverseerStateUpdate-99424178101223431-127.0.0.1:61003_solr-n_0000000000) [n:127.0.0.1:61003_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testSimple1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"testSimple1_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:61001/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 3199179 INFO (OverseerStateUpdate-99424178101223431-127.0.0.1:61003_solr-n_0000000000) [n:127.0.0.1:61003_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testSimple1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"testSimple1_shard1_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:61003/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 3199182 INFO (OverseerStateUpdate-99424178101223431-127.0.0.1:61003_solr-n_0000000000) [n:127.0.0.1:61003_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testSimple1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"testSimple1_shard2_replica_n4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:61001/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 3199185 INFO (OverseerStateUpdate-99424178101223431-127.0.0.1:61003_solr-n_0000000000) [n:127.0.0.1:61003_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testSimple1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"testSimple1_shard2_replica_n6",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:61003/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 3199395 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n1&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 3199395 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 3199395 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n4&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 3199396 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n2&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 3199396 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n6&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 3199397 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 3199506 INFO (zkCallback-6035-thread-1-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3199506 INFO (zkCallback-6039-thread-1-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3199506 INFO (zkCallback-6039-thread-2-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3199506 INFO (zkCallback-6035-thread-2-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3200413 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 3200414 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 3200413 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 3200431 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 3200432 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.IndexSchema [testSimple1_shard2_replica_n6] Schema name=minimal
[junit4] 2> 3200434 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.IndexSchema [testSimple1_shard2_replica_n4] Schema name=minimal
[junit4] 2> 3200434 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 3200434 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard2_replica_n6' using configuration from collection testSimple1, trusted=true
[junit4] 2> 3200435 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61003.solr.core.testSimple1.shard2.replica_n6' (registry 'solr.core.testSimple1.shard2.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ded7df8
[junit4] 2> 3200435 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 3200435 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrCore [[testSimple1_shard2_replica_n6] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001/node3/testSimple1_shard2_replica_n6], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001/node3/./testSimple1_shard2_replica_n6/data/]
[junit4] 2> 3200435 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 3200435 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard2_replica_n4' using configuration from collection testSimple1, trusted=true
[junit4] 2> 3200436 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61001.solr.core.testSimple1.shard2.replica_n4' (registry 'solr.core.testSimple1.shard2.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ded7df8
[junit4] 2> 3200436 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 3200436 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SolrCore [[testSimple1_shard2_replica_n4] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001/node1/testSimple1_shard2_replica_n4], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001/node1/./testSimple1_shard2_replica_n4/data/]
[junit4] 2> 3200439 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.IndexSchema [testSimple1_shard1_replica_n2] Schema name=minimal
[junit4] 2> 3200439 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.IndexSchema [testSimple1_shard1_replica_n1] Schema name=minimal
[junit4] 2> 3200440 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 3200440 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard1_replica_n2' using configuration from collection testSimple1, trusted=true
[junit4] 2> 3200440 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61003.solr.core.testSimple1.shard1.replica_n2' (registry 'solr.core.testSimple1.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ded7df8
[junit4] 2> 3200441 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 3200441 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrCore [[testSimple1_shard1_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001/node3/testSimple1_shard1_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001/node3/./testSimple1_shard1_replica_n2/data/]
[junit4] 2> 3200441 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 3200441 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testSimple1_shard1_replica_n1' using configuration from collection testSimple1, trusted=true
[junit4] 2> 3200441 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61001.solr.core.testSimple1.shard1.replica_n1' (registry 'solr.core.testSimple1.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ded7df8
[junit4] 2> 3200441 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 3200442 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrCore [[testSimple1_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001/node1/testSimple1_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001/node1/./testSimple1_shard1_replica_n1/data/]
[junit4] 2> 3200706 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 3200706 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 3200707 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 3200707 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 3200707 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 3200707 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 3200708 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 3200708 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 3200709 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 3200709 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 3200709 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 3200709 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 3200709 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 3200710 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 3200710 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@428ebc92[testSimple1_shard1_replica_n1] main]
[junit4] 2> 3200711 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@385cb823[testSimple1_shard1_replica_n2] main]
[junit4] 2> 3200712 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 3200712 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 3200712 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 3200712 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 3200712 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 3200713 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 3200713 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 3200713 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590786851865624576
[junit4] 2> 3200713 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 3200713 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590786851865624576
[junit4] 2> 3200713 INFO (searcherExecutor-11047-thread-1-processing-n:127.0.0.1:61001_solr x:testSimple1_shard1_replica_n1 s:shard1 c:testSimple1 r:core_node3) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.SolrCore [testSimple1_shard1_replica_n1] Registered new searcher Searcher@428ebc92[testSimple1_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 3200714 INFO (searcherExecutor-11046-thread-1-processing-n:127.0.0.1:61003_solr x:testSimple1_shard1_replica_n2 s:shard1 c:testSimple1 r:core_node5) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SolrCore [testSimple1_shard1_replica_n2] Registered new searcher Searcher@385cb823[testSimple1_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 3200715 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.SolrIndexSearcher Opening [Searcher@6bb18ca9[testSimple1_shard2_replica_n6] main]
[junit4] 2> 3200716 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 3200716 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 3200717 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 3200717 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590786851869818880
[junit4] 2> 3200718 INFO (searcherExecutor-11044-thread-1-processing-n:127.0.0.1:61003_solr x:testSimple1_shard2_replica_n6 s:shard2 c:testSimple1 r:core_node8) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SolrCore [testSimple1_shard2_replica_n6] Registered new searcher Searcher@6bb18ca9[testSimple1_shard2_replica_n6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 3200721 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 3200721 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 3200721 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:61003/solr/testSimple1_shard1_replica_n2/
[junit4] 2> 3200721 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=testSimple1_shard1_replica_n2 url=http://127.0.0.1:61003/solr START replicas=[http://127.0.0.1:61001/solr/testSimple1_shard1_replica_n1/] nUpdates=100
[junit4] 2> 3200723 INFO (qtp829746145-27349) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.c.S.Request [testSimple1_shard1_replica_n1] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 3200723 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.SolrIndexSearcher Opening [Searcher@49a7a428[testSimple1_shard2_replica_n4] main]
[junit4] 2> 3200724 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
[junit4] 2> 3200725 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 3200725 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 3200725 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 3200726 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590786851879256064
[junit4] 2> 3200726 INFO (searcherExecutor-11045-thread-1-processing-n:127.0.0.1:61001_solr x:testSimple1_shard2_replica_n4 s:shard2 c:testSimple1 r:core_node7) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.SolrCore [testSimple1_shard2_replica_n4] Registered new searcher Searcher@49a7a428[testSimple1_shard2_replica_n4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 3200825 INFO (zkCallback-6035-thread-2-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3200825 INFO (zkCallback-6039-thread-1-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3200825 INFO (zkCallback-6035-thread-3-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3200825 INFO (zkCallback-6039-thread-2-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201028 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.u.PeerSync PeerSync: core=testSimple1_shard1_replica_n2 url=http://127.0.0.1:61003/solr DONE. We have no versions. sync failed.
[junit4] 2> 3201028 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 3201028 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 3201029 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 3201033 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:61003/solr/testSimple1_shard1_replica_n2/ shard1
[junit4] 2> 3201135 INFO (zkCallback-6035-thread-3-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201136 INFO (zkCallback-6035-thread-2-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201136 INFO (zkCallback-6039-thread-2-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201136 INFO (zkCallback-6039-thread-1-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201194 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 3201197 INFO (qtp2027826936-27363) [n:127.0.0.1:61003_solr c:testSimple1 s:shard1 r:core_node5 x:testSimple1_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n2&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1800
[junit4] 2> 3201232 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 3201232 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 3201232 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:61003/solr/testSimple1_shard2_replica_n6/
[junit4] 2> 3201232 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.PeerSync PeerSync: core=testSimple1_shard2_replica_n6 url=http://127.0.0.1:61003/solr START replicas=[http://127.0.0.1:61001/solr/testSimple1_shard2_replica_n4/] nUpdates=100
[junit4] 2> 3201233 INFO (qtp829746145-27350) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.c.S.Request [testSimple1_shard2_replica_n4] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 3201299 INFO (zkCallback-6035-thread-2-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201299 INFO (zkCallback-6039-thread-1-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201299 INFO (zkCallback-6039-thread-2-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201299 INFO (zkCallback-6035-thread-3-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201533 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.u.PeerSync PeerSync: core=testSimple1_shard2_replica_n6 url=http://127.0.0.1:61003/solr DONE. We have no versions. sync failed.
[junit4] 2> 3201533 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 3201533 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 3201533 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 3201538 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:61003/solr/testSimple1_shard2_replica_n6/ shard2
[junit4] 2> 3201640 INFO (zkCallback-6035-thread-3-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201640 INFO (zkCallback-6039-thread-1-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201640 INFO (zkCallback-6039-thread-2-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201640 INFO (zkCallback-6035-thread-2-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201690 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 3201692 INFO (qtp2027826936-27368) [n:127.0.0.1:61003_solr c:testSimple1 s:shard2 r:core_node8 x:testSimple1_shard2_replica_n6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n6&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2295
[junit4] 2> 3201728 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr c:testSimple1 s:shard1 r:core_node3 x:testSimple1_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testSimple1_shard1_replica_n1&action=CREATE&numShards=2&collection=testSimple1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2333
[junit4] 2> 3201731 INFO (qtp829746145-27346) [n:127.0.0.1:61001_solr c:testSimple1 s:shard2 r:core_node7 x:testSimple1_shard2_replica_n4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=testSimple1_shard2_replica_n4&action=CREATE&numShards=2&collection=testSimple1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2335
[junit4] 2> 3201734 INFO (qtp829746145-27344) [n:127.0.0.1:61001_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 3201836 INFO (zkCallback-6039-thread-2-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201836 INFO (zkCallback-6035-thread-3-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201836 INFO (zkCallback-6039-thread-1-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3201836 INFO (zkCallback-6035-thread-2-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple1/state.json] for collection [testSimple1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3202734 INFO (qtp829746145-27344) [n:127.0.0.1:61001_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&collection.configName=conf&maxShardsPerNode=2&autoAddReplicas=true&name=testSimple1&nrtReplicas=2&action=CREATE&numShards=2&createNodeSet=127.0.0.1:61001_solr,127.0.0.1:61003_solr&wt=javabin&version=2} status=0 QTime=3674
[junit4] 2> 3202735 INFO (qtp829746145-27349) [n:127.0.0.1:61001_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&autoAddReplicas=false&name=testSimple2&nrtReplicas=2&action=CREATE&numShards=1&createNodeSet=127.0.0.1:61003_solr,127.0.0.1:61002_solr&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 3202738 INFO (OverseerThreadFactory-11037-thread-2-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.a.c.CreateCollectionCmd Create collection testSimple2
[junit4] 2> 3202738 INFO (OverseerCollectionConfigSetProcessor-99424178101223431-127.0.0.1:61003_solr-n_0000000000) [n:127.0.0.1:61003_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 3202958 INFO (OverseerStateUpdate-99424178101223431-127.0.0.1:61003_solr-n_0000000000) [n:127.0.0.1:61003_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testSimple2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"testSimple2_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:61003/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 3202965 INFO (OverseerStateUpdate-99424178101223431-127.0.0.1:61003_solr-n_0000000000) [n:127.0.0.1:61003_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testSimple2",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"testSimple2_shard1_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:61002/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 3203167 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testSimple2_shard1_replica_n1&action=CREATE&numShards=1&collection=testSimple2&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 3203169 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=testSimple2_shard1_replica_n2&action=CREATE&numShards=1&collection=testSimple2&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 3203171 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 3203289 INFO (zkCallback-6037-thread-1-processing-n:127.0.0.1:61002_solr) [n:127.0.0.1:61002_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3203289 INFO (zkCallback-6035-thread-3-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3203289 INFO (zkCallback-6035-thread-2-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3203289 INFO (zkCallback-6037-thread-2-processing-n:127.0.0.1:61002_solr) [n:127.0.0.1:61002_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3204183 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 3204186 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 3204190 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.s.IndexSchema [testSimple2_shard1_replica_n1] Schema name=minimal
[junit4] 2> 3204192 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 3204192 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testSimple2_shard1_replica_n1' using configuration from collection testSimple2, trusted=true
[junit4] 2> 3204192 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.s.IndexSchema [testSimple2_shard1_replica_n2] Schema name=minimal
[junit4] 2> 3204193 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61003.solr.core.testSimple2.shard1.replica_n1' (registry 'solr.core.testSimple2.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ded7df8
[junit4] 2> 3204193 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 3204193 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.SolrCore [[testSimple2_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001/node3/testSimple2_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001/node3/./testSimple2_shard1_replica_n1/data/]
[junit4] 2> 3204194 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 3204194 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testSimple2_shard1_replica_n2' using configuration from collection testSimple2, trusted=true
[junit4] 2> 3204194 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_61002.solr.core.testSimple2.shard1.replica_n2' (registry 'solr.core.testSimple2.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1ded7df8
[junit4] 2> 3204194 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 3204194 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.c.SolrCore [[testSimple2_shard1_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001/node2/testSimple2_shard1_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001/tempDir-001/node2/./testSimple2_shard1_replica_n2/data/]
[junit4] 2> 3204239 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 3204239 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 3204242 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 3204242 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 3204243 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@68c81b0[testSimple2_shard1_replica_n1] main]
[junit4] 2> 3204245 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 3204245 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 3204246 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 3204246 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590786855570243584
[junit4] 2> 3204247 INFO (searcherExecutor-11064-thread-1-processing-n:127.0.0.1:61003_solr x:testSimple2_shard1_replica_n1 s:shard1 c:testSimple2 r:core_node3) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.SolrCore [testSimple2_shard1_replica_n1] Registered new searcher Searcher@68c81b0[testSimple2_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 3204254 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=9999ms
[junit4] 2> 3204256 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 3204256 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 3204258 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 3204258 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 3204260 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@6b094501[testSimple2_shard1_replica_n2] main]
[junit4] 2> 3204261 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 3204262 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 3204262 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 3204263 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590786855588069376
[junit4] 2> 3204263 INFO (searcherExecutor-11065-thread-1-processing-n:127.0.0.1:61002_solr x:testSimple2_shard1_replica_n2 s:shard1 c:testSimple2 r:core_node4) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.c.SolrCore [testSimple2_shard1_replica_n2] Registered new searcher Searcher@6b094501[testSimple2_shard1_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 3204356 INFO (zkCallback-6035-thread-2-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3204356 INFO (zkCallback-6037-thread-1-processing-n:127.0.0.1:61002_solr) [n:127.0.0.1:61002_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3204356 INFO (zkCallback-6037-thread-2-processing-n:127.0.0.1:61002_solr) [n:127.0.0.1:61002_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3204356 INFO (zkCallback-6035-thread-3-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3204758 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 3204758 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 3204758 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:61003/solr/testSimple2_shard1_replica_n1/
[junit4] 2> 3204758 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=testSimple2_shard1_replica_n1 url=http://127.0.0.1:61003/solr START replicas=[http://127.0.0.1:61002/solr/testSimple2_shard1_replica_n2/] nUpdates=100
[junit4] 2> 3204759 INFO (qtp257857186-27357) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.c.S.Request [testSimple2_shard1_replica_n2] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 3205061 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.u.PeerSync PeerSync: core=testSimple2_shard1_replica_n1 url=http://127.0.0.1:61003/solr DONE. We have no versions. sync failed.
[junit4] 2> 3205062 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 3205062 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 3205062 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 3205074 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:61003/solr/testSimple2_shard1_replica_n1/ shard1
[junit4] 2> 3205181 INFO (zkCallback-6035-thread-3-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3205181 INFO (zkCallback-6035-thread-2-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3205181 INFO (zkCallback-6037-thread-1-processing-n:127.0.0.1:61002_solr) [n:127.0.0.1:61002_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3205181 INFO (zkCallback-6037-thread-2-processing-n:127.0.0.1:61002_solr) [n:127.0.0.1:61002_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3205234 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 3205236 INFO (qtp2027826936-27364) [n:127.0.0.1:61003_solr c:testSimple2 s:shard1 r:core_node3 x:testSimple2_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testSimple2_shard1_replica_n1&action=CREATE&numShards=1&collection=testSimple2&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2068
[junit4] 2> 3205283 INFO (qtp257857186-27352) [n:127.0.0.1:61002_solr c:testSimple2 s:shard1 r:core_node4 x:testSimple2_shard1_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=testSimple2_shard1_replica_n2&action=CREATE&numShards=1&collection=testSimple2&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=2114
[junit4] 2> 3205288 INFO (qtp829746145-27349) [n:127.0.0.1:61001_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 3205396 INFO (zkCallback-6037-thread-2-processing-n:127.0.0.1:61002_solr) [n:127.0.0.1:61002_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3205396 INFO (zkCallback-6037-thread-1-processing-n:127.0.0.1:61002_solr) [n:127.0.0.1:61002_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3205396 INFO (zkCallback-6035-thread-2-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3205396 INFO (zkCallback-6035-thread-3-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testSimple2/state.json] for collection [testSimple2] has occurred - updating... (live nodes size: [3])
[junit4] 2> 3206299 INFO (qtp829746145-27349) [n:127.0.0.1:61001_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&collection.configName=conf&maxShardsPerNode=1&autoAddReplicas=false&name=testSimple2&nrtReplicas=2&action=CREATE&numShards=1&createNodeSet=127.0.0.1:61003_solr,127.0.0.1:61002_solr&wt=javabin&version=2} status=0 QTime=3563
[junit4] 2> 3206299 INFO (qtp829746145-27348) [n:127.0.0.1:61001_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf&maxShardsPerNode=3&autoAddReplicas=false&name=testSimple3&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=127.0.0.1:61001_solr&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 3206302 INFO (OverseerThreadFactory-11037-thread-3-processing-n:127.0.0.1:61003_solr) [n:127.0.0.1:61003_solr ] o.a.s.c.a.c.CreateCollectionCmd Create collection testSimple3
[junit4] 2> 3206303 INFO (OverseerCollectionConfigSetProcessor-99424178101223431-127.0.0.1:61003_solr-n_0000000000) [n:127.0.0.1:61003_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 3206528 INFO (OverseerStateUpdate-99424178101223431-127.0.0.1:61003_solr-n_0000000000) [n:127.0.0.1:61003_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testSimple3",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"testSimple3_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:61001/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 3206532 INFO (OverseerStateUpdate-99424178101223431-127.0.0.1:61003_solr-n_0000000000) [n:127.0.0.1:61003_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testSimple3",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"testSimple3_shard2_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:61001/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 3206535 INFO (OverseerStateUpdate-99424178101223431-127.0.0.1:61003_solr-n_0000000000) [n:127.0.0.1:61003_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testSimple3",
[junit4] 2> "shard":"shard3",
[junit4] 2> "core":"testSimple3_shard3_replica_n4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:61001/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 3206752 INFO (qtp829746145-27350) [n:127.0.0.1:61001_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node6&collection.configName=conf&newCollection=true&name=testSimple3_shard3_replica_n4&action=CREATE&numShards=3&collection=testSimple3&shard=shard3&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 3206753 INFO (qtp829746145-27349) [n:127.0.0.1:61001_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=testSimple3_shard2_replica_n2&action=CREATE&numShards=3&collection=testSimple3&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 3206753 INFO (qtp829746145-27351) [n:127.0.0.1:61001_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testSimple3_shard1_replica_n1&action=CREATE&numShards=3&collection=testSimple3&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 3206862 INFO (zkCallback-6039-thread-1-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.c.ZkStateReader A cluster state cha
[...truncated too long message...]
059-thread-1-processing-n:127.0.0.1:61001_solr) [n:127.0.0.1:61001_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:61001_solr
[junit4] 2> 3212142 INFO (jetty-closer-6014-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@954bca6{/solr,null,UNAVAILABLE}
[junit4] 2> 3212142 INFO (jetty-closer-6014-thread-2) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 3212147 INFO (coreCloseExecutor-11122-thread-5) [n:127.0.0.1:61001_solr c:testSimple3 s:shard2 r:core_node5 x:testSimple3_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testSimple3.shard2.leader, tag=1082942899
[junit4] 2> 3212149 INFO (jetty-closer-6014-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4ccd6b95{/solr,null,UNAVAILABLE}
[junit4] 2> 3212150 INFO (jetty-closer-6014-thread-1) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 3212151 ERROR (SUITE-AutoAddReplicasPlanActionTest-seed#[76984BF2D9AAB1EF]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 3212151 INFO (SUITE-AutoAddReplicasPlanActionTest-seed#[76984BF2D9AAB1EF]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60999 60999
[junit4] 2> 3212169 INFO (Thread-4909) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60999 60999
[junit4] 2> 3212307 WARN (Thread-4909) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/aliases.json
[junit4] 2> 5 /solr/clusterprops.json
[junit4] 2> 4 /solr/security.json
[junit4] 2> 4 /solr/configs/conf
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 21 /solr/collections/testSimple1/state.json
[junit4] 2> 14 /solr/collections/testSimple2/state.json
[junit4] 2> 10 /solr/collections/testSimple3/state.json
[junit4] 2> 5 /solr/clusterstate.json
[junit4] 2> 3 /solr/overseer_elect/election/99424178101223431-127.0.0.1:61003_solr-n_0000000000
[junit4] 2> 2 /solr/collections/testSimple1/leader_elect/shard2/election/99424178101223431-core_node8-n_0000000000
[junit4] 2> 2 /solr/collections/testSimple1/leader_elect/shard1/election/99424178101223431-core_node5-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/live_nodes
[junit4] 2> 5 /solr/collections
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.AutoAddReplicasPlanActionTest_76984BF2D9AAB1EF-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1350, maxMBSortInHeap=7.478660892857108, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2710ea3e), locale=so, timezone=Europe/London
[junit4] 2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 9 (64-bit)/cpus=3,threads=1,free=294719192,total=527433728
[junit4] 2> NOTE: All tests run in this JVM: [FileBasedSpellCheckerTest, TestDefaultStatsCache, UpdateRequestProcessorFactoryTest, TestComponentsName, TestRandomRequestDistribution, TestUninvertingReader, TestManagedSchemaThreadSafety, TestSegmentSorting, SchemaApiFailureTest, TestReversedWildcardFilterFactory, TestCoreBackup, TestCollectionAPI, SpellCheckComponentTest, CachingDirectoryFactoryTest, DebugComponentTest, HLLUtilTest, TestLegacyTerms, TestFieldCollectionResource, ConnectionManagerTest, SaslZkACLProviderTest, LeaderInitiatedRecoveryOnShardRestartTest, TestNoOpRegenerator, TestDistributedMissingSort, TestSimpleTrackingShardHandler, TestLegacyNumericRangeQueryBuilder, TestNumericRangeQuery32, ExternalFileFieldSortTest, TestManagedSchema, BlockJoinFacetSimpleTest, HdfsWriteToMultipleCollectionsTest, ExitableDirectoryReaderTest, TestFilteredDocIdSet, TestGraphMLResponseWriter, AssignTest, SolrCmdDistributorTest, TestIndexingPerformance, AsyncCallRequestStatusResponseTest, TestRandomDVFaceting, TestSchemaSimilarityResource, TemplateUpdateProcessorTest, OverriddenZkACLAndCredentialsProvidersTest, RegexBoostProcessorTest, TestRTGBase, TestCSVLoader, CdcrRequestHandlerTest, TestCustomSort, BigEndianAscendingWordDeserializerTest, TestManagedSynonymGraphFilterFactory, RuleEngineTest, TestConfigSetsAPI, BlockJoinFacetDistribTest, V2ApiIntegrationTest, QueryParsingTest, SimpleFacetsTest, ActionThrottleTest, TestDocTermOrds, TestJsonRequest, DocValuesTest, TestRequestForwarding, HdfsChaosMonkeyNothingIsSafeTest, AnalyticsMergeStrategyTest, TestValueSourceCache, DistributedSpellCheckComponentTest, TestTriggerIntegration, TestNumericTokenStream, TestPullReplicaErrorHandling, TestConfigSetsAPIZkFailure, TestManagedResourceStorage, TestSolr4Spatial, TestJettySolrRunner, TestCloudSearcherWarming, TestSubQueryTransformerDistrib, TestAtomicUpdateErrorCases, ReplicationFactorTest, TestManagedStopFilterFactory, BooleanFieldTest, TestUnifiedSolrHighlighter, BasicAuthStandaloneTest, TestInPlaceUpdatesDistrib, MoveReplicaHDFSFailoverTest, TestMissingGroups, NodeAddedTriggerTest, TestFieldCacheReopen, ManagedSchemaRoundRobinCloudTest, TestSort, TestRandomFaceting, HdfsNNFailoverTest, TestSubQueryTransformer, TestExactSharedStatsCache, RequestLoggingTest, TestAuthorizationFramework, SolrCoreMetricManagerTest, TestEmbeddedSolrServerConstructors, SystemInfoHandlerTest, CdcrReplicationDistributedZkTest, TestSQLHandlerNonCloud, TestSolrConfigHandlerConcurrent, DocValuesMultiTest, CurrencyFieldTypeTest, DocValuesNotIndexedTest, CleanupOldIndexTest, TestHighFrequencyDictionaryFactory, TestFieldCacheSortRandom, AnalysisErrorHandlingTest, SegmentsInfoRequestHandlerTest, HdfsRecoveryZkTest, TestConfigOverlay, NodeMutatorTest, TestExceedMaxTermLength, LoggingHandlerTest, TestConfig, RankQueryTest, DOMUtilTest, ResponseLogComponentTest, TestPolicyCloud, SolrXmlInZkTest, TestSmileRequest, TestDownShardTolerantSearch, UUIDUpdateProcessorFallbackTest, TestStressVersions, TestStressInPlaceUpdates, TestDocumentBuilder, EchoParamsTest, FullSolrCloudDistribCmdsTest, HdfsDirectoryTest, FieldAnalysisRequestHandlerTest, TestNRTOpen, DistribDocExpirationUpdateProcessorTest, TestSolrCloudWithSecureImpersonation, TestSolrCoreProperties, SolrCoreCheckLockOnStartupTest, HdfsTlogReplayBufferedWhileIndexingTest, TestLeaderInitiatedRecoveryThread, TestHdfsCloudBackupRestore, DistributedExpandComponentTest, TestCoreDiscovery, TestDynamicLoading, UUIDFieldTest, BasicDistributedZk2Test, ConjunctionSolrSpellCheckerTest, SpellCheckCollatorTest, CreateRoutedAliasTest, TlogReplayBufferedWhileIndexingTest, TestLeaderElectionWithEmptyReplica, TestDelegationWithHadoopAuth, FacetPivotSmallTest, BadIndexSchemaTest, TestHashQParserPlugin, DistributedIntervalFacetingTest, OverseerTest, DistributedTermsComponentTest, TestWordDelimiterFilterFactory, TestSerializedLuceneMatchVersion, ClusterStateTest, TestSolrConfigHandler, TestReplicaProperties, SpellPossibilityIteratorTest, TestClusterStateProvider, UpdateLogTest, HttpTriggerListenerTest, BlobRepositoryCloudTest, PeerSyncTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TimeZoneUtilsTest, TestLRUStatsCache, OverseerStatusTest, TestSchemaManager, TestTrieFacet, TestDynamicFieldCollectionResource, TestBinaryResponseWriter, TestLuceneMatchVersion, AddSchemaFieldsUpdateProcessorFactoryTest, TestPushWriter, TestPolicyCloud, LeaderFailureAfterFreshStartTest, SpatialRPTFieldTypeTest, TestCloudPseudoReturnFields, TestFastWriter, ChaosMonkeySafeLeaderWithPullReplicasTest, TestDFISimilarityFactory, TestConfigSetProperties, DocValuesMissingTest, TestObjectReleaseTracker, TestAddFieldRealTimeGet, ZkStateWriterTest, CollectionTooManyReplicasTest, TestClusterStateMutator, TestQueryWrapperFilter, PropertiesRequestHandlerTest, ShardRoutingTest, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, SyncSliceTest, BasicZkTest, ClusterStateUpdateTest, ZkSolrClientTest, TestDistributedGrouping, TestFaceting, TestRecovery, TermVectorComponentDistributedTest, TestRealTimeGet, SolrCoreTest, TestGroupingSearch, QueryElevationComponentTest, BasicFunctionalityTest, TestBadConfig, SoftAutoCommitTest, MinimalSchemaTest, TestPostingsSolrHighlighter, TestCodecSupport, TestBM25SimilarityFactory, ScriptEngineTest, URLClassifyProcessorTest, DateMathParserTest, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, TestSuggestSpellingConverter, SpellingQueryConverterTest, RAMDirectoryFactoryTest, TestSolrJ, TestLRUCache, TestUtils, ZkNodePropsTest, SliceStateTest, FileUtilsTest, CircularListTest, DistributedMLTComponentTest, SolrTestCaseJ4Test, TestCrossCoreJoin, TestLuceneIndexBackCompat, ConnectionReuseTest, AliasIntegrationTest, CloudExitableDirectoryReaderTest, CollectionsAPISolrJTest, DeleteShardTest, DistribCursorPagingTest, HealthCheckHandlerTest, HttpPartitionTest, MigrateRouteKeyTest, MissingSegmentRecoveryTest, MoveReplicaHDFSTest, MoveReplicaTest, PeerSyncReplicationTest, SharedFSAutoReplicaFailoverTest, SolrCLIZkUtilsTest, TestAuthenticationFramework, TestCloudRecovery, TestDeleteCollectionOnDownNodes, TestExclusionRuleCollectionAccess, TestMiniSolrCloudClusterSSL, TestRandomFlRTGCloud, TestSSLRandomization, TestSizeLimitedDistributedMap, TestStressCloudBlindAtomicUpdates, TestTolerantUpdateProcessorRandomCloud, TriLevelCompositeIdRoutingTest, VMParamsZkACLAndCredentialsProvidersTest, CollectionReloadTest, CollectionsAPIDistributedZkTest, SimpleCollectionCreateDeleteTest, TestRequestStatusCollectionAPI, AutoAddReplicasIntegrationTest, AutoAddReplicasPlanActionTest]
[junit4] Completed [581/770 (1!)] on J1 in 13.75s, 1 test, 1 error <<< FAILURES!
[...truncated 41420 lines...]
[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-9) - Build # 4411 -
Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4411/
Java: 64bit/jdk-9 -XX:+UseCompressedOops -XX:+UseParallelGC
4 tests failed.
FAILED: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test
Error Message:
Could not find collection:collection2
Stack Trace:
java.lang.AssertionError: Could not find collection:collection2
at __randomizedtesting.SeedInfo.seed([2B150420ABF229A6:A3413BFA050E445E]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:140)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:135)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:915)
at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:612)
at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple
Error Message:
IOException occured when talking to server at: http://127.0.0.1:53621/solr
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://127.0.0.1:53621/solr
at __randomizedtesting.SeedInfo.seed([2B150420ABF229A6:13A620DE8C01FD77]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple(AutoAddReplicasPlanActionTest.java:110)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.net.SocketException: Connection reset
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:141)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:542)
... 47 more
FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testListeners
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([2B150420ABF229A6:C6D59330169EB47F]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testListeners(TriggerIntegrationTest.java:1053)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([2B150420ABF229A6:911933AFF41AFFE9]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNull(Assert.java:551)
at org.junit.Assert.assertNull(Assert.java:562)
at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger(TriggerIntegrationTest.java:1575)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
Build Log:
[...truncated 12572 lines...]
[junit4] Suite: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest
[junit4] 2> 1528557 INFO (SUITE-FullSolrCloudDistribCmdsTest-seed#[2B150420ABF229A6]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/init-core-data-001
[junit4] 2> 1528558 WARN (SUITE-FullSolrCloudDistribCmdsTest-seed#[2B150420ABF229A6]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
[junit4] 2> 1528558 INFO (SUITE-FullSolrCloudDistribCmdsTest-seed#[2B150420ABF229A6]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 1528558 INFO (SUITE-FullSolrCloudDistribCmdsTest-seed#[2B150420ABF229A6]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl="https://issues.apache.org/jira/browse/SOLR-5776") w/ MAC_OS_X supressed clientAuth
[junit4] 2> 1528558 INFO (SUITE-FullSolrCloudDistribCmdsTest-seed#[2B150420ABF229A6]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /_ta/
[junit4] 2> 1528559 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1528560 INFO (Thread-1669) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1528560 INFO (Thread-1669) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1528562 ERROR (Thread-1669) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1528666 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.ZkTestServer start zk server on port:61747
[junit4] 2> 1528676 INFO (zkConnectionManagerCallback-1987-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1528680 INFO (zkConnectionManagerCallback-1989-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1528688 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1528691 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 1528693 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1528695 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1528697 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1528699 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1528701 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1528703 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1528705 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1528707 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1528709 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1528712 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
[junit4] 2> 1528992 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T11:27:37+14:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 1528993 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1528993 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1528993 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 1528993 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@12b4d29e{/_ta,null,AVAILABLE}
[junit4] 2> 1528993 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@1e4cb569{HTTP/1.1,[http/1.1]}{127.0.0.1:61753}
[junit4] 2> 1528993 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.Server Started @1530976ms
[junit4] 2> 1528993 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/_ta, solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/tempDir-001/control/data, hostPort=61753, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/control-001/cores}
[junit4] 2> 1528993 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1528993 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 1528993 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1528993 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1528993 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-27T11:09:00.092661Z
[junit4] 2> 1528998 INFO (zkConnectionManagerCallback-1991-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1528999 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1528999 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/control-001/solr.xml
[junit4] 2> 1529000 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 1529001 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1529001 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 1529005 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61747/solr
[junit4] 2> 1529006 INFO (zkConnectionManagerCallback-1995-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1529010 INFO (zkConnectionManagerCallback-1997-thread-1-processing-n:127.0.0.1:61753__ta) [n:127.0.0.1:61753__ta ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1529164 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61753__ta ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1529165 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61753__ta ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:61753__ta
[junit4] 2> 1529166 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61753__ta ] o.a.s.c.Overseer Overseer (id=99421476596023300-127.0.0.1:61753__ta-n_0000000000) starting
[junit4] 2> 1529175 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61753__ta ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61753__ta
[junit4] 2> 1529177 INFO (zkCallback-1996-thread-1-processing-n:127.0.0.1:61753__ta) [n:127.0.0.1:61753__ta ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1529244 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61753__ta ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06
[junit4] 2> 1529253 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61753__ta ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06
[junit4] 2> 1529253 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61753__ta ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06
[junit4] 2> 1529255 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61753__ta ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/control-001/cores
[junit4] 2> 1529269 INFO (zkConnectionManagerCallback-2003-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1529270 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1529272 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:61747/solr ready
[junit4] 2> 1529274 INFO (qtp1696281420-11001) [n:127.0.0.1:61753__ta ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:61753__ta&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1529277 INFO (OverseerThreadFactory-3973-thread-1-processing-n:127.0.0.1:61753__ta) [n:127.0.0.1:61753__ta ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
[junit4] 2> 1529386 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1529387 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 1529493 INFO (zkCallback-1996-thread-1-processing-n:127.0.0.1:61753__ta) [n:127.0.0.1:61753__ta ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1530405 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1530412 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
[junit4] 2> 1530464 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1530483 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
[junit4] 2> 1530484 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06
[junit4] 2> 1530484 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1530484 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/control-001/cores/control_collection_shard1_replica_n1/data/]
[junit4] 2> 1530486 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=44.90625, floorSegmentMB=2.1748046875, forceMergeDeletesPctAllowed=17.118262385358864, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.41266757696283385
[junit4] 2> 1530488 WARN (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1530521 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1530521 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1530522 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1530522 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1530523 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=96.369140625, floorSegmentMB=2.1884765625, forceMergeDeletesPctAllowed=29.25245235511709, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.24394702053507686
[junit4] 2> 1530523 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@69a43077[control_collection_shard1_replica_n1] main]
[junit4] 2> 1530524 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1530525 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1530525 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1530526 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590743627595776000
[junit4] 2> 1530526 INFO (searcherExecutor-3976-thread-1-processing-n:127.0.0.1:61753__ta x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@69a43077[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1530531 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1530531 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1530531 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:61753/_ta/control_collection_shard1_replica_n1/
[junit4] 2> 1530531 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1530531 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:61753/_ta/control_collection_shard1_replica_n1/ has no replicas
[junit4] 2> 1530531 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1530536 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:61753/_ta/control_collection_shard1_replica_n1/ shard1
[junit4] 2> 1530643 INFO (zkCallback-1996-thread-1-processing-n:127.0.0.1:61753__ta) [n:127.0.0.1:61753__ta ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1530644 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1530646 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1259
[junit4] 2> 1530649 INFO (qtp1696281420-11001) [n:127.0.0.1:61753__ta ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 1530755 INFO (zkCallback-1996-thread-1-processing-n:127.0.0.1:61753__ta) [n:127.0.0.1:61753__ta ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1531285 INFO (OverseerCollectionConfigSetProcessor-99421476596023300-127.0.0.1:61753__ta-n_0000000000) [n:127.0.0.1:61753__ta ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1531651 INFO (qtp1696281420-11001) [n:127.0.0.1:61753__ta ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:61753__ta&wt=javabin&version=2} status=0 QTime=2377
[junit4] 2> 1531655 INFO (zkConnectionManagerCallback-2007-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1531656 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1531658 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:61747/solr ready
[junit4] 2> 1531658 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1531659 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1531662 INFO (OverseerThreadFactory-3973-thread-2-processing-n:127.0.0.1:61753__ta) [n:127.0.0.1:61753__ta ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 1531663 WARN (OverseerThreadFactory-3973-thread-2-processing-n:127.0.0.1:61753__ta) [n:127.0.0.1:61753__ta ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
[junit4] 2> 1531878 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 1531878 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=3&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=219
[junit4] 2> 1532078 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-1-001 of type NRT
[junit4] 2> 1532078 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T11:27:37+14:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 1532080 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1532080 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1532080 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 1532080 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@69f78a31{/_ta,null,AVAILABLE}
[junit4] 2> 1532080 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@16faa8aa{HTTP/1.1,[http/1.1]}{127.0.0.1:61773}
[junit4] 2> 1532080 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.Server Started @1534063ms
[junit4] 2> 1532080 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/_ta, solrconfig=solrconfig.xml, solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/tempDir-001/jetty1, hostPort=61773, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-1-001/cores, replicaType=NRT}
[junit4] 2> 1532080 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1532081 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 1532081 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1532081 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1532081 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-27T11:09:03.180187Z
[junit4] 2> 1532082 INFO (zkConnectionManagerCallback-2009-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1532083 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1532083 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-1-001/solr.xml
[junit4] 2> 1532085 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 1532085 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1532092 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 1532094 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61747/solr
[junit4] 2> 1532095 INFO (zkConnectionManagerCallback-2013-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1532098 INFO (zkConnectionManagerCallback-2015-thread-1-processing-n:127.0.0.1:61773__ta) [n:127.0.0.1:61773__ta ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1532107 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61773__ta ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1532110 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61773__ta ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1532114 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61773__ta ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61773__ta
[junit4] 2> 1532115 INFO (zkCallback-1996-thread-1-processing-n:127.0.0.1:61753__ta) [n:127.0.0.1:61753__ta ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1532115 INFO (zkCallback-2006-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1532116 INFO (zkCallback-2014-thread-1-processing-n:127.0.0.1:61773__ta) [n:127.0.0.1:61773__ta ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1532146 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61773__ta ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06
[junit4] 2> 1532156 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61773__ta ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06
[junit4] 2> 1532156 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61773__ta ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06
[junit4] 2> 1532158 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61773__ta ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-1-001/cores
[junit4] 2> 1532179 INFO (qtp1696281420-11003) [n:127.0.0.1:61753__ta ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:61773__ta&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1532182 INFO (OverseerCollectionConfigSetProcessor-99421476596023300-127.0.0.1:61753__ta-n_0000000000) [n:127.0.0.1:61753__ta ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1532183 INFO (OverseerThreadFactory-3973-thread-3-processing-n:127.0.0.1:61753__ta) [n:127.0.0.1:61753__ta ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:61773__ta for creating new replica
[junit4] 2> 1532187 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n61&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1532188 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 1533210 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1533220 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.s.IndexSchema [collection1_shard2_replica_n61] Schema name=test
[junit4] 2> 1533283 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1533305 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n61' using configuration from collection collection1, trusted=true
[junit4] 2> 1533306 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n61' (registry 'solr.core.collection1.shard2.replica_n61') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06
[junit4] 2> 1533306 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1533306 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.SolrCore [[collection1_shard2_replica_n61] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-1-001/cores/collection1_shard2_replica_n61], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-1-001/cores/collection1_shard2_replica_n61/data/]
[junit4] 2> 1533308 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=44.90625, floorSegmentMB=2.1748046875, forceMergeDeletesPctAllowed=17.118262385358864, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.41266757696283385
[junit4] 2> 1533310 WARN (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1533351 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1533351 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1533353 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1533353 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1533354 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=96.369140625, floorSegmentMB=2.1884765625, forceMergeDeletesPctAllowed=29.25245235511709, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.24394702053507686
[junit4] 2> 1533354 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.s.SolrIndexSearcher Opening [Searcher@ee54b57[collection1_shard2_replica_n61] main]
[junit4] 2> 1533355 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1533356 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1533357 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1533357 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590743630564294656
[junit4] 2> 1533357 INFO (searcherExecutor-3987-thread-1-processing-n:127.0.0.1:61773__ta x:collection1_shard2_replica_n61 s:shard2 c:collection1) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.SolrCore [collection1_shard2_replica_n61] Registered new searcher Searcher@ee54b57[collection1_shard2_replica_n61] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1533363 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1533363 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1533363 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:61773/_ta/collection1_shard2_replica_n61/
[junit4] 2> 1533363 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1533363 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.SyncStrategy http://127.0.0.1:61773/_ta/collection1_shard2_replica_n61/ has no replicas
[junit4] 2> 1533363 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1533368 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:61773/_ta/collection1_shard2_replica_n61/ shard2
[junit4] 2> 1533523 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1533525 INFO (qtp953194595-11050) [n:127.0.0.1:61773__ta c:collection1 s:shard2 x:collection1_shard2_replica_n61] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n61&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1338
[junit4] 2> 1533528 INFO (qtp1696281420-11003) [n:127.0.0.1:61753__ta ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:61773__ta&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1349
[junit4] 2> 1533792 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-2-001 of type NRT
[junit4] 2> 1533792 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T11:27:37+14:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 1533794 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1533794 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1533794 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 1533794 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@562dc4a6{/_ta,null,AVAILABLE}
[junit4] 2> 1533794 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@63ea17e3{HTTP/1.1,[http/1.1]}{127.0.0.1:61784}
[junit4] 2> 1533794 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.Server Started @1535777ms
[junit4] 2> 1533794 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/_ta, solrconfig=solrconfig.xml, solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/tempDir-001/jetty2, hostPort=61784, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-2-001/cores, replicaType=NRT}
[junit4] 2> 1533795 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1533795 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 1533795 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1533795 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1533795 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-27T11:09:04.894359Z
[junit4] 2> 1533797 INFO (zkConnectionManagerCallback-2019-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1533799 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1533799 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-2-001/solr.xml
[junit4] 2> 1533802 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 1533802 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1533802 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 1533805 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61747/solr
[junit4] 2> 1533806 INFO (zkConnectionManagerCallback-2023-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1533809 INFO (zkConnectionManagerCallback-2025-thread-1-processing-n:127.0.0.1:61784__ta) [n:127.0.0.1:61784__ta ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1533821 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61784__ta ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1533825 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61784__ta ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1533828 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61784__ta ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61784__ta
[junit4] 2> 1533831 INFO (zkCallback-1996-thread-2-processing-n:127.0.0.1:61753__ta) [n:127.0.0.1:61753__ta ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1533831 INFO (zkCallback-2014-thread-1-processing-n:127.0.0.1:61773__ta) [n:127.0.0.1:61773__ta ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1533831 INFO (zkCallback-2006-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1533832 INFO (zkCallback-2024-thread-1-processing-n:127.0.0.1:61784__ta) [n:127.0.0.1:61784__ta ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1533878 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61784__ta ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06
[junit4] 2> 1533890 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61784__ta ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06
[junit4] 2> 1533891 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61784__ta ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06
[junit4] 2> 1533892 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61784__ta ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-2-001/cores
[junit4] 2> 1533924 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:61784__ta&action=ADDREPLICA&collection=collection1&shard=shard3&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1533930 INFO (OverseerCollectionConfigSetProcessor-99421476596023300-127.0.0.1:61753__ta-n_0000000000) [n:127.0.0.1:61753__ta ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1533930 INFO (OverseerThreadFactory-3973-thread-4-processing-n:127.0.0.1:61753__ta) [n:127.0.0.1:61753__ta ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:61784__ta for creating new replica
[junit4] 2> 1533937 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_n63&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1533938 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 1534990 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1535035 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.s.IndexSchema [collection1_shard3_replica_n63] Schema name=test
[junit4] 2> 1535201 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1535228 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard3_replica_n63' using configuration from collection collection1, trusted=true
[junit4] 2> 1535230 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard3.replica_n63' (registry 'solr.core.collection1.shard3.replica_n63') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06
[junit4] 2> 1535230 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1535230 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.SolrCore [[collection1_shard3_replica_n63] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-2-001/cores/collection1_shard3_replica_n63], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-2-001/cores/collection1_shard3_replica_n63/data/]
[junit4] 2> 1535234 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=17, maxMergeAtOnceExplicit=46, maxMergedSegmentMB=44.90625, floorSegmentMB=2.1748046875, forceMergeDeletesPctAllowed=17.118262385358864, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.41266757696283385
[junit4] 2> 1535241 WARN (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1535282 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1535282 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1535284 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1535284 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1535285 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=96.369140625, floorSegmentMB=2.1884765625, forceMergeDeletesPctAllowed=29.25245235511709, segmentsPerTier=42.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.24394702053507686
[junit4] 2> 1535286 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.s.SolrIndexSearcher Opening [Searcher@5a046d89[collection1_shard3_replica_n63] main]
[junit4] 2> 1535288 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1535289 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1535289 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1535290 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590743632591192064
[junit4] 2> 1535291 INFO (searcherExecutor-3998-thread-1-processing-n:127.0.0.1:61784__ta x:collection1_shard3_replica_n63 s:shard3 c:collection1) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.SolrCore [collection1_shard3_replica_n63] Registered new searcher Searcher@5a046d89[collection1_shard3_replica_n63] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1535296 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1535296 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1535296 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:61784/_ta/collection1_shard3_replica_n63/
[junit4] 2> 1535296 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1535296 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.SyncStrategy http://127.0.0.1:61784/_ta/collection1_shard3_replica_n63/ has no replicas
[junit4] 2> 1535296 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1535301 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:61784/_ta/collection1_shard3_replica_n63/ shard3
[junit4] 2> 1535414 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1535417 INFO (qtp1424787760-11081) [n:127.0.0.1:61784__ta c:collection1 s:shard3 x:collection1_shard3_replica_n63] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard3_replica_n63&action=CREATE&collection=collection1&shard=shard3&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1479
[junit4] 2> 1535420 INFO (qtp1696281420-11005) [n:127.0.0.1:61753__ta ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:61784__ta&action=ADDREPLICA&collection=collection1&shard=shard3&type=NRT&wt=javabin&version=2} status=0 QTime=1496
[junit4] 2> 1535661 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-3-001 of type NRT
[junit4] 2> 1535662 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T11:27:37+14:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 1535663 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1535663 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1535663 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 1535663 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@24a7c81d{/_ta,null,AVAILABLE}
[junit4] 2> 1535664 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4a477614{HTTP/1.1,[http/1.1]}{127.0.0.1:61796}
[junit4] 2> 1535665 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.e.j.s.Server Started @1537648ms
[junit4] 2> 1535665 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/_ta, solrconfig=solrconfig.xml, solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/tempDir-001/jetty3, hostPort=61796, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-3-001/cores, replicaType=NRT}
[junit4] 2> 1535665 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1535665 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 1535665 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1535665 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1535665 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-27T11:09:06.764666Z
[junit4] 2> 1535667 INFO (zkConnectionManagerCallback-2029-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1535668 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1535669 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-3-001/solr.xml
[junit4] 2> 1535670 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 1535670 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1535671 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 1535674 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:61747/solr
[junit4] 2> 1535675 INFO (zkConnectionManagerCallback-2033-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1535677 INFO (zkConnectionManagerCallback-2035-thread-1-processing-n:127.0.0.1:61796__ta) [n:127.0.0.1:61796__ta ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1535687 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61796__ta ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 1535690 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61796__ta ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1535693 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61796__ta ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:61796__ta
[junit4] 2> 1535695 INFO (zkCallback-1996-thread-1-processing-n:127.0.0.1:61753__ta) [n:127.0.0.1:61753__ta ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1535695 INFO (zkCallback-2014-thread-1-processing-n:127.0.0.1:61773__ta) [n:127.0.0.1:61773__ta ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1535696 INFO (zkCallback-2006-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1535696 INFO (zkCallback-2024-thread-1-processing-n:127.0.0.1:61784__ta) [n:127.0.0.1:61784__ta ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1535696 INFO (zkCallback-2034-thread-1-processing-n:127.0.0.1:61796__ta) [n:127.0.0.1:61796__ta ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1535732 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61796__ta ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06
[junit4] 2> 1535741 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61796__ta ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06
[junit4] 2> 1535741 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61796__ta ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@cb64a06
[junit4] 2> 1535743 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[2B150420ABF229A6]) [n:127.0.0.1:61796__ta ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_2B150420ABF229A6-001/shard-3-001/cores
[junit4] 2> 1535783 INFO (qtp1696281420-11003) [n:127.0.0.1:61753__ta ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:61796__ta&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1535786 INFO (OverseerCollectionConfigSetProcessor-99421476596023300-127.0.0.1:61753__ta-n_0000000000) [n:127.0.0.1:61753__ta ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1535786 INFO (OverseerThreadFactory-3973-thread-5-processing-n:127.0.0.1:61753__ta) [n:127.0.0.1:61753__ta ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:61796__ta for creating new replica
[junit4] 2> 1535790 INFO (qtp2120368932-11112) [n:127.0.0.1:61796__ta ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n65&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1535791 INFO (qtp2120368932-11112) [n:127.0.0.1:61796__ta ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 1536813 INFO (qtp2120368932-11112) [n:127.0.0.1:6179
[...truncated too long message...]
[junit4] 2>
[junit4] 2> 149 /solr/collections/collection1/state.json
[junit4] 2> 21 /solr/clusterstate.json
[junit4] 2> 13 /solr/collections/testMetricTrigger/state.json
[junit4] 2> 6 /solr/autoscaling.json
[junit4] 2> 5 /solr/overseer_elect/election/99421602739912739-127.0.0.1:54033_solr-n_0000000010
[junit4] 2> 3 /solr/overseer_elect/election/99421602739912733-127.0.0.1:54025_solr-n_0000000008
[junit4] 2> 3 /solr/collections/collection1/leader_elect/shard1/election/99421602739912724-core_node6-n_0000000002
[junit4] 2> 2 /solr/overseer_elect/election/99421602739912751-127.0.0.1:54308_solr-n_0000000014
[junit4] 2> 2 /solr/collections/collection1/leader_elect/shard1/election/99421602739912730-core_node3-n_0000000000
[junit4] 2> 2 /solr/collections/collection1/leader_elect/shard1/election/99421602739912733-core_node10-n_0000000004
[junit4] 2> 2 /solr/overseer_elect/election/99421602739912727-127.0.0.1:53965_solr-n_0000000006
[junit4] 2> 2 /solr/overseer_elect/election/99421602739912730-127.0.0.1:54003_solr-n_0000000007
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 21 /solr/live_nodes
[junit4] 2> 21 /solr/collections
[junit4] 2> 6 /solr/overseer/queue-work
[junit4] 2> 6 /solr/autoscaling/events/.auto_add_replicas
[junit4] 2> 6 /solr/overseer/queue
[junit4] 2> 6 /solr/overseer/collection-queue-work
[junit4] 2> 2 /solr/autoscaling/events/node_lost_triggerMR
[junit4] 2> 2 /solr/autoscaling/events/node_added_triggerCTOOR
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_2B150420ABF229A6-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@13ac42b1), locale=mas-TZ, timezone=Asia/Novosibirsk
[junit4] 2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 9 (64-bit)/cpus=3,threads=1,free=229951592,total=514850816
[junit4] 2> NOTE: All tests run in this JVM: [ClusterStateUpdateTest, HdfsDirectoryTest, TestSchemaNameResource, TestRandomDVFaceting, TestUninvertingReader, ClassificationUpdateProcessorIntegrationTest, TestLRUCache, TestHashPartitioner, TestDownShardTolerantSearch, SpellCheckCollatorTest, DeleteStatusTest, SortByFunctionTest, TestManagedResourceStorage, ClusterStateTest, HttpTriggerListenerTest, TestAuthorizationFramework, RollingRestartTest, CdcrReplicationDistributedZkTest, TestEmbeddedSolrServerConstructors, SegmentsInfoRequestHandlerTest, TestCodecSupport, TestNumericRangeQuery32, UUIDFieldTest, JvmMetricsTest, RequestLoggingTest, FileBasedSpellCheckerTest, JsonLoaderTest, TestRecoveryHdfs, ReturnFieldsTest, TestFieldCacheSortRandom, ReplicaListTransformerTest, UpdateRequestProcessorFactoryTest, TestPerFieldSimilarityWithDefaultOverride, CSVRequestHandlerTest, TestFastWriter, CachingDirectoryFactoryTest, TestSolrDeletionPolicy2, PeerSyncReplicationTest, TestDistributedMissingSort, TestHdfsCloudBackupRestore, TestWordDelimiterFilterFactory, SubstringBytesRefFilterTest, BadComponentTest, QueryParsingTest, TestBlobHandler, AnalysisErrorHandlingTest, BlockJoinFacetSimpleTest, JavabinLoaderTest, AssignTest, DirectUpdateHandlerTest, CollectionTooManyReplicasTest, UninvertDocValuesMergePolicyTest, TestInPlaceUpdatesDistrib, SolrSlf4jReporterTest, ShardRoutingTest, CurrencyRangeFacetCloudTest, TestQueryWrapperFilter, TestCloudSearcherWarming, NodeAddedTriggerTest, OverseerStatusTest, ManagedSchemaRoundRobinCloudTest, TestTlogReplica, TestDeleteCollectionOnDownNodes, TestAnalyzedSuggestions, TestClusterStateProvider, TestSmileRequest, TestSort, TestConfigSetsAPIZkFailure, TestManagedSchemaThreadSafety, TestClusterStateMutator, DistribDocExpirationUpdateProcessorTest, ShardSplitTest, TestSolrQueryResponse, RegexBoostProcessorTest, TemplateUpdateProcessorTest, ShowFileRequestHandlerTest, TestPolicyCloud, DistributedQueueTest, IndexSchemaTest, TestHighFrequencyDictionaryFactory, SolrCoreCheckLockOnStartupTest, TestLRUStatsCache, TestSolrCloudWithDelegationTokens, PreAnalyzedUpdateProcessorTest, TestStressReorder, TestComponentsName, TestObjectReleaseTracker, TestLegacyTerms, TestUseDocValuesAsStored, DebugComponentTest, SolrJmxReporterCloudTest, TestJsonFacetRefinement, TestPivotHelperCode, TestCSVLoader, TestPHPSerializedResponseWriter, TestStressLucene, TestReplicaProperties, BooleanFieldTest, SimpleMLTQParserTest, HdfsChaosMonkeyNothingIsSafeTest, DocValuesMultiTest, TestQuerySenderListener, FullSolrCloudDistribCmdsTest, TestMissingGroups, RequestHandlersTest, DisMaxRequestHandlerTest, ExternalFileFieldSortTest, SimpleFacetsTest, PropertiesRequestHandlerTest, BlobRepositoryCloudTest, TestTriggerIntegration, TestCloudManagedSchema, TestLMDirichletSimilarityFactory, TestSolrCLIRunExample, MinimalSchemaTest, SliceStateTest, TestSolrCloudSnapshots, TestCorePropertiesReload, TestNumericTokenStream, TestNumericTerms32, ExecutePlanActionTest, TestValueSourceCache, HealthCheckHandlerTest, TestSolrQueryParser, SimplePostToolTest, CoreAdminHandlerTest, TestUnifiedSolrHighlighter, MoveReplicaHDFSFailoverTest, HLLUtilTest, AnalyticsMergeStrategyTest, XsltUpdateRequestHandlerTest, AutoscalingHistoryHandlerTest, DistributedQueryComponentOptimizationTest, QueryResultKeyTest, LeaderElectionContextKeyTest, CleanupOldIndexTest, UpdateParamsTest, TestShardHandlerFactory, TestDistribStateManager, TestLuceneMatchVersion, TestSolrIndexConfig, TestReplicationHandlerBackup, TestConfigOverlay, TestSimpleTrackingShardHandler, TestStandardQParsers, TestFieldCacheReopen, TestPayloadCheckQParserPlugin, OverseerModifyCollectionTest, URLClassifyProcessorTest, ZkStateWriterTest, BigEndianAscendingWordDeserializerTest, TestSolrConfigHandler, SaslZkACLProviderTest, TestDFRSimilarityFactory, AtomicUpdatesTest, TestSizeLimitedDistributedMap, TestSerializedLuceneMatchVersion, TestGenericDistributedQueue, SyncSliceTest, TestMacroExpander, TestSubQueryTransformer, IndexBasedSpellCheckerTest, CoreAdminRequestStatusTest, NodeMutatorTest, ChaosMonkeySafeLeaderTest, TestDynamicFieldCollectionResource, PrimUtilsTest, AddSchemaFieldsUpdateProcessorFactoryTest, CurrencyFieldTypeTest, TestExceedMaxTermLength, CheckHdfsIndexTest, CustomHighlightComponentTest, TestCollationFieldDocValues, TestDelegationWithHadoopAuth, TestLegacyFieldReuse, TestSearchPerf, PeerSyncTest, TestDocTermOrds, TestMinMaxOnMultiValuedField, TimeRoutedAliasUpdateProcessorTest, SystemInfoHandlerTest, TestFileDictionaryLookup, InfoHandlerTest, SolrXmlInZkTest, BitVectorTest, RuleEngineTest, TestJsonRequest, TestCloudPseudoReturnFields, MigrateRouteKeyTest, TestDistributedGrouping, ZkControllerTest, TestStressVersions, DistributedTermsComponentTest, StatsComponentTest, QueryElevationComponentTest, BasicFunctionalityTest, TestLazyCores, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, TestExtendedDismaxParser, SolrRequestParserTest, TestFoldingMultitermQuery, DocValuesTest, SuggesterTSTTest, SuggesterTest, SpatialFilterTest, SuggesterWFSTTest, PolyFieldTest, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, TestPseudoReturnFields, TestUpdate, FieldMutatingUpdateProcessorTest, TestAtomicUpdateErrorCases, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, TestRemoteStreaming, DistanceFunctionTest, SolrInfoBeanTest, TestQueryUtils, TestWriterPerf, DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, DocumentAnalysisRequestHandlerTest, PrimitiveFieldTypeTest, TermsComponentTest, XmlUpdateRequestHandlerTest, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, TermVectorComponentTest, TestIndexingPerformance, MoreLikeThisHandlerTest, RequiredFieldsTest, FieldAnalysisRequestHandlerTest, FastVectorHighlighterTest, LoggingHandlerTest, TestCSVResponseWriter, PingRequestHandlerTest, HighlighterConfigTest, AlternateDirectoryTest, TestQuerySenderNoQuery, ResponseLogComponentTest, TestStressRecovery, TestMergePolicyConfig, TestDocSet, TestBinaryField, TestElisionMultitermQuery, TestConfig, OutputWriterTest, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties, NotRequiredUniqueKeyTest, SpellPossibilityIteratorTest, EchoParamsTest, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TimeZoneUtilsTest, ResourceLoaderTest, ScriptEngineTest, OpenExchangeRatesOrgProviderTest, TestFastLRUCache, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, AssignBackwardCompatibilityTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, CloudExitableDirectoryReaderTest, CollectionStateFormat2Test, ConcurrentCreateRoutedAliasTest, ConfigSetsAPITest, ConnectionManagerTest, CreateRoutedAliasTest, DeleteLastCustomShardedReplicaTest, DeleteReplicaTest, DeleteShardTest, DistribCursorPagingTest, DistribJoinFromCollectionTest, DistributedVersionInfoTest, HttpPartitionTest, LeaderInitiatedRecoveryOnCommitTest, MultiThreadedOCPTest, TestCloudDeleteByQuery, TestCloudJSONFacetJoinDomain, TestDistribDocBasedVersion, TestExclusionRuleCollectionAccess, TestLeaderElectionWithEmptyReplica, TestLeaderInitiatedRecoveryThread, TestOnReconnectListenerSupport, TestPullReplicaErrorHandling, TestRandomRequestDistribution, TestShortCircuitedRequests, TestSolrCloudWithKerberosAlt, TestSolrCloudWithSecureImpersonation, TestStressInPlaceUpdates, CollectionReloadTest, CollectionsAPIAsyncDistributedZkTest, ConcurrentDeleteAndCreateCollectionTest, CustomCollectionTest, HdfsCollectionsAPIDistributedZkTest, SimpleCollectionCreateDeleteTest, TestCollectionAPI, TestRequestStatusCollectionAPI, AutoAddReplicasIntegrationTest, SystemLogListenerTest, TriggerIntegrationTest]
[junit4] Completed [621/770 (3!)] on J1 in 190.62s, 14 tests, 2 failures <<< FAILURES!
[...truncated 41308 lines...]
[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4410
- Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4410/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC
2 tests failed.
FAILED: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test
Error Message:
shard2 is not consistent. Got 18 from http://127.0.0.1:65392/collection1_shard2_replica_n41 (previous client) and got 15 from http://127.0.0.1:65402/collection1_shard2_replica_n45
Stack Trace:
java.lang.AssertionError: shard2 is not consistent. Got 18 from http://127.0.0.1:65392/collection1_shard2_replica_n41 (previous client) and got 15 from http://127.0.0.1:65402/collection1_shard2_replica_n45
at __randomizedtesting.SeedInfo.seed([2055D7445EB42AE:8A5162AEEB172F56]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1330)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1309)
at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test(ChaosMonkeySafeLeaderTest.java:162)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeLostTrigger
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([2055D7445EB42AE:6FBF4667F21C034]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeLostTrigger(TriggerIntegrationTest.java:521)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12271 lines...]
[junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4] 2> 1340893 INFO (SUITE-ChaosMonkeySafeLeaderTest-seed#[2055D7445EB42AE]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/init-core-data-001
[junit4] 2> 1340894 WARN (SUITE-ChaosMonkeySafeLeaderTest-seed#[2055D7445EB42AE]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=9 numCloses=9
[junit4] 2> 1340894 INFO (SUITE-ChaosMonkeySafeLeaderTest-seed#[2055D7445EB42AE]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 1340897 INFO (SUITE-ChaosMonkeySafeLeaderTest-seed#[2055D7445EB42AE]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
[junit4] 2> 1340897 INFO (SUITE-ChaosMonkeySafeLeaderTest-seed#[2055D7445EB42AE]-worker) [ ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
[junit4] 2> 1340900 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1340900 INFO (Thread-2568) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1340900 INFO (Thread-2568) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1340906 ERROR (Thread-2568) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1341004 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.ZkTestServer start zk server on port:65380
[junit4] 2> 1341092 INFO (zkConnectionManagerCallback-2916-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1341095 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x161350079680000, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1341100 INFO (zkConnectionManagerCallback-2918-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1341107 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4] 2> 1341110 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
[junit4] 2> 1341112 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
[junit4] 2> 1341114 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4] 2> 1341115 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4] 2> 1341117 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4] 2> 1341119 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
[junit4] 2> 1341121 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4] 2> 1341122 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4] 2> 1341124 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4] 2> 1341126 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4] 2> 1341128 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
[junit4] 2> 1341355 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T18:27:37-03:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 1341356 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1341357 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1341357 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 1341357 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@28a4a071{/,null,AVAILABLE}
[junit4] 2> 1341361 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@54d423c7{HTTP/1.1,[http/1.1]}{127.0.0.1:65383}
[junit4] 2> 1341361 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.Server Started @1345394ms
[junit4] 2> 1341361 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/tempDir-001/control/data, replicaType=NRT, hostContext=/, hostPort=65383, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/control-001/cores}
[junit4] 2> 1341361 ERROR (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1341361 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 1341361 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1341361 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1341361 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-27T00:24:39.473Z
[junit4] 2> 1341363 INFO (zkConnectionManagerCallback-2920-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1341364 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1341364 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/control-001/solr.xml
[junit4] 2> 1341371 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 1341371 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1341371 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 1341374 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65380/solr
[junit4] 2> 1341376 INFO (zkConnectionManagerCallback-2924-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1341378 INFO (zkConnectionManagerCallback-2926-thread-1-processing-n:127.0.0.1:65383_) [n:127.0.0.1:65383_ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1341540 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65383_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1341541 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65383_ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:65383_
[junit4] 2> 1341543 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65383_ ] o.a.s.c.Overseer Overseer (id=99418942931992580-127.0.0.1:65383_-n_0000000000) starting
[junit4] 2> 1341556 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65383_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65383_
[junit4] 2> 1341559 INFO (OverseerStateUpdate-99418942931992580-127.0.0.1:65383_-n_0000000000) [n:127.0.0.1:65383_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1341705 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65383_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1341717 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65383_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1341717 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65383_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1341719 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65383_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/control-001/cores
[junit4] 2> 1341743 INFO (zkConnectionManagerCallback-2932-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1341745 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1341746 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:65380/solr ready
[junit4] 2> 1341753 INFO (qtp1499288484-13681) [n:127.0.0.1:65383_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:65383_&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1341758 INFO (OverseerThreadFactory-5500-thread-1-processing-n:127.0.0.1:65383_) [n:127.0.0.1:65383_ ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
[junit4] 2> 1341872 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1341874 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 1341996 INFO (zkCallback-2925-thread-1-processing-n:127.0.0.1:65383_) [n:127.0.0.1:65383_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1342916 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1342933 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
[junit4] 2> 1343003 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1343022 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from collection control_collection, trusted=true
[junit4] 2> 1343023 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1343023 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1343023 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/control-001/cores/control_collection_shard1_replica_n1/data/]
[junit4] 2> 1343026 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=27.876953125, floorSegmentMB=1.43359375, forceMergeDeletesPctAllowed=16.119863658364675, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1343030 WARN (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1343075 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1343075 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1343076 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 1343076 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1343077 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=81.54296875, floorSegmentMB=0.9990234375, forceMergeDeletesPctAllowed=4.537871920032415, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.737975664255631
[junit4] 2> 1343078 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1d4f376e[control_collection_shard1_replica_n1] main]
[junit4] 2> 1343079 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1343080 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1343081 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1343081 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590703089194631168
[junit4] 2> 1343082 INFO (searcherExecutor-5503-thread-1-processing-n:127.0.0.1:65383_ x:control_collection_shard1_replica_n1 s:shard1 c:control_collection) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@1d4f376e[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1343088 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1343088 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1343088 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:65383/control_collection_shard1_replica_n1/
[junit4] 2> 1343088 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1343088 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:65383/control_collection_shard1_replica_n1/ has no replicas
[junit4] 2> 1343088 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1343094 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:65383/control_collection_shard1_replica_n1/ shard1
[junit4] 2> 1343207 INFO (zkCallback-2925-thread-2-processing-n:127.0.0.1:65383_) [n:127.0.0.1:65383_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1343259 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1343264 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ c:control_collection s:shard1 x:control_collection_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1392
[junit4] 2> 1343272 INFO (qtp1499288484-13681) [n:127.0.0.1:65383_ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 1343369 INFO (zkCallback-2925-thread-2-processing-n:127.0.0.1:65383_) [n:127.0.0.1:65383_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
[junit4] 2> 1343765 INFO (OverseerCollectionConfigSetProcessor-99418942931992580-127.0.0.1:65383_-n_0000000000) [n:127.0.0.1:65383_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1344280 INFO (qtp1499288484-13681) [n:127.0.0.1:65383_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:65383_&wt=javabin&version=2} status=0 QTime=2527
[junit4] 2> 1344285 INFO (zkConnectionManagerCallback-2936-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1344287 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1344288 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:65380/solr ready
[junit4] 2> 1344288 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
[junit4] 2> 1344289 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1344292 INFO (OverseerThreadFactory-5500-thread-2-processing-n:127.0.0.1:65383_) [n:127.0.0.1:65383_ ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 1344310 WARN (OverseerThreadFactory-5500-thread-2-processing-n:127.0.0.1:65383_) [n:127.0.0.1:65383_ ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
[junit4] 2> 1344525 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 1344525 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=2&wt=javabin&version=2} status=0 QTime=235
[junit4] 2> 1344850 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-1-001 of type NRT
[junit4] 2> 1344853 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T18:27:37-03:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 1344858 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1344858 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1344858 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 1344858 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4e6d22bc{/,null,AVAILABLE}
[junit4] 2> 1344859 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@2cb98c7f{HTTP/1.1,[http/1.1]}{127.0.0.1:65392}
[junit4] 2> 1344859 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.Server Started @1348891ms
[junit4] 2> 1344859 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=65392, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-1-001/cores}
[junit4] 2> 1344859 ERROR (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1344859 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 1344859 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1344859 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1344859 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-27T00:24:42.971Z
[junit4] 2> 1344861 INFO (zkConnectionManagerCallback-2938-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1344863 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1344863 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-1-001/solr.xml
[junit4] 2> 1344868 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 1344868 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1344869 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 1344874 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65380/solr
[junit4] 2> 1344876 INFO (zkConnectionManagerCallback-2942-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1344877 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x161350079680008, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 1344879 INFO (zkConnectionManagerCallback-2944-thread-1-processing-n:127.0.0.1:65392_) [n:127.0.0.1:65392_ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1344893 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65392_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1344897 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65392_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1344901 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65392_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65392_
[junit4] 2> 1344904 INFO (zkCallback-2925-thread-2-processing-n:127.0.0.1:65383_) [n:127.0.0.1:65383_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1344907 INFO (zkCallback-2935-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1344909 INFO (zkCallback-2943-thread-1-processing-n:127.0.0.1:65392_) [n:127.0.0.1:65392_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1344992 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65392_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1345004 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65392_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1345004 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65392_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1345006 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65392_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-1-001/cores
[junit4] 2> 1345036 INFO (qtp1499288484-13686) [n:127.0.0.1:65383_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:65392_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1345040 INFO (OverseerCollectionConfigSetProcessor-99418942931992580-127.0.0.1:65383_-n_0000000000) [n:127.0.0.1:65383_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1345041 INFO (OverseerThreadFactory-5500-thread-3-processing-n:127.0.0.1:65383_) [n:127.0.0.1:65383_ ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:65392_ for creating new replica
[junit4] 2> 1345052 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1345053 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 1345168 INFO (zkCallback-2943-thread-1-processing-n:127.0.0.1:65392_) [n:127.0.0.1:65392_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1346087 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1346097 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema [collection1_shard2_replica_n41] Schema name=test
[junit4] 2> 1346188 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1346216 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n41' using configuration from collection collection1, trusted=true
[junit4] 2> 1346217 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n41' (registry 'solr.core.collection1.shard2.replica_n41') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1346217 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1346217 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [[collection1_shard2_replica_n41] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-1-001/cores/collection1_shard2_replica_n41], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-1-001/cores/collection1_shard2_replica_n41/data/]
[junit4] 2> 1346222 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=27.876953125, floorSegmentMB=1.43359375, forceMergeDeletesPctAllowed=16.119863658364675, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1346224 WARN (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1346287 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1346287 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1346288 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 1346288 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1346289 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=81.54296875, floorSegmentMB=0.9990234375, forceMergeDeletesPctAllowed=4.537871920032415, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.737975664255631
[junit4] 2> 1346289 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.s.SolrIndexSearcher Opening [Searcher@3506f6f4[collection1_shard2_replica_n41] main]
[junit4] 2> 1346291 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1346292 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1346293 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1346293 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590703092562657280
[junit4] 2> 1346294 INFO (searcherExecutor-5514-thread-1-processing-n:127.0.0.1:65392_ x:collection1_shard2_replica_n41 s:shard2 c:collection1) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SolrCore [collection1_shard2_replica_n41] Registered new searcher Searcher@3506f6f4[collection1_shard2_replica_n41] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1346300 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1346300 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1346300 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:65392/collection1_shard2_replica_n41/
[junit4] 2> 1346300 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1346300 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.SyncStrategy http://127.0.0.1:65392/collection1_shard2_replica_n41/ has no replicas
[junit4] 2> 1346300 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1346306 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:65392/collection1_shard2_replica_n41/ shard2
[junit4] 2> 1346419 INFO (zkCallback-2943-thread-1-processing-n:127.0.0.1:65392_) [n:127.0.0.1:65392_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1346478 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1346480 INFO (qtp2110380002-13729) [n:127.0.0.1:65392_ c:collection1 s:shard2 x:collection1_shard2_replica_n41] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n41&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1430
[junit4] 2> 1346484 INFO (qtp1499288484-13686) [n:127.0.0.1:65383_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:65392_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2} status=0 QTime=1447
[junit4] 2> 1346583 INFO (zkCallback-2943-thread-1-processing-n:127.0.0.1:65392_) [n:127.0.0.1:65392_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1346781 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-2-001 of type NRT
[junit4] 2> 1346782 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T18:27:37-03:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 1346783 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1346783 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1346783 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 1346783 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@676583d3{/,null,AVAILABLE}
[junit4] 2> 1346784 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@3c71a58e{HTTP/1.1,[http/1.1]}{127.0.0.1:65397}
[junit4] 2> 1346784 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.Server Started @1350817ms
[junit4] 2> 1346784 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/tempDir-001/jetty2, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=65397, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-2-001/cores}
[junit4] 2> 1346784 ERROR (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1346784 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 1346784 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1346784 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1346784 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-27T00:24:44.896Z
[junit4] 2> 1346786 INFO (zkConnectionManagerCallback-2948-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1346787 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1346787 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-2-001/solr.xml
[junit4] 2> 1346793 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 1346793 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1346793 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 1346798 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65380/solr
[junit4] 2> 1346799 INFO (zkConnectionManagerCallback-2952-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1346802 INFO (zkConnectionManagerCallback-2954-thread-1-processing-n:127.0.0.1:65397_) [n:127.0.0.1:65397_ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1346814 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65397_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1346817 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65397_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1346821 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65397_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65397_
[junit4] 2> 1346822 INFO (zkCallback-2935-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1346822 INFO (zkCallback-2925-thread-2-processing-n:127.0.0.1:65383_) [n:127.0.0.1:65383_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1346822 INFO (zkCallback-2943-thread-1-processing-n:127.0.0.1:65392_) [n:127.0.0.1:65392_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1346825 INFO (zkCallback-2953-thread-1-processing-n:127.0.0.1:65397_) [n:127.0.0.1:65397_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1346908 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65397_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1346920 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65397_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1346920 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65397_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1346921 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65397_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-2-001/cores
[junit4] 2> 1347003 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:65397_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1347007 INFO (OverseerCollectionConfigSetProcessor-99418942931992580-127.0.0.1:65383_-n_0000000000) [n:127.0.0.1:65383_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000004 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1347008 INFO (OverseerThreadFactory-5500-thread-4-processing-n:127.0.0.1:65383_) [n:127.0.0.1:65383_ ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:65397_ for creating new replica
[junit4] 2> 1347015 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1347021 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 1347133 INFO (zkCallback-2953-thread-1-processing-n:127.0.0.1:65397_) [n:127.0.0.1:65397_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1347133 INFO (zkCallback-2943-thread-1-processing-n:127.0.0.1:65392_) [n:127.0.0.1:65392_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1348053 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1348062 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema [collection1_shard1_replica_n43] Schema name=test
[junit4] 2> 1348143 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1348169 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_n43' using configuration from collection collection1, trusted=true
[junit4] 2> 1348170 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard1.replica_n43' (registry 'solr.core.collection1.shard1.replica_n43') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1348170 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1348170 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [[collection1_shard1_replica_n43] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-2-001/cores/collection1_shard1_replica_n43], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-2-001/cores/collection1_shard1_replica_n43/data/]
[junit4] 2> 1348175 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=27.876953125, floorSegmentMB=1.43359375, forceMergeDeletesPctAllowed=16.119863658364675, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1348179 WARN (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1348230 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1348230 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1348231 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 1348231 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1348232 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=81.54296875, floorSegmentMB=0.9990234375, forceMergeDeletesPctAllowed=4.537871920032415, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.737975664255631
[junit4] 2> 1348232 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.s.SolrIndexSearcher Opening [Searcher@5dde2f3a[collection1_shard1_replica_n43] main]
[junit4] 2> 1348234 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1348234 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1348235 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1348235 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590703094598991872
[junit4] 2> 1348236 INFO (searcherExecutor-5525-thread-1-processing-n:127.0.0.1:65397_ x:collection1_shard1_replica_n43 s:shard1 c:collection1) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SolrCore [collection1_shard1_replica_n43] Registered new searcher Searcher@5dde2f3a[collection1_shard1_replica_n43] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1348240 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1348240 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1348240 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:65397/collection1_shard1_replica_n43/
[junit4] 2> 1348240 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1348240 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.SyncStrategy http://127.0.0.1:65397/collection1_shard1_replica_n43/ has no replicas
[junit4] 2> 1348240 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1348245 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:65397/collection1_shard1_replica_n43/ shard1
[junit4] 2> 1348347 INFO (zkCallback-2943-thread-1-processing-n:127.0.0.1:65392_) [n:127.0.0.1:65392_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1348347 INFO (zkCallback-2953-thread-1-processing-n:127.0.0.1:65397_) [n:127.0.0.1:65397_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1348358 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1348360 INFO (qtp487481438-13761) [n:127.0.0.1:65397_ c:collection1 s:shard1 x:collection1_shard1_replica_n43] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_replica_n43&action=CREATE&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1346
[junit4] 2> 1348363 INFO (qtp1499288484-13685) [n:127.0.0.1:65383_ ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={node=127.0.0.1:65397_&action=ADDREPLICA&collection=collection1&shard=shard1&type=NRT&wt=javabin&version=2} status=0 QTime=1360
[junit4] 2> 1348462 INFO (zkCallback-2943-thread-1-processing-n:127.0.0.1:65392_) [n:127.0.0.1:65392_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1348462 INFO (zkCallback-2953-thread-1-processing-n:127.0.0.1:65397_) [n:127.0.0.1:65397_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1348680 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-3-001 of type NRT
[junit4] 2> 1348681 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T18:27:37-03:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 1348682 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1348682 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1348682 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 1348682 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1cd3133f{/,null,AVAILABLE}
[junit4] 2> 1348682 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@2d038460{HTTP/1.1,[http/1.1]}{127.0.0.1:65402}
[junit4] 2> 1348682 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.e.j.s.Server Started @1352715ms
[junit4] 2> 1348682 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/tempDir-001/jetty3, replicaType=NRT, solrconfig=solrconfig.xml, hostContext=/, hostPort=65402, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-3-001/cores}
[junit4] 2> 1348683 ERROR (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1348683 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 1348683 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1348683 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1348683 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-27T00:24:46.795Z
[junit4] 2> 1348684 INFO (zkConnectionManagerCallback-2958-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1348685 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
[junit4] 2> 1348686 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-3-001/solr.xml
[junit4] 2> 1348690 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
[junit4] 2> 1348690 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
[junit4] 2> 1348690 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76, but no JMX reporters were configured - adding default JMX reporter.
[junit4] 2> 1348693 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:65380/solr
[junit4] 2> 1348695 INFO (zkConnectionManagerCallback-2962-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1348697 INFO (zkConnectionManagerCallback-2964-thread-1-processing-n:127.0.0.1:65402_) [n:127.0.0.1:65402_ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1348706 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65402_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 1348710 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65402_ ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1348712 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65402_ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65402_
[junit4] 2> 1348714 INFO (zkCallback-2925-thread-2-processing-n:127.0.0.1:65383_) [n:127.0.0.1:65383_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1348714 INFO (zkCallback-2935-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1348714 INFO (zkCallback-2943-thread-1-processing-n:127.0.0.1:65392_) [n:127.0.0.1:65392_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1348714 INFO (zkCallback-2953-thread-1-processing-n:127.0.0.1:65397_) [n:127.0.0.1:65397_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1348717 INFO (zkCallback-2963-thread-1-processing-n:127.0.0.1:65402_) [n:127.0.0.1:65402_ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 1348775 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65402_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1348783 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65402_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1348783 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65402_ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1348785 INFO (TEST-ChaosMonkeySafeLeaderTest.test-seed#[2055D7445EB42AE]) [n:127.0.0.1:65402_ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-3-001/cores
[junit4] 2> 1348832 INFO (qtp1499288484-13686) [n:127.0.0.1:65383_ ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :addreplica with params node=127.0.0.1:65402_&action=ADDREPLICA&collection=collection1&shard=shard2&type=NRT&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1348835 INFO (OverseerCollectionConfigSetProcessor-99418942931992580-127.0.0.1:65383_-n_0000000000) [n:127.0.0.1:65383_ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000006 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1348836 INFO (OverseerThreadFactory-5500-thread-5-processing-n:127.0.0.1:65383_) [n:127.0.0.1:65383_ ] o.a.s.c.a.c.AddReplicaCmd Node Identified 127.0.0.1:65402_ for creating new replica
[junit4] 2> 1348844 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&name=collection1_shard2_replica_n45&action=CREATE&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1348846 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
[junit4] 2> 1348961 INFO (zkCallback-2943-thread-1-processing-n:127.0.0.1:65392_) [n:127.0.0.1:65392_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1348961 INFO (zkCallback-2953-thread-1-processing-n:127.0.0.1:65397_) [n:127.0.0.1:65397_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1348961 INFO (zkCallback-2963-thread-1-processing-n:127.0.0.1:65402_) [n:127.0.0.1:65402_ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [4])
[junit4] 2> 1349869 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1349896 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.s.IndexSchema [collection1_shard2_replica_n45] Schema name=test
[junit4] 2> 1349942 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
[junit4] 2> 1349964 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_n45' using configuration from collection collection1, trusted=true
[junit4] 2> 1349965 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.collection1.shard2.replica_n45' (registry 'solr.core.collection1.shard2.replica_n45') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@42e9af76
[junit4] 2> 1349965 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1349965 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.c.SolrCore [[collection1_shard2_replica_n45] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-3-001/cores/collection1_shard2_replica_n45], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderTest_2055D7445EB42AE-001/shard-3-001/cores/collection1_shard2_replica_n45/data/]
[junit4] 2> 1349969 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=36, maxMergedSegmentMB=27.876953125, floorSegmentMB=1.43359375, forceMergeDeletesPctAllowed=16.119863658364675, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
[junit4] 2> 1349973 WARN (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
[junit4] 2> 1350036 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1350036 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1350037 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms;
[junit4] 2> 1350037 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1350038 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=38, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=81.54296875, floorSegmentMB=0.9990234375, forceMergeDeletesPctAllowed=4.537871920032415, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.737975664255631
[junit4] 2> 1350038 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.s.SolrIndexSearcher Opening [Searcher@4164c949[collection1_shard2_replica_n45] main]
[junit4] 2> 1350039 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
[junit4] 2> 1350039 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
[junit4] 2> 1350039 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1350039 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590703096490622976
[junit4] 2> 1350039 INFO (searcherExecutor-5536-thread-1-processing-n:127.0.0.1:65402_ x:collection1_shard2_replica_n45 s:shard2 c:collection1) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.c.SolrCore [collection1_shard2_replica_n45] Registered new searcher Searcher@4164c949[collection1_shard2_replica_n45] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1350047 INFO (qtp2102412462-13792) [n:127.0.0.1:65402_ c:collection1 s:shard2 x:collection1_shard2_replica_n45] o.a.s.c.ZkController Core needs to recover:collection1_shard2_replica_n45
[junit4] 2> 1350047 INFO (updateExecutor-2959-thread-1-processing-n:127.0.0.1:65402_ x:collection1_shard2_replica_n45 s:shard2 c:collection1) [n:127.0.0.1:65402_ c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.u.DefaultSolrCoreState Running recovery
[junit4] 2> 1350048 INFO (recoveryExecutor-2960-thread-1-processing-n:127.0.0.1:65402_ x:collection1_shard2_replica_n45 s:shard2 c:collection1 r:core_node46) [n:127.0.0.1:65402_ c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.c.RecoveryStrategy Starting recovery process. recoveringAfterStartup=true
[junit4] 2> 1350048 INFO (recoveryExecutor-2960-thread-1-processing-n:127.0.0.1:65402_ x:collection1_shard2_replica_n45 s:shard2 c:collection1 r:core_node46) [n:127.0.0.1:65402_ c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.c.RecoveryStrategy ###### startupVersions=[[]]
[junit4] 2> 1350048 INFO (recoveryExecutor-2960-thread-1-processing-n:127.0.0.1:65402_ x:collection1_shard2_replica_n45 s:shard2 c:collection1 r:core_node46) [n:127.0.0.1:65402_ c:collection1 s:shard2 r:core_node46 x:collection1_shard2_replica_n45] o.a.s.c.RecoveryStrategy Begin buffering updates. core=[collection1_shard2_
[...truncated too long message...]
(jetty-closer-1632-thread-10) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@c22979e{/solr,null,UNAVAILABLE}
[junit4] 2> 2212472 INFO (jetty-closer-1632-thread-10) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 2212477 INFO (coreCloseExecutor-3950-thread-1) [n:127.0.0.1:51934_solr c:collection1 s:shard1 r:core_node10 x:collection1_shard1_replica_n9] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=1928724495
[junit4] 2> 2212477 INFO (coreCloseExecutor-3956-thread-1) [n:127.0.0.1:51917_solr c:collection1 s:shard1 r:core_node4 x:collection1_shard1_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=79760623
[junit4] 2> 2212480 INFO (jetty-closer-1632-thread-7) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1a5387e5{/solr,null,UNAVAILABLE}
[junit4] 2> 2212480 INFO (jetty-closer-1632-thread-7) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 2212484 INFO (jetty-closer-1632-thread-8) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@76393493{/solr,null,UNAVAILABLE}
[junit4] 2> 2212484 INFO (jetty-closer-1632-thread-8) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 2212485 INFO (coreCloseExecutor-3945-thread-2) [n:127.0.0.1:51714_solr c:collection1 s:shard1 r:core_node12 x:collection1_shard1_replica_n11] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=1415909573
[junit4] 2> 2212485 INFO (coreCloseExecutor-3945-thread-1) [n:127.0.0.1:51714_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.testMetricTrigger.shard2.leader, tag=1710995259
[junit4] 2> 2212488 INFO (jetty-closer-1632-thread-1) [ ] o.a.s.c.Overseer Overseer (id=99418986003038231-127.0.0.1:51714_solr-n_0000000006) closing
[junit4] 2> 2212488 INFO (OverseerStateUpdate-99418986003038231-127.0.0.1:51714_solr-n_0000000006) [n:127.0.0.1:51714_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:51714_solr
[junit4] 2> 2212489 WARN (OverseerAutoScalingTriggerThread-99418986003038231-127.0.0.1:51714_solr-n_0000000006) [n:127.0.0.1:51714_solr ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 2212489 DEBUG (jetty-closer-1632-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 2212492 INFO (jetty-closer-1632-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@207694c2{/solr,null,UNAVAILABLE}
[junit4] 2> 2212492 INFO (jetty-closer-1632-thread-1) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 2213615 INFO (jetty-closer-1632-thread-6) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7992cf25{/solr,null,UNAVAILABLE}
[junit4] 2> 2213615 INFO (jetty-closer-1632-thread-6) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 2213615 INFO (jetty-closer-1632-thread-5) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6f80606{/solr,null,UNAVAILABLE}
[junit4] 2> 2213615 INFO (jetty-closer-1632-thread-5) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 2213640 WARN (zkCallback-1857-thread-1-processing-n:127.0.0.1:52186_solr) [n:127.0.0.1:52186_solr ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
[junit4] 2> 2213640 INFO (jetty-closer-1632-thread-13) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1cb43e7b{/solr,null,UNAVAILABLE}
[junit4] 2> 2213640 INFO (jetty-closer-1632-thread-13) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 2213642 ERROR (SUITE-TriggerIntegrationTest-seed#[2055D7445EB42AE]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2213642 INFO (SUITE-TriggerIntegrationTest-seed#[2055D7445EB42AE]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51625 51625
[junit4] 2> 2213659 INFO (Thread-1229) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51625 51625
[junit4] 2> 2213783 WARN (Thread-1229) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 22 /solr/clusterprops.json
[junit4] 2> 22 /solr/aliases.json
[junit4] 2> 21 /solr/security.json
[junit4] 2> 11 /solr/configs/conf
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 157 /solr/collections/collection1/state.json
[junit4] 2> 49 /solr/collections/testMetricTrigger/state.json
[junit4] 2> 22 /solr/clusterstate.json
[junit4] 2> 6 /solr/autoscaling.json
[junit4] 2> 4 /solr/overseer_elect/election/99418986003038231-127.0.0.1:51714_solr-n_0000000006
[junit4] 2> 3 /solr/collections/collection1/leader_elect/shard1/election/99418986003038253-core_node4-n_0000000000
[junit4] 2> 2 /solr/overseer_elect/election/99418986003038265-127.0.0.1:51979_solr-n_0000000017
[junit4] 2> 2 /solr/overseer_elect/election/99418986003038268-127.0.0.1:51986_solr-n_0000000018
[junit4] 2> 2 /solr/collections/collection1/leader_elect/shard1/election/99418986003038262-core_node6-n_0000000002
[junit4] 2> 2 /solr/collections/collection1/leader_elect/shard1/election/99418986003038231-core_node12-n_0000000005
[junit4] 2> 2 /solr/overseer_elect/election/99418986003038219-127.0.0.1:51663_solr-n_0000000002
[junit4] 2> 2 /solr/overseer_elect/election/99418986003038256-127.0.0.1:51934_solr-n_0000000014
[junit4] 2> 2 /solr/overseer_elect/election/99418986003038240-127.0.0.1:51796_solr-n_0000000009
[junit4] 2> 2 /solr/overseer_elect/election/99418986003038234-127.0.0.1:51733_solr-n_0000000007
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 22 /solr/live_nodes
[junit4] 2> 22 /solr/collections
[junit4] 2> 6 /solr/overseer/queue-work
[junit4] 2> 6 /solr/autoscaling/events/.auto_add_replicas
[junit4] 2> 6 /solr/overseer/queue
[junit4] 2> 6 /solr/overseer/collection-queue-work
[junit4] 2> 2 /solr/autoscaling/events/node_lost_triggerMR
[junit4] 2> 2 /solr/autoscaling/events/node_added_triggerCTOOR
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_2055D7445EB42AE-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {id=PostingsFormat(name=LuceneVarGapDocFreqInterval), x_s=PostingsFormat(name=Asserting)}, docValues:{}, maxPointsInLeafNode=67, maxMBSortInHeap=7.234568452708891, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@66090402), locale=ar-BH, timezone=NZ
[junit4] 2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_144 (64-bit)/cpus=3,threads=1,free=45820000,total=349544448
[junit4] 2> NOTE: All tests run in this JVM: [ClassificationUpdateProcessorTest, TestCollectionAPIs, TestLeaderElectionZkExpiry, TestConfigSetImmutable, TestHdfsBackupRestoreCore, TestRandomFlRTGCloud, InfixSuggestersTest, HdfsRecoverLeaseTest, TestMacros, ParsingFieldUpdateProcessorsTest, TestSolrJ, TestLFUCache, CloudExitableDirectoryReaderTest, TestOmitPositions, SolrJmxReporterTest, SolrIndexConfigTest, TestRecovery, AnalyticsQueryTest, TestRTimerTree, TestClassicSimilarityFactory, HdfsDirectoryFactoryTest, MultiThreadedOCPTest, TestUtils, ConnectionReuseTest, OpenCloseCoreStressTest, TestFieldCache, DistributedDebugComponentTest, TestPointFields, DistributedSuggestComponentTest, BlockJoinFacetRandomTest, TestInitQParser, TestImplicitCoreProperties, TestFastOutputStream, TestLockTree, TestMultiWordSynonyms, TestSolrDeletionPolicy1, AddBlockUpdateTest, TestConfigSetsAPIExclusivity, DocExpirationUpdateProcessorFactoryTest, EnumFieldTest, TestLegacyNumericUtils, HdfsChaosMonkeySafeLeaderTest, SolrTestCaseJ4Test, TestOverriddenPrefixQueryForCustomFieldType, OverseerTaskQueueTest, RequiredFieldsTest, SynonymTokenizerTest, HdfsBasicDistributedZkTest, BJQParserTest, TestFiltering, TestShortCircuitedRequests, DistribJoinFromCollectionTest, StatsComponentTest, HttpPartitionTest, MoreLikeThisHandlerTest, TestFieldResource, TestBM25SimilarityFactory, StressHdfsTest, LukeRequestHandlerTest, SecurityConfHandlerTest, TestFunctionQuery, SolrIndexSplitterTest, SuggestComponentContextFilterQueryTest, PKIAuthenticationIntegrationTest, LeaderElectionTest, DistribCursorPagingTest, TestLazyCores, TestMultiValuedNumericRangeQuery, TestBackupRepositoryFactory, SpatialFilterTest, TestCollectionsAPIViaSolrCloudCluster, TestRangeQuery, FastVectorHighlighterTest, LeaderInitiatedRecoveryOnCommitTest, BinaryUpdateRequestHandlerTest, TestExecutePlanAction, PathHierarchyTokenizerFactoryTest, TestSolrCoreSnapshots, CursorMarkTest, TestCloudPivotFacet, TestBlendedInfixSuggestions, TestGeoJSONResponseWriter, TestFreeTextSuggestions, LeaderFailoverAfterPartitionTest, TestFieldCacheVsDocValues, TestTolerantUpdateProcessorCloud, TestCSVResponseWriter, UniqFieldsUpdateProcessorFactoryTest, CdcrVersionReplicationTest, QueryElevationComponentTest, FieldMutatingUpdateProcessorTest, DirectoryFactoryTest, TestStressUserVersions, TestPullReplica, SolrCloudExampleTest, TestSurroundQueryParser, CloudMLTQParserTest, TriggerIntegrationTest]
[junit4] Completed [303/770 (2!)] on J0 in 216.11s, 14 tests, 1 failure <<< FAILURES!
[...truncated 49724 lines...]
[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-9) - Build # 4409 -
Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4409/
Java: 64bit/jdk-9 -XX:+UseCompressedOops -XX:+UseSerialGC
2 tests failed.
FAILED: org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test
Error Message:
Could not find collection:collection2
Stack Trace:
java.lang.AssertionError: Could not find collection:collection2
at __randomizedtesting.SeedInfo.seed([8B1938496AA18B5C:34D0793C45DE6A4]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:140)
at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:135)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:915)
at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:612)
at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeLostTriggerRestoreState
Error Message:
The trigger did not fire at all
Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
at __randomizedtesting.SeedInfo.seed([8B1938496AA18B5C:A0E6ED12F0D99E8C]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeLostTriggerRestoreState(TriggerIntegrationTest.java:368)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
Build Log:
[...truncated 12846 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
[junit4] 2> 1260039 INFO (SUITE-TriggerIntegrationTest-seed#[8B1938496AA18B5C]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_8B1938496AA18B5C-001/init-core-data-001
[junit4] 2> 1260039 WARN (SUITE-TriggerIntegrationTest-seed#[8B1938496AA18B5C]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=82 numCloses=82
[junit4] 2> 1260039 INFO (SUITE-TriggerIntegrationTest-seed#[8B1938496AA18B5C]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 1260040 INFO (SUITE-TriggerIntegrationTest-seed#[8B1938496AA18B5C]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0) w/ MAC_OS_X supressed clientAuth
[junit4] 2> 1260040 INFO (SUITE-TriggerIntegrationTest-seed#[8B1938496AA18B5C]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_8B1938496AA18B5C-001/tempDir-001
[junit4] 2> 1260040 INFO (SUITE-TriggerIntegrationTest-seed#[8B1938496AA18B5C]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1260040 INFO (Thread-3068) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1260040 INFO (Thread-3068) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1260043 ERROR (Thread-3068) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1260141 INFO (SUITE-TriggerIntegrationTest-seed#[8B1938496AA18B5C]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:55854
[junit4] 2> 1260151 INFO (zkConnectionManagerCallback-3697-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1260155 INFO (jetty-launcher-3694-thread-1) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T03:27:37+06:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 1260156 INFO (jetty-launcher-3694-thread-2) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T03:27:37+06:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 1260156 INFO (jetty-launcher-3694-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1260156 INFO (jetty-launcher-3694-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1260156 INFO (jetty-launcher-3694-thread-1) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 1260157 INFO (jetty-launcher-3694-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1260157 INFO (jetty-launcher-3694-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1260157 INFO (jetty-launcher-3694-thread-2) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 1260157 INFO (jetty-launcher-3694-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@194f334b{/solr,null,AVAILABLE}
[junit4] 2> 1260157 INFO (jetty-launcher-3694-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1f0a1d7b{/solr,null,AVAILABLE}
[junit4] 2> 1260160 INFO (jetty-launcher-3694-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4940fb37{HTTP/1.1,[http/1.1]}{127.0.0.1:55860}
[junit4] 2> 1260160 INFO (jetty-launcher-3694-thread-2) [ ] o.e.j.s.Server Started @1261807ms
[junit4] 2> 1260160 INFO (jetty-launcher-3694-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@1d168a64{HTTP/1.1,[http/1.1]}{127.0.0.1:55859}
[junit4] 2> 1260160 INFO (jetty-launcher-3694-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55860}
[junit4] 2> 1260160 INFO (jetty-launcher-3694-thread-1) [ ] o.e.j.s.Server Started @1261807ms
[junit4] 2> 1260160 INFO (jetty-launcher-3694-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55859}
[junit4] 2> 1260160 ERROR (jetty-launcher-3694-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1260160 ERROR (jetty-launcher-3694-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1260160 INFO (jetty-launcher-3694-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 1260160 INFO (jetty-launcher-3694-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1260160 INFO (jetty-launcher-3694-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1260160 INFO (jetty-launcher-3694-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-26T13:10:46.634736Z
[junit4] 2> 1260160 INFO (jetty-launcher-3694-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 1260160 INFO (jetty-launcher-3694-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1260160 INFO (jetty-launcher-3694-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1260160 INFO (jetty-launcher-3694-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-26T13:10:46.634980Z
[junit4] 2> 1260162 INFO (zkConnectionManagerCallback-3699-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1260162 INFO (zkConnectionManagerCallback-3701-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1260163 INFO (jetty-launcher-3694-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1260163 INFO (jetty-launcher-3694-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1260176 INFO (jetty-launcher-3694-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:55854/solr
[junit4] 2> 1260176 INFO (jetty-launcher-3694-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:55854/solr
[junit4] 2> 1260177 INFO (zkConnectionManagerCallback-3707-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1260178 INFO (zkConnectionManagerCallback-3709-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1260180 INFO (zkConnectionManagerCallback-3711-thread-1-processing-n:127.0.0.1:55860_solr) [n:127.0.0.1:55860_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1260180 INFO (zkConnectionManagerCallback-3713-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1260321 INFO (jetty-launcher-3694-thread-1) [n:127.0.0.1:55859_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1260321 INFO (jetty-launcher-3694-thread-2) [n:127.0.0.1:55860_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1260322 INFO (jetty-launcher-3694-thread-1) [n:127.0.0.1:55859_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:55859_solr
[junit4] 2> 1260323 INFO (jetty-launcher-3694-thread-2) [n:127.0.0.1:55860_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55860_solr
[junit4] 2> 1260323 INFO (jetty-launcher-3694-thread-1) [n:127.0.0.1:55859_solr ] o.a.s.c.Overseer Overseer (id=99416293147672582-127.0.0.1:55859_solr-n_0000000000) starting
[junit4] 2> 1260325 INFO (zkCallback-3710-thread-1-processing-n:127.0.0.1:55860_solr) [n:127.0.0.1:55860_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1260325 INFO (zkCallback-3712-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1260333 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 1260334 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 1260335 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:55860_solr]
[junit4] 2> 1260335 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
[junit4] 2> 1260335 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 1260335 INFO (jetty-launcher-3694-thread-1) [n:127.0.0.1:55859_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55859_solr
[junit4] 2> 1260337 INFO (zkCallback-3710-thread-1-processing-n:127.0.0.1:55860_solr) [n:127.0.0.1:55860_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1260337 INFO (zkCallback-3712-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1260338 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1260338 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 1260339 DEBUG (ScheduledTrigger-6548-thread-1) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 1260366 INFO (jetty-launcher-3694-thread-2) [n:127.0.0.1:55860_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55860.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@23e22de8
[junit4] 2> 1260373 INFO (jetty-launcher-3694-thread-2) [n:127.0.0.1:55860_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55860.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@23e22de8
[junit4] 2> 1260373 INFO (jetty-launcher-3694-thread-2) [n:127.0.0.1:55860_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55860.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@23e22de8
[junit4] 2> 1260374 INFO (jetty-launcher-3694-thread-2) [n:127.0.0.1:55860_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_8B1938496AA18B5C-001/tempDir-001/node2/.
[junit4] 2> 1260381 INFO (jetty-launcher-3694-thread-1) [n:127.0.0.1:55859_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55859.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@23e22de8
[junit4] 2> 1260409 INFO (jetty-launcher-3694-thread-1) [n:127.0.0.1:55859_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55859.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@23e22de8
[junit4] 2> 1260410 INFO (jetty-launcher-3694-thread-1) [n:127.0.0.1:55859_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55859.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@23e22de8
[junit4] 2> 1260412 INFO (jetty-launcher-3694-thread-1) [n:127.0.0.1:55859_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_8B1938496AA18B5C-001/tempDir-001/node1/.
[junit4] 2> 1260438 INFO (zkConnectionManagerCallback-3719-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1260441 INFO (zkConnectionManagerCallback-3723-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1260442 INFO (SUITE-TriggerIntegrationTest-seed#[8B1938496AA18B5C]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1260443 INFO (SUITE-TriggerIntegrationTest-seed#[8B1938496AA18B5C]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:55854/solr ready
[junit4] 2> 1260460 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
[junit4] 2> 1260461 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 2
[junit4] 2> 1260461 DEBUG (zkCallback-3712-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 1260461 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 1260463 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 1260464 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1260464 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 1260469 INFO (qtp1972881530-15144) [n:127.0.0.1:55859_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 1260471 INFO (qtp979890054-15157) [n:127.0.0.1:55860_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 1260471 DEBUG (qtp979890054-15150) [n:127.0.0.1:55860_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1260471 INFO (qtp979890054-15150) [n:127.0.0.1:55860_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=3
[junit4] 2> 1260472 DEBUG (zkCallback-3712-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 1260472 DEBUG (zkCallback-3712-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:55859_solr, 127.0.0.1:55860_solr]
[junit4] 2> 1260473 DEBUG (zkCallback-3712-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
[junit4] 2> 1260473 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 1260474 INFO (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
[junit4] 2> 1260475 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 1260475 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T03:27:37+06:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 1260476 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1260476 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1260476 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 1260476 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@36947c23{/solr,null,AVAILABLE}
[junit4] 2> 1260476 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 1260477 DEBUG (ScheduledTrigger-6548-thread-1) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1260477 DEBUG (ScheduledTrigger-6548-thread-1) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 1260477 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@230045e7{HTTP/1.1,[http/1.1]}{127.0.0.1:55872}
[junit4] 2> 1260477 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.e.j.s.Server Started @1262124ms
[junit4] 2> 1260477 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55872}
[junit4] 2> 1260477 ERROR (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1260477 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 1260477 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1260477 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1260477 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-26T13:10:46.951749Z
[junit4] 2> 1260479 INFO (zkConnectionManagerCallback-3725-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1260479 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1260484 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:55854/solr
[junit4] 2> 1260485 INFO (zkConnectionManagerCallback-3729-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1260488 INFO (zkConnectionManagerCallback-3731-thread-1-processing-n:127.0.0.1:55872_solr) [n:127.0.0.1:55872_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1260496 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [n:127.0.0.1:55872_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1260499 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [n:127.0.0.1:55872_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 1260502 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [n:127.0.0.1:55872_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55872_solr
[junit4] 2> 1260504 INFO (zkCallback-3722-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1260505 INFO (zkCallback-3730-thread-1-processing-n:127.0.0.1:55872_solr) [n:127.0.0.1:55872_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1260505 INFO (zkCallback-3710-thread-1-processing-n:127.0.0.1:55860_solr) [n:127.0.0.1:55860_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1260505 INFO (zkCallback-3712-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 1260549 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [n:127.0.0.1:55872_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55872.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@23e22de8
[junit4] 2> 1260559 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [n:127.0.0.1:55872_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55872.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@23e22de8
[junit4] 2> 1260560 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [n:127.0.0.1:55872_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55872.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@23e22de8
[junit4] 2> 1260561 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [n:127.0.0.1:55872_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_8B1938496AA18B5C-001/tempDir-001/node3/.
[junit4] 2> 1261482 DEBUG (ScheduledTrigger-6548-thread-2) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1261482 DEBUG (ScheduledTrigger-6548-thread-2) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1261482 DEBUG (ScheduledTrigger-6548-thread-2) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:55872_solr at time 672766943545138
[junit4] 2> 1262485 DEBUG (ScheduledTrigger-6548-thread-2) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1262486 DEBUG (ScheduledTrigger-6548-thread-2) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1263492 DEBUG (ScheduledTrigger-6548-thread-2) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 1263492 DEBUG (ScheduledTrigger-6548-thread-2) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 1263492 DEBUG (ScheduledTrigger-6548-thread-2) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:55872_solr] added at times [672766943545138], now=672768953561630
[junit4] 2> 1263495 DEBUG (ScheduledTrigger-6548-thread-2) [n:127.0.0.1:55859_solr ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"263e0c2afc732Tcpf7ld4x5w4fcc10qvhwwqbm",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":672766943545138,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[672766943545138],
[junit4] 2> "nodeNames":["127.0.0.1:55872_solr"]}}
[junit4] 2> 1263496 DEBUG (ScheduledTrigger-6548-thread-2) [n:127.0.0.1:55859_solr ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_trigger]
[junit4] 2> 1263499 INFO (ScheduledTrigger-6548-thread-2) [n:127.0.0.1:55859_solr ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
[junit4] 2> 1263500 DEBUG (AutoscalingActionExecutor-6549-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"263e0c2afc732Tcpf7ld4x5w4fcc10qvhwwqbm",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":672766943545138,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[672766943545138],
[junit4] 2> "_enqueue_time_":672768958130795,
[junit4] 2> "nodeNames":["127.0.0.1:55872_solr"]}}
[junit4] 2> 1263505 DEBUG (AutoscalingActionExecutor-6549-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 5000ms
[junit4] 2> 1263505 DEBUG (AutoscalingActionExecutor-6549-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.a.ScheduledTriggers -- processing took 5 ms for event id=263e0c2afc732Tcpf7ld4x5w4fcc10qvhwwqbm
[junit4] 2> 1263516 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=2
[junit4] 2> 1263520 INFO (qtp979890054-15157) [n:127.0.0.1:55860_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 1263524 INFO (qtp311662694-15207) [n:127.0.0.1:55872_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 1263525 DEBUG (qtp979890054-15154) [n:127.0.0.1:55860_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1263525 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=18
[junit4] 2> 1263526 DEBUG (zkCallback-3712-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 1263527 DEBUG (zkCallback-3712-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:55859_solr, 127.0.0.1:55872_solr, 127.0.0.1:55860_solr]
[junit4] 2> 1263527 DEBUG (zkCallback-3712-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
[junit4] 2> 1263527 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 1263528 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 1263528 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 1265534 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[8B1938496AA18B5C]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTrigger
[junit4] 2> 1265554 INFO (TEST-TriggerIntegrationTest.testMetricTrigger-seed#[8B1938496AA18B5C]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testMetricTrigger
[junit4] 2> 1265554 INFO (TEST-TriggerIntegrationTest.testMetricTrigger-seed#[8B1938496AA18B5C]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 5
[junit4] 2> 1265555 DEBUG (zkCallback-3712-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
[junit4] 2> 1265555 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 1265557 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 1265557 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1265558 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
[junit4] 2> 1265561 INFO (zkConnectionManagerCallback-3735-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1265565 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=1&collection.configName=conf&name=testMetricTrigger&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1265568 INFO (OverseerThreadFactory-6550-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.a.c.CreateCollectionCmd Create collection testMetricTrigger
[junit4] 2> 1265681 INFO (OverseerStateUpdate-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testMetricTrigger",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"testMetricTrigger_shard1_replica_n1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:55859/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1265684 INFO (OverseerStateUpdate-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"testMetricTrigger",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"testMetricTrigger_shard2_replica_n2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:55860/solr",
[junit4] 2> "type":"NRT",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1265901 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testMetricTrigger_shard1_replica_n1&action=CREATE&numShards=2&collection=testMetricTrigger&shard=shard1&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1265901 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=testMetricTrigger_shard2_replica_n2&action=CREATE&numShards=2&collection=testMetricTrigger&shard=shard2&wt=javabin&version=2&replicaType=NRT
[junit4] 2> 1265901 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1265902 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1266020 INFO (zkCallback-3710-thread-1-processing-n:127.0.0.1:55860_solr) [n:127.0.0.1:55860_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1266020 INFO (zkCallback-3712-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1266020 INFO (zkCallback-3710-thread-2-processing-n:127.0.0.1:55860_solr) [n:127.0.0.1:55860_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1266021 INFO (zkCallback-3712-thread-2-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1266925 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1266929 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1266951 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.s.IndexSchema [testMetricTrigger_shard2_replica_n2] Schema name=minimal
[junit4] 2> 1266953 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1266953 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.CoreContainer Creating SolrCore 'testMetricTrigger_shard2_replica_n2' using configuration from collection testMetricTrigger, trusted=true
[junit4] 2> 1266953 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.s.IndexSchema [testMetricTrigger_shard1_replica_n1] Schema name=minimal
[junit4] 2> 1266954 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55860.solr.core.testMetricTrigger.shard2.replica_n2' (registry 'solr.core.testMetricTrigger.shard2.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@23e22de8
[junit4] 2> 1266954 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1266954 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.SolrCore [[testMetricTrigger_shard2_replica_n2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_8B1938496AA18B5C-001/tempDir-001/node2/testMetricTrigger_shard2_replica_n2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_8B1938496AA18B5C-001/tempDir-001/node2/./testMetricTrigger_shard2_replica_n2/data/]
[junit4] 2> 1266954 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1266954 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'testMetricTrigger_shard1_replica_n1' using configuration from collection testMetricTrigger, trusted=true
[junit4] 2> 1266955 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55859.solr.core.testMetricTrigger.shard1.replica_n1' (registry 'solr.core.testMetricTrigger.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@23e22de8
[junit4] 2> 1266955 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
[junit4] 2> 1266955 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.SolrCore [[testMetricTrigger_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_8B1938496AA18B5C-001/tempDir-001/node1/testMetricTrigger_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_8B1938496AA18B5C-001/tempDir-001/node1/./testMetricTrigger_shard1_replica_n1/data/]
[junit4] 2> 1267010 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1267010 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1267010 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1267010 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1267011 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1267011 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1267012 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1267012 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1267013 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.s.SolrIndexSearcher Opening [Searcher@4c8cf12e[testMetricTrigger_shard2_replica_n2] main]
[junit4] 2> 1267013 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7f1b5fc9[testMetricTrigger_shard1_replica_n1] main]
[junit4] 2> 1267014 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 1267014 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
[junit4] 2> 1267015 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 1267015 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
[junit4] 2> 1267015 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1267016 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1267016 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590660697675530240
[junit4] 2> 1267016 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1590660697675530240
[junit4] 2> 1267016 INFO (searcherExecutor-6561-thread-1-processing-n:127.0.0.1:55860_solr x:testMetricTrigger_shard2_replica_n2 s:shard2 c:testMetricTrigger r:core_node4) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.SolrCore [testMetricTrigger_shard2_replica_n2] Registered new searcher Searcher@4c8cf12e[testMetricTrigger_shard2_replica_n2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1267017 INFO (searcherExecutor-6562-thread-1-processing-n:127.0.0.1:55859_solr x:testMetricTrigger_shard1_replica_n1 s:shard1 c:testMetricTrigger r:core_node3) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.SolrCore [testMetricTrigger_shard1_replica_n1] Registered new searcher Searcher@7f1b5fc9[testMetricTrigger_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1267023 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1267023 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1267023 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1267023 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1267023 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:55860/solr/testMetricTrigger_shard2_replica_n2/
[junit4] 2> 1267023 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:55859/solr/testMetricTrigger_shard1_replica_n1/
[junit4] 2> 1267023 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1267023 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
[junit4] 2> 1267023 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.SyncStrategy http://127.0.0.1:55860/solr/testMetricTrigger_shard2_replica_n2/ has no replicas
[junit4] 2> 1267023 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:55859/solr/testMetricTrigger_shard1_replica_n1/ has no replicas
[junit4] 2> 1267023 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1267023 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1267029 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:55860/solr/testMetricTrigger_shard2_replica_n2/ shard2
[junit4] 2> 1267029 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:55859/solr/testMetricTrigger_shard1_replica_n1/ shard1
[junit4] 2> 1267132 INFO (zkCallback-3710-thread-2-processing-n:127.0.0.1:55860_solr) [n:127.0.0.1:55860_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1267132 INFO (zkCallback-3712-thread-2-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1267132 INFO (zkCallback-3712-thread-3-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1267132 INFO (zkCallback-3710-thread-1-processing-n:127.0.0.1:55860_solr) [n:127.0.0.1:55860_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1267182 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1267182 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1267185 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node4&collection.configName=conf&newCollection=true&name=testMetricTrigger_shard2_replica_n2&action=CREATE&numShards=2&collection=testMetricTrigger&shard=shard2&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1284
[junit4] 2> 1267185 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=testMetricTrigger_shard1_replica_n1&action=CREATE&numShards=2&collection=testMetricTrigger&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1284
[junit4] 2> 1267188 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 1267289 INFO (zkCallback-3710-thread-1-processing-n:127.0.0.1:55860_solr) [n:127.0.0.1:55860_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1267289 INFO (zkCallback-3712-thread-1-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1267289 INFO (zkCallback-3710-thread-2-processing-n:127.0.0.1:55860_solr) [n:127.0.0.1:55860_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1267289 INFO (zkCallback-3712-thread-3-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/testMetricTrigger/state.json] for collection [testMetricTrigger] has occurred - updating... (live nodes size: [3])
[junit4] 2> 1267574 INFO (OverseerCollectionConfigSetProcessor-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1268191 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=1&collection.configName=conf&name=testMetricTrigger&nrtReplicas=1&action=CREATE&numShards=2&wt=javabin&version=2} status=0 QTime=2626
[junit4] 2> 1268208 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.testMetricTrigger.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=10
[junit4] 2> 1268210 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 1268211 INFO (qtp311662694-15207) [n:127.0.0.1:55872_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 1268222 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.testMetricTrigger.shard2.replica_n2:INDEX.sizeInBytes} status=0 QTime=10
[junit4] 2> 1268224 INFO (qtp979890054-15158) [n:127.0.0.1:55860_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 1268224 DEBUG (qtp1972881530-15151) [n:127.0.0.1:55859_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1268224 INFO (qtp1972881530-15151) [n:127.0.0.1:55859_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=29
[junit4] 2> 1268225 DEBUG (zkCallback-3712-thread-3-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
[junit4] 2> 1268247 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
[junit4] 2> 1268250 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 1268250 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1268250 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
[junit4] 2> 1268256 INFO (qtp1972881530-15151) [n:127.0.0.1:55859_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.testMetricTrigger.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=10
[junit4] 2> 1268257 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.testMetricTrigger.shard1.replica_n1:INDEX.sizeInBytes} status=0 QTime=4
[junit4] 2> 1268257 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 1268258 INFO (qtp311662694-15209) [n:127.0.0.1:55872_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 1268269 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={wt=javabin&version=2&key=solr.core.testMetricTrigger.shard2.replica_n2:INDEX.sizeInBytes} status=0 QTime=10
[junit4] 2> 1268270 INFO (qtp979890054-15158) [n:127.0.0.1:55860_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 1268270 DEBUG (qtp1972881530-15146) [n:127.0.0.1:55859_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 1268271 INFO (qtp1972881530-15146) [n:127.0.0.1:55859_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=26
[junit4] 2> 1268271 DEBUG (zkCallback-3712-thread-3-processing-n:127.0.0.1:55859_solr) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
[junit4] 2> 1268272 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
[junit4] 2> 1268273 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 1268273 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 1268273 DEBUG (OverseerAutoScalingTriggerThread-99416293147672582-127.0.0.1:55859_solr-n_0000000000) [n:127.0.0.1:55859_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
[junit4] 2> 1268274 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[0 (1590660698992541696)]} 0 1
[junit4] 2> 1268275 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[1 (1590660698995687424)]} 0 0
[junit4] 2> 1268289 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[2 (1590660698997784576)]} 0 11
[junit4] 2> 1268290 INFO (qtp979890054-15158) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[3 (1590660699010367488)]} 0 0
[junit4] 2> 1268291 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[4 (1590660699012464640)]} 0 0
[junit4] 2> 1268291 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[5 (1590660699012464640)]} 0 0
[junit4] 2> 1268292 INFO (qtp979890054-15158) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[6 (1590660699013513216)]} 0 0
[junit4] 2> 1268293 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[7 (1590660699014561792)]} 0 0
[junit4] 2> 1268293 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[8 (1590660699014561792)]} 0 0
[junit4] 2> 1268294 INFO (qtp979890054-15158) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[9 (1590660699015610368)]} 0 0
[junit4] 2> 1268295 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[10 (1590660699016658944)]} 0 0
[junit4] 2> 1268295 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[11 (1590660699016658945)]} 0 0
[junit4] 2> 1268296 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[12 (1590660699017707520)]} 0 0
[junit4] 2> 1268297 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[13 (1590660699018756096)]} 0 0
[junit4] 2> 1268297 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[14 (1590660699018756097)]} 0 0
[junit4] 2> 1268298 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[15 (1590660699019804672)]} 0 0
[junit4] 2> 1268299 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[16 (1590660699020853248)]} 0 0
[junit4] 2> 1268299 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[17 (1590660699020853248)]} 0 0
[junit4] 2> 1268300 INFO (qtp979890054-15158) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[18 (1590660699021901824)]} 0 0
[junit4] 2> 1268301 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[19 (1590660699022950400)]} 0 0
[junit4] 2> 1268301 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[20 (1590660699022950400)]} 0 0
[junit4] 2> 1268302 INFO (qtp979890054-15158) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[21 (1590660699023998976)]} 0 0
[junit4] 2> 1268302 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[22 (1590660699023998976)]} 0 0
[junit4] 2> 1268303 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[23 (1590660699025047552)]} 0 0
[junit4] 2> 1268304 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[24 (1590660699026096128)]} 0 0
[junit4] 2> 1268304 INFO (qtp979890054-15158) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[25 (1590660699026096128)]} 0 0
[junit4] 2> 1268305 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[26 (1590660699027144704)]} 0 0
[junit4] 2> 1268306 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[27 (1590660699028193280)]} 0 0
[junit4] 2> 1268306 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[28 (1590660699028193281)]} 0 0
[junit4] 2> 1268307 INFO (qtp979890054-15158) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[29 (1590660699029241856)]} 0 0
[junit4] 2> 1268308 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[30 (1590660699029241857)]} 0 0
[junit4] 2> 1268308 INFO (qtp979890054-15158) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[31 (1590660699030290432)]} 0 0
[junit4] 2> 1268309 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[32 (1590660699031339008)]} 0 0
[junit4] 2> 1268310 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[33 (1590660699031339008)]} 0 0
[junit4] 2> 1268310 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[34 (1590660699032387584)]} 0 0
[junit4] 2> 1268311 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[35 (1590660699033436160)]} 0 0
[junit4] 2> 1268311 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[36 (1590660699033436161)]} 0 0
[junit4] 2> 1268312 INFO (qtp979890054-15158) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[37 (1590660699034484736)]} 0 0
[junit4] 2> 1268313 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[38 (1590660699035533312)]} 0 0
[junit4] 2> 1268313 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[39 (1590660699035533312)]} 0 0
[junit4] 2> 1268314 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[40 (1590660699036581888)]} 0 0
[junit4] 2> 1268315 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[41 (1590660699037630464)]} 0 0
[junit4] 2> 1268315 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[42 (1590660699037630465)]} 0 0
[junit4] 2> 1268316 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[43 (1590660699038679040)]} 0 0
[junit4] 2> 1268316 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[44 (1590660699038679041)]} 0 0
[junit4] 2> 1268317 INFO (qtp979890054-15158) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[45 (1590660699039727616)]} 0 0
[junit4] 2> 1268318 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[46 (1590660699040776192)]} 0 0
[junit4] 2> 1268319 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[47 (1590660699041824768)]} 0 0
[junit4] 2> 1268320 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[48 (1590660699042873344)]} 0 0
[junit4] 2> 1268320 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[49 (1590660699042873345)]} 0 0
[junit4] 2> 1268321 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[50 (1590660699043921920)]} 0 0
[junit4] 2> 1268321 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[51 (1590660699043921921)]} 0 0
[junit4] 2> 1268322 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[52 (1590660699044970496)]} 0 0
[junit4] 2> 1268323 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[53 (1590660699046019072)]} 0 0
[junit4] 2> 1268324 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[54 (1590660699047067648)]} 0 0
[junit4] 2> 1268324 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[55 (1590660699047067649)]} 0 0
[junit4] 2> 1268325 INFO (qtp979890054-15158) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[56 (1590660699048116224)]} 0 0
[junit4] 2> 1268325 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[57 (1590660699048116225)]} 0 0
[junit4] 2> 1268326 INFO (qtp979890054-15158) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[58 (1590660699049164800)]} 0 0
[junit4] 2> 1268327 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[59 (1590660699050213376)]} 0 0
[junit4] 2> 1268327 INFO (qtp979890054-15154) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[60 (1590660699050213376)]} 0 0
[junit4] 2> 1268328 INFO (qtp1972881530-15148) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&version=2}{add=[61 (1590660699051261952)]} 0 0
[junit4] 2> 1268329 INFO (qtp979890054-15158) [n:127.0.0.1:55860_solr c:testMetricTrigger s:shard2 r:core_node4 x:testMetricTrigger_shard2_replica_n2] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard2_replica_n2] webapp=/solr path=/update params={wt=javabin&version=2}{add=[62 (1590660699052310528)]} 0 0
[junit4] 2> 1268329 INFO (qtp1972881530-15149) [n:127.0.0.1:55859_solr c:testMetricTrigger s:shard1 r:core_node3 x:testMetricTrigger_shard1_replica_n1] o.a.s.u.p.LogUpdateProcessorFactory [testMetricTrigger_shard1_replica_n1] webapp=/solr path=/update params={wt=javabin&versi
[...truncated too long message...]
496AA18B5C]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@454b51fe{/_,null,UNAVAILABLE}
[junit4] 2> 1900967 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[8B1938496AA18B5C]) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 1900968 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[8B1938496AA18B5C]) [ ] o.a.s.c.ChaosMonkey monkey: stop jetty! 58902
[junit4] 2> 1900969 ERROR (TEST-FullSolrCloudDistribCmdsTest.test-seed#[8B1938496AA18B5C]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1900969 INFO (TEST-FullSolrCloudDistribCmdsTest.test-seed#[8B1938496AA18B5C]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58816 58816
[junit4] 2> 1900979 INFO (Thread-3029) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58816 58816
[junit4] 2> 1901141 WARN (Thread-3029) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 15 /solr/aliases.json
[junit4] 2> 11 /solr/configs/conf1
[junit4] 2> 7 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 15 /solr/clusterprops.json
[junit4] 2> 15 /solr/clusterstate.json
[junit4] 2> 6 /solr/collections/collection1/state.json
[junit4] 2> 5 /solr/collections/collection2/state.json
[junit4] 2> 2 /solr/collections/collection2/leader_elect/shard4/election/99416331153965077-core_node30-n_0000000000
[junit4] 2> 2 /solr/overseer_elect/election/99416331153965068-127.0.0.1:58853__-n_0000000002
[junit4] 2> 2 /solr/collections/collection2/leader_elect/shard3/election/99416331153965065-core_node23-n_0000000000
[junit4] 2> 2 /solr/collections/collection2/leader_elect/shard6/election/99416331153965068-core_node37-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 15 /solr/live_nodes
[junit4] 2> 15 /solr/collections
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=FullSolrCloudDistribCmdsTest -Dtests.method=test -Dtests.seed=8B1938496AA18B5C -Dtests.slow=true -Dtests.locale=jgo-CM -Dtests.timezone=Brazil/DeNoronha -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 61.3s J1 | FullSolrCloudDistribCmdsTest.test <<<
[junit4] > Throwable #1: java.lang.AssertionError: Could not find collection:collection2
[junit4] > at __randomizedtesting.SeedInfo.seed([8B1938496AA18B5C:34D0793C45DE6A4]:0)
[junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:155)
[junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:140)
[junit4] > at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:135)
[junit4] > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:915)
[junit4] > at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:612)
[junit4] > at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:564)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:993)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:968)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.FullSolrCloudDistribCmdsTest_8B1938496AA18B5C-001
[junit4] 2> Jan 26, 2018 1:21:27 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=CheapBastard, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@53da833), locale=jgo-CM, timezone=Brazil/DeNoronha
[junit4] 2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 9 (64-bit)/cpus=3,threads=1,free=179564128,total=430940160
[junit4] 2> NOTE: All tests run in this JVM: [HdfsSyncSliceTest, ClassificationUpdateProcessorIntegrationTest, LeaderFailureAfterFreshStartTest, TestConfig, InfoHandlerTest, PropertiesRequestHandlerTest, TestCorePropertiesReload, TestExactSharedStatsCache, DocValuesMultiTest, ZkStateWriterTest, TestTriggerIntegration, PreAnalyzedFieldTest, CurrencyRangeFacetCloudTest, AsyncCallRequestStatusResponseTest, CleanupOldIndexTest, TestConfigOverlay, TestShardHandlerFactory, JsonLoaderTest, DebugComponentTest, UUIDUpdateProcessorFallbackTest, TestFilteredDocIdSet, URLClassifyProcessorTest, TestCollationFieldDocValues, MetricUtilsTest, TestSolrIndexConfig, ScriptEngineTest, SolrCLIZkUtilsTest, TestLeaderInitiatedRecoveryThread, DefaultValueUpdateProcessorTest, TestConfigsApi, TestMacroExpander, TestLegacyFieldReuse, TestSolrDeletionPolicy2, TestFileDictionaryLookup, TestDeleteCollectionOnDownNodes, AutoscalingHistoryHandlerTest, HdfsWriteToMultipleCollectionsTest, TestGenericDistributedQueue, SaslZkACLProviderTest, TestPHPSerializedResponseWriter, TestDFISimilarityFactory, CurrencyFieldTypeTest, DisMaxRequestHandlerTest, SolrXmlInZkTest, HdfsNNFailoverTest, TestPolicyCloud, TestUseDocValuesAsStored, ExitableDirectoryReaderTest, BigEndianAscendingWordDeserializerTest, TestSolrCoreProperties, TestDownShardTolerantSearch, TestGraphMLResponseWriter, BadComponentTest, TestJsonFacetRefinement, HdfsRecoveryZkTest, TestBadConfig, MoveReplicaTest, TestCoreDiscovery, ManagedSchemaRoundRobinCloudTest, XsltUpdateRequestHandlerTest, IndexSchemaTest, TestExtendedDismaxParser, PeerSyncTest, TestNumericTerms32, TimeZoneUtilsTest, CollectionTooManyReplicasTest, OverseerModifyCollectionTest, TestEmbeddedSolrServerConstructors, TestDFRSimilarityFactory, AtomicUpdatesTest, SuggestComponentTest, TestScoreJoinQPNoScore, TestConfigSetsAPI, SolrIndexMetricsTest, CollectionsAPIDistributedZkTest, CollectionStateFormat2Test, TestPushWriter, TestConfigSetsAPIZkFailure, CloneFieldUpdateProcessorFactoryTest, TestCloudNestedDocsSort, RegexBoostProcessorTest, TestSubQueryTransformerDistrib, TestReplicaProperties, LeaderElectionContextKeyTest, NotRequiredUniqueKeyTest, TestExpandComponent, SubstringBytesRefFilterTest, TestDocTermOrds, TestFastLRUCache, TestHashPartitioner, AssignTest, TestComponentsName, TestSchemaNameResource, TestFieldCacheSortRandom, HLLUtilTest, TestStressVersions, SpatialRPTFieldTypeTest, JavabinLoaderTest, TestDynamicFieldCollectionResource, TermVectorComponentTest, TestTrieFacet, PrimUtilsTest, TestObjectReleaseTracker, RollingRestartTest, ZkSolrClientTest, BlockJoinFacetSimpleTest, TestSchemaSimilarityResource, DeleteLastCustomShardedReplicaTest, TestLegacyNumericRangeQueryBuilder, SyncSliceTest, OverseerStatusTest, DataDrivenBlockJoinTest, TestFieldCacheWithThreads, HttpTriggerListenerTest, FullHLLTest, TestLMDirichletSimilarityFactory, QueryResultKeyTest, TestQueryWrapperFilter, SchemaApiFailureTest, TestCloudManagedSchema, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestStressRecovery, TestSubQueryTransformer, RequestLoggingTest, TestAuthorizationFramework, TestDistributedGrouping, TestStressReorder, TestSortByMinMaxFunction, TestCSVLoader, UpdateRequestProcessorFactoryTest, NoCacheHeaderTest, TestExportWriter, TestNRTOpen, TestDelegationWithHadoopAuth, TestMinMaxOnMultiValuedField, FieldAnalysisRequestHandlerTest, DocValuesMissingTest, PrimitiveFieldTypeTest, QueryParsingTest, TestLegacyTerms, TestLMJelinekMercerSimilarityFactory, SolrJmxReporterCloudTest, BooleanFieldTest, TestFastWriter, TestRestoreCore, TestSearcherReuse, VMParamsZkACLAndCredentialsProvidersTest, V2ApiIntegrationTest, TestBlobHandler, TestCollectionAPI, TestNumericRangeQuery32, DirectUpdateHandlerTest, TestLuceneIndexBackCompat, TestFieldCacheReopen, RuleEngineTest, HdfsBasicDistributedZk2Test, TestManagedSchema, BasicDistributedZk2Test, ReplicationFactorTest, TestPerFieldSimilarityWithDefaultOverride, ZkControllerTest, TestCrossCoreJoin, SimplePostToolTest, HighlighterConfigTest, DistributedSpellCheckComponentTest, ShardSplitTest, CoreAdminHandlerTest, TemplateUpdateProcessorTest, TestRandomDVFaceting, ClusterStateTest, ChaosMonkeyNothingIsSafeTest, TestSolrConfigHandlerConcurrent, BasicAuthStandaloneTest, JSONWriterTest, CoreSorterTest, TestManagedResourceStorage, TestSolrCloudSnapshots, TestFuzzyAnalyzedSuggestions, TestSolrCloudWithSecureImpersonation, CachingDirectoryFactoryTest, TestAtomicUpdateErrorCases, BlobRepositoryCloudTest, TestUpdate, CdcrReplicationDistributedZkTest, AutoScalingHandlerTest, TestManagedSchemaThreadSafety, DistributedTermsComponentTest, TestCustomSort, LargeFieldTest, PreAnalyzedUpdateProcessorTest, ChaosMonkeySafeLeaderTest, UnloadDistributedZkTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, BasicZkTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest]
[junit4] Completed [392/768 (2!)] on J1 in 61.80s, 1 test, 1 failure <<< FAILURES!
[...truncated 42009 lines...]
[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-9) - Build # 4408 -
Still unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4408/
Java: 64bit/jdk-9 -XX:+UseCompressedOops -XX:+UseSerialGC
4 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.ltr.feature.TestFeatureExtractionFromMultipleSegments
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.ltr.feature.TestFeatureExtractionFromMultipleSegments: 1) Thread[id=57, name=qtp1850466759-57, state=TIMED_WAITING, group=TGRP-TestFeatureExtractionFromMultipleSegments] at java.base@9/jdk.internal.misc.Unsafe.park(Native Method) at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192) at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308) at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.base@9/java.lang.Thread.run(Thread.java:844)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.ltr.feature.TestFeatureExtractionFromMultipleSegments:
1) Thread[id=57, name=qtp1850466759-57, state=TIMED_WAITING, group=TGRP-TestFeatureExtractionFromMultipleSegments]
at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.base@9/java.lang.Thread.run(Thread.java:844)
at __randomizedtesting.SeedInfo.seed([63908A209C36A0E1]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.ltr.feature.TestFeatureExtractionFromMultipleSegments
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=57, name=qtp1850466759-57, state=TIMED_WAITING, group=TGRP-TestFeatureExtractionFromMultipleSegments] at java.base@9/jdk.internal.misc.Unsafe.park(Native Method) at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192) at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308) at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626) at java.base@9/java.lang.Thread.run(Thread.java:844)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=57, name=qtp1850466759-57, state=TIMED_WAITING, group=TGRP-TestFeatureExtractionFromMultipleSegments]
at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.base@9/java.lang.Thread.run(Thread.java:844)
at __randomizedtesting.SeedInfo.seed([63908A209C36A0E1]:0)
FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeAddedTriggerRestoreState
Error Message:
The trigger did not fire at all
Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
at __randomizedtesting.SeedInfo.seed([E9410546733603A2:617C8C3949F6E20F]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testNodeAddedTriggerRestoreState(TriggerIntegrationTest.java:426)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventFromRestoredState
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([E9410546733603A2:E977B1865A3BA7C8]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testEventFromRestoredState(TriggerIntegrationTest.java:760)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
Build Log:
[...truncated 13634 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_E9410546733603A2-001/init-core-data-001
[junit4] 2> 3299050 INFO (SUITE-TriggerIntegrationTest-seed#[E9410546733603A2]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 3299051 INFO (SUITE-TriggerIntegrationTest-seed#[E9410546733603A2]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0) w/ MAC_OS_X supressed clientAuth
[junit4] 2> 3299051 INFO (SUITE-TriggerIntegrationTest-seed#[E9410546733603A2]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 3299052 INFO (SUITE-TriggerIntegrationTest-seed#[E9410546733603A2]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_E9410546733603A2-001/tempDir-001
[junit4] 2> 3299052 INFO (SUITE-TriggerIntegrationTest-seed#[E9410546733603A2]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 3299052 INFO (Thread-5132) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 3299052 INFO (Thread-5132) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 3299057 ERROR (Thread-5132) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 3299162 INFO (SUITE-TriggerIntegrationTest-seed#[E9410546733603A2]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:56421
[junit4] 2> 3299174 INFO (zkConnectionManagerCallback-6235-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3299180 INFO (jetty-launcher-6232-thread-1) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T01:27:37+04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 3299180 INFO (jetty-launcher-6232-thread-2) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T01:27:37+04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 3299182 INFO (jetty-launcher-6232-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 3299183 INFO (jetty-launcher-6232-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 3299183 INFO (jetty-launcher-6232-thread-1) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 3299183 INFO (jetty-launcher-6232-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 3299183 INFO (jetty-launcher-6232-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 3299183 INFO (jetty-launcher-6232-thread-2) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 3299183 INFO (jetty-launcher-6232-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@158783cd{/solr,null,AVAILABLE}
[junit4] 2> 3299183 INFO (jetty-launcher-6232-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@510c68bc{/solr,null,AVAILABLE}
[junit4] 2> 3299185 INFO (jetty-launcher-6232-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@ca87268{HTTP/1.1,[http/1.1]}{127.0.0.1:56424}
[junit4] 2> 3299185 INFO (jetty-launcher-6232-thread-1) [ ] o.e.j.s.Server Started @3301085ms
[junit4] 2> 3299185 INFO (jetty-launcher-6232-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=56424}
[junit4] 2> 3299186 ERROR (jetty-launcher-6232-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3299186 INFO (jetty-launcher-6232-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 3299186 INFO (jetty-launcher-6232-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3299186 INFO (jetty-launcher-6232-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3299186 INFO (jetty-launcher-6232-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-26T02:31:24.534518Z
[junit4] 2> 3299187 INFO (jetty-launcher-6232-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@15655ba6{HTTP/1.1,[http/1.1]}{127.0.0.1:56425}
[junit4] 2> 3299187 INFO (jetty-launcher-6232-thread-2) [ ] o.e.j.s.Server Started @3301087ms
[junit4] 2> 3299187 INFO (jetty-launcher-6232-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=56425}
[junit4] 2> 3299187 ERROR (jetty-launcher-6232-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3299188 INFO (zkConnectionManagerCallback-6237-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3299189 INFO (jetty-launcher-6232-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3299190 INFO (jetty-launcher-6232-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 3299191 INFO (jetty-launcher-6232-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3299191 INFO (jetty-launcher-6232-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3299191 INFO (jetty-launcher-6232-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-26T02:31:24.539299Z
[junit4] 2> 3299193 INFO (zkConnectionManagerCallback-6239-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3299194 INFO (jetty-launcher-6232-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3299196 INFO (jetty-launcher-6232-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56421/solr
[junit4] 2> 3299198 INFO (zkConnectionManagerCallback-6243-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3299200 INFO (jetty-launcher-6232-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56421/solr
[junit4] 2> 3299201 INFO (zkConnectionManagerCallback-6247-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3299202 INFO (zkConnectionManagerCallback-6249-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3299205 INFO (zkConnectionManagerCallback-6251-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3299390 INFO (jetty-launcher-6232-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3299392 INFO (jetty-launcher-6232-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:56425_solr
[junit4] 2> 3299393 INFO (jetty-launcher-6232-thread-1) [n:127.0.0.1:56424_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3299393 INFO (jetty-launcher-6232-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.c.Overseer Overseer (id=99413779048300550-127.0.0.1:56425_solr-n_0000000000) starting
[junit4] 2> 3299395 INFO (jetty-launcher-6232-thread-1) [n:127.0.0.1:56424_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56424_solr
[junit4] 2> 3299397 INFO (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 3299397 INFO (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 3299404 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 3299406 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 3299407 INFO (jetty-launcher-6232-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56425_solr
[junit4] 2> 3299407 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:56424_solr]
[junit4] 2> 3299407 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
[junit4] 2> 3299407 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 3299410 INFO (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 3299410 INFO (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 3299412 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3299412 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 3299413 DEBUG (ScheduledTrigger-12113-thread-1) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 3299458 INFO (jetty-launcher-6232-thread-1) [n:127.0.0.1:56424_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56424.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65ca5e4
[junit4] 2> 3299468 INFO (jetty-launcher-6232-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56425.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65ca5e4
[junit4] 2> 3299469 INFO (jetty-launcher-6232-thread-1) [n:127.0.0.1:56424_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56424.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65ca5e4
[junit4] 2> 3299469 INFO (jetty-launcher-6232-thread-1) [n:127.0.0.1:56424_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56424.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65ca5e4
[junit4] 2> 3299471 INFO (jetty-launcher-6232-thread-1) [n:127.0.0.1:56424_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_E9410546733603A2-001/tempDir-001/node1/.
[junit4] 2> 3299481 INFO (jetty-launcher-6232-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56425.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65ca5e4
[junit4] 2> 3299482 INFO (jetty-launcher-6232-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56425.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65ca5e4
[junit4] 2> 3299483 INFO (jetty-launcher-6232-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_E9410546733603A2-001/tempDir-001/node2/.
[junit4] 2> 3299522 INFO (zkConnectionManagerCallback-6257-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3299529 INFO (zkConnectionManagerCallback-6261-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3299531 INFO (SUITE-TriggerIntegrationTest-seed#[E9410546733603A2]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 3299532 INFO (SUITE-TriggerIntegrationTest-seed#[E9410546733603A2]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:56421/solr ready
[junit4] 2> 3299548 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
[junit4] 2> 3299549 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 2
[junit4] 2> 3299550 DEBUG (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 3299550 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 3299553 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 3299553 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3299554 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 3299562 INFO (qtp1346920357-28511) [n:127.0.0.1:56424_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3299565 INFO (qtp582093706-28521) [n:127.0.0.1:56425_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3299565 DEBUG (qtp582093706-28512) [n:127.0.0.1:56425_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3299566 INFO (qtp582093706-28512) [n:127.0.0.1:56425_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=6
[junit4] 2> 3299567 DEBUG (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 3299568 DEBUG (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:56424_solr, 127.0.0.1:56425_solr]
[junit4] 2> 3299568 DEBUG (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
[junit4] 2> 3299568 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 3299578 INFO (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
[junit4] 2> 3299579 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T01:27:37+04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 3299581 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 3299581 DEBUG (ScheduledTrigger-12113-thread-1) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 3299581 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 3299582 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 3299582 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 3299581 DEBUG (ScheduledTrigger-12113-thread-1) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 3299582 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 3299582 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5f978aa2{/solr,null,AVAILABLE}
[junit4] 2> 3299583 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@431d98e{HTTP/1.1,[http/1.1]}{127.0.0.1:56437}
[junit4] 2> 3299583 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.e.j.s.Server Started @3301483ms
[junit4] 2> 3299583 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=56437}
[junit4] 2> 3299583 ERROR (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3299584 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 3299584 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3299584 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3299584 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-26T02:31:24.932251Z
[junit4] 2> 3299586 INFO (zkConnectionManagerCallback-6263-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3299587 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3299597 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56421/solr
[junit4] 2> 3299599 INFO (zkConnectionManagerCallback-6267-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3299603 INFO (zkConnectionManagerCallback-6269-thread-1-processing-n:127.0.0.1:56437_solr) [n:127.0.0.1:56437_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3299620 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [n:127.0.0.1:56437_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 3299624 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [n:127.0.0.1:56437_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3299627 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [n:127.0.0.1:56437_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56437_solr
[junit4] 2> 3299636 INFO (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 3299636 INFO (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 3299636 INFO (zkCallback-6260-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 3299636 INFO (zkCallback-6268-thread-1-processing-n:127.0.0.1:56437_solr) [n:127.0.0.1:56437_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 3299684 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [n:127.0.0.1:56437_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56437.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65ca5e4
[junit4] 2> 3299702 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [n:127.0.0.1:56437_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56437.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65ca5e4
[junit4] 2> 3299702 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [n:127.0.0.1:56437_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56437.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65ca5e4
[junit4] 2> 3299704 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [n:127.0.0.1:56437_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_E9410546733603A2-001/tempDir-001/node3/.
[junit4] 2> 3300589 DEBUG (ScheduledTrigger-12113-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 3300589 DEBUG (ScheduledTrigger-12113-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3300589 DEBUG (ScheduledTrigger-12113-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:56437_solr at time 634404920396706
[junit4] 2> 3301599 DEBUG (ScheduledTrigger-12113-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 3301599 DEBUG (ScheduledTrigger-12113-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3301599 DEBUG (ScheduledTrigger-12113-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:56437_solr] added at times [634404920396706], now=634405930534836
[junit4] 2> 3301599 DEBUG (ScheduledTrigger-12113-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"240fce7ec8ba2Tflrqbucqcqunn1cigejayyto",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":634404920396706,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[634404920396706],
[junit4] 2> "nodeNames":["127.0.0.1:56437_solr"]}}
[junit4] 2> 3301600 DEBUG (ScheduledTrigger-12113-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_added_trigger]
[junit4] 2> 3301601 INFO (ScheduledTrigger-12113-thread-2) [n:127.0.0.1:56425_solr ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
[junit4] 2> 3301601 DEBUG (AutoscalingActionExecutor-12114-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"240fce7ec8ba2Tflrqbucqcqunn1cigejayyto",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":634404920396706,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[634404920396706],
[junit4] 2> "_enqueue_time_":634405930954955,
[junit4] 2> "nodeNames":["127.0.0.1:56437_solr"]}}
[junit4] 2> 3301602 DEBUG (AutoscalingActionExecutor-12114-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 5000ms
[junit4] 2> 3301603 DEBUG (AutoscalingActionExecutor-12114-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.ScheduledTriggers -- processing took 1 ms for event id=240fce7ec8ba2Tflrqbucqcqunn1cigejayyto
[junit4] 2> 3301604 INFO (qtp1346920357-28520) [n:127.0.0.1:56424_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3301605 INFO (qtp582093706-28521) [n:127.0.0.1:56425_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3301607 INFO (qtp1149754475-28573) [n:127.0.0.1:56437_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3301607 DEBUG (qtp1346920357-28516) [n:127.0.0.1:56424_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3301608 INFO (qtp1346920357-28516) [n:127.0.0.1:56424_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=5
[junit4] 2> 3301609 DEBUG (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 3301609 DEBUG (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:56424_solr, 127.0.0.1:56437_solr, 127.0.0.1:56425_solr]
[junit4] 2> 3301609 DEBUG (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestTriggerAction}], enabled=true}
[junit4] 2> 3301610 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 3301610 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 3301611 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 3303619 INFO (TEST-TriggerIntegrationTest.testNodeAddedTrigger-seed#[E9410546733603A2]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTrigger
[junit4] 2> 3303650 INFO (TEST-TriggerIntegrationTest.testSetProperties-seed#[E9410546733603A2]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testSetProperties
[junit4] 2> 3303652 INFO (TEST-TriggerIntegrationTest.testSetProperties-seed#[E9410546733603A2]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 5
[junit4] 2> 3303653 DEBUG (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
[junit4] 2> 3303653 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 3303656 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 3303656 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3303657 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
[junit4] 2> 3304675 DEBUG (TEST-TriggerIntegrationTest.testSetProperties-seed#[E9410546733603A2]) [ ] o.a.s.c.a.ScheduledTriggers Changing value of autoscaling property: triggerScheduleDelaySeconds from: 1 to: 4
[junit4] 2> 3312698 DEBUG (TEST-TriggerIntegrationTest.testSetProperties-seed#[E9410546733603A2]) [ ] o.a.s.c.a.ScheduledTriggers Changing value of autoscaling property: triggerScheduleDelaySeconds from: 4 to: 1
[junit4] 2> 3322723 DEBUG (TEST-TriggerIntegrationTest.testSetProperties-seed#[E9410546733603A2]) [ ] o.a.s.c.a.ScheduledTriggers Changing value of autoscaling property: triggerCorePoolSize from: 4 to: 6
[junit4] 2> 3332780 INFO (TEST-TriggerIntegrationTest.testSetProperties-seed#[E9410546733603A2]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testSetProperties
[junit4] 2> 3332798 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTriggerRestoreState
[junit4] 2> 3332799 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 6
[junit4] 2> 3332799 DEBUG (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
[junit4] 2> 3332799 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
[junit4] 2> 3332799 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 3332799 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3332799 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
[junit4] 2> 3332804 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T01:27:37+04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 3332805 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 3332805 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 3332805 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 3332806 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4bbc6aef{/solr,null,AVAILABLE}
[junit4] 2> 3332806 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@22a1b869{HTTP/1.1,[http/1.1]}{127.0.0.1:56455}
[junit4] 2> 3332806 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.e.j.s.Server Started @3334706ms
[junit4] 2> 3332806 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=56455}
[junit4] 2> 3332806 ERROR (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3332806 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 3332807 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3332807 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3332807 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-26T02:31:58.155118Z
[junit4] 2> 3332808 INFO (zkConnectionManagerCallback-6273-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3332809 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3332814 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56421/solr
[junit4] 2> 3332816 INFO (zkConnectionManagerCallback-6277-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3332819 INFO (zkConnectionManagerCallback-6279-thread-1-processing-n:127.0.0.1:56455_solr) [n:127.0.0.1:56455_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3332829 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [n:127.0.0.1:56455_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 3332833 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [n:127.0.0.1:56455_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3332836 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [n:127.0.0.1:56455_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56455_solr
[junit4] 2> 3332838 INFO (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3332838 INFO (zkCallback-6260-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3332838 INFO (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3332838 INFO (zkCallback-6278-thread-1-processing-n:127.0.0.1:56455_solr) [n:127.0.0.1:56455_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3332838 INFO (zkCallback-6268-thread-1-processing-n:127.0.0.1:56437_solr) [n:127.0.0.1:56437_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3332873 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [n:127.0.0.1:56455_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56455.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65ca5e4
[junit4] 2> 3332883 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [n:127.0.0.1:56455_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56455.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65ca5e4
[junit4] 2> 3332883 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [n:127.0.0.1:56455_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56455.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65ca5e4
[junit4] 2> 3332884 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [n:127.0.0.1:56455_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_E9410546733603A2-001/tempDir-001/node4/.
[junit4] 2> 3332942 INFO (qtp1346920357-28516) [n:127.0.0.1:56424_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3332943 INFO (qtp582093706-28521) [n:127.0.0.1:56425_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3332945 INFO (qtp1149754475-28578) [n:127.0.0.1:56437_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3332946 INFO (qtp267796602-28608) [n:127.0.0.1:56455_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3332946 DEBUG (qtp1149754475-28577) [n:127.0.0.1:56437_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3332947 INFO (qtp1149754475-28577) [n:127.0.0.1:56437_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=5
[junit4] 2> 3332947 DEBUG (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
[junit4] 2> 3332948 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@22a1b869{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 3332948 DEBUG (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:56424_solr, 127.0.0.1:56437_solr, 127.0.0.1:56425_solr, 127.0.0.1:56455_solr]
[junit4] 2> 3332948 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1081531820
[junit4] 2> 3332948 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
[junit4] 2> 3332948 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 3332948 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@601fc6d0: rootName = solr_56455, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@6aa582de
[junit4] 2> 3332950 INFO (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
[junit4] 2> 3332950 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3332951 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
[junit4] 2> 3332951 DEBUG (ScheduledTrigger-12113-thread-4) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 4
[junit4] 2> 3332954 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 3332955 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@847fb9: rootName = solr_56455, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@7872aed1
[junit4] 2> 3332960 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 3332960 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5254f80a: rootName = solr_56455, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@3d703fbb
[junit4] 2> 3332961 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:56455_solr
[junit4] 2> 3332962 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 3332963 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 3332963 INFO (zkCallback-6260-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 3332963 INFO (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 3332963 INFO (zkCallback-6268-thread-1-processing-n:127.0.0.1:56437_solr) [n:127.0.0.1:56437_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 3332963 INFO (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 3332963 INFO (zkCallback-6278-thread-1-processing-n:127.0.0.1:56455_solr) [n:127.0.0.1:56455_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 3332964 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4bbc6aef{/solr,null,UNAVAILABLE}
[junit4] 2> 3332964 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 3333962 DEBUG (ScheduledTrigger-12113-thread-4) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 3
[junit4] 2> 3333962 DEBUG (ScheduledTrigger-12113-thread-4) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:56455_solr
[junit4] 2> 3334479 INFO (qtp1346920357-28514) [n:127.0.0.1:56424_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3334480 INFO (qtp582093706-28522) [n:127.0.0.1:56425_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3334482 INFO (qtp1149754475-28577) [n:127.0.0.1:56437_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3334482 DEBUG (qtp1149754475-28575) [n:127.0.0.1:56437_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3334483 INFO (qtp1149754475-28575) [n:127.0.0.1:56437_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=7
[junit4] 2> 3334484 DEBUG (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
[junit4] 2> 3334485 DEBUG (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:56424_solr, 127.0.0.1:56437_solr, 127.0.0.1:56425_solr]
[junit4] 2> 3334486 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
[junit4] 2> 3334488 INFO (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
[junit4] 2> 3334489 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:56455_solr
[junit4] 2> 3334491 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3334491 DEBUG (ScheduledTrigger-12113-thread-1) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_restore_trigger with currently live nodes: 3
[junit4] 2> 3334491 DEBUG (ScheduledTrigger-12113-thread-1) [n:127.0.0.1:56425_solr ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:56455_solr]
[junit4] 2> 3334491 DEBUG (ScheduledTrigger-12113-thread-1) [n:127.0.0.1:56425_solr ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"24104cc7e8a56Tflrqbucqcqunn1cigejayytq",
[junit4] 2> "source":"node_lost_restore_trigger",
[junit4] 2> "eventTime":634438819940950,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[634438819940950],
[junit4] 2> "nodeNames":["127.0.0.1:56455_solr"]}}
[junit4] 2> 3334492 DEBUG (ScheduledTrigger-12113-thread-1) [n:127.0.0.1:56425_solr ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_lost_restore_trigger]
[junit4] 2> 3334492 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
[junit4] 2> 3334493 INFO (ScheduledTrigger-12113-thread-1) [n:127.0.0.1:56425_solr ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
[junit4] 2> 3334493 DEBUG (AutoscalingActionExecutor-12114-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"24104cc7e8a56Tflrqbucqcqunn1cigejayytq",
[junit4] 2> "source":"node_lost_restore_trigger",
[junit4] 2> "eventTime":634438819940950,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[634438819940950],
[junit4] 2> "_enqueue_time_":634438822801504,
[junit4] 2> "nodeNames":["127.0.0.1:56455_solr"]}}
[junit4] 2> 3334494 INFO (TEST-TriggerIntegrationTest.testNodeLostTriggerRestoreState-seed#[E9410546733603A2]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTriggerRestoreState
[junit4] 2> 3334502 DEBUG (AutoscalingActionExecutor-12114-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_restore_trigger after 5000ms
[junit4] 2> 3334503 DEBUG (AutoscalingActionExecutor-12114-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.ScheduledTriggers -- processing took 9 ms for event id=24104cc7e8a56Tflrqbucqcqunn1cigejayytq
[junit4] 2> 3334530 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeMarkersRegistration
[junit4] 2> 3334531 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 9
[junit4] 2> 3334532 DEBUG (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
[junit4] 2> 3334532 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
[junit4] 2> 3334535 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 3334535 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3334536 DEBUG (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
[junit4] 2> 3334538 INFO (qtp1346920357-28516) [n:127.0.0.1:56424_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 3334546 INFO (qtp1346920357-28516) [n:127.0.0.1:56424_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=7
[junit4] 2> 3334546 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T01:27:37+04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 3334548 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 3334548 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 3334548 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 3334548 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@45c87837{/solr,null,AVAILABLE}
[junit4] 2> 3334549 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@69d62c95{HTTP/1.1,[http/1.1]}{127.0.0.1:56464}
[junit4] 2> 3334549 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.Server Started @3336449ms
[junit4] 2> 3334549 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=56464}
[junit4] 2> 3334549 ERROR (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 3334549 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 3334549 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 3334549 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 3334549 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-26T02:31:59.897918Z
[junit4] 2> 3334551 INFO (zkConnectionManagerCallback-6283-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3334552 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 3334558 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56421/solr
[junit4] 2> 3334560 INFO (zkConnectionManagerCallback-6287-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3334562 INFO (zkConnectionManagerCallback-6289-thread-1-processing-n:127.0.0.1:56464_solr) [n:127.0.0.1:56464_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 3334573 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [n:127.0.0.1:56464_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 3334577 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [n:127.0.0.1:56464_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 3334580 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [n:127.0.0.1:56464_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56464_solr
[junit4] 2> 3334582 INFO (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3334582 INFO (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3334582 INFO (zkCallback-6268-thread-1-processing-n:127.0.0.1:56437_solr) [n:127.0.0.1:56437_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3334582 INFO (zkCallback-6260-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3334582 INFO (zkCallback-6288-thread-1-processing-n:127.0.0.1:56464_solr) [n:127.0.0.1:56464_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 3334614 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [n:127.0.0.1:56464_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56464.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65ca5e4
[junit4] 2> 3334624 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [n:127.0.0.1:56464_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56464.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65ca5e4
[junit4] 2> 3334624 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [n:127.0.0.1:56464_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_56464.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@65ca5e4
[junit4] 2> 3334625 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [n:127.0.0.1:56464_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.TriggerIntegrationTest_E9410546733603A2-001/tempDir-001/node5/.
[junit4] 2> 3334671 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.c.a.TriggerIntegrationTest ====== KILL OVERSEER 1
[junit4] 2> 3334676 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@15655ba6{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 3334676 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=52434664
[junit4] 2> 3334676 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 3334676 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@afaa508: rootName = solr_56425, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@82148a0
[junit4] 2> 3334683 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 3334683 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4b0203ef: rootName = solr_56425, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@7872aed1
[junit4] 2> 3334688 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 3334688 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6bac577a: rootName = solr_56425, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@3d703fbb
[junit4] 2> 3334690 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:56425_solr
[junit4] 2> 3334691 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 3334692 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 3334692 INFO (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 3334692 INFO (zkCallback-6260-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 3334692 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.c.Overseer Overseer (id=99413779048300550-127.0.0.1:56425_solr-n_0000000000) closing
[junit4] 2> 3334692 INFO (zkCallback-6268-thread-1-processing-n:127.0.0.1:56437_solr) [n:127.0.0.1:56437_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 3334692 INFO (zkCallback-6250-thread-1-processing-n:127.0.0.1:56425_solr) [n:127.0.0.1:56425_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 3334692 INFO (zkCallback-6288-thread-1-processing-n:127.0.0.1:56464_solr) [n:127.0.0.1:56464_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 3334692 INFO (OverseerStateUpdate-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:56425_solr
[junit4] 2> 3334693 WARN (OverseerAutoScalingTriggerThread-99413779048300550-127.0.0.1:56425_solr-n_0000000000) [n:127.0.0.1:56425_solr ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 3334693 DEBUG (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 3334695 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@510c68bc{/solr,null,UNAVAILABLE}
[junit4] 2> 3334695 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 3334695 INFO (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:56424_solr
[junit4] 2> 3334697 INFO (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.Overseer Overseer (id=99413779048300549-127.0.0.1:56424_solr-n_0000000001) starting
[junit4] 2> 3334705 DEBUG (OverseerAutoScalingTriggerThread-99413779048300549-127.0.0.1:56424_solr-n_0000000001) [n:127.0.0.1:56424_solr ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 3334706 DEBUG (OverseerAutoScalingTriggerThread-99413779048300549-127.0.0.1:56424_solr-n_0000000001) [n:127.0.0.1:56424_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
[junit4] 2> 3334707 DEBUG (OverseerAutoScalingTriggerThread-99413779048300549-127.0.0.1:56424_solr-n_0000000001) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:56424_solr, 127.0.0.1:56437_solr, 127.0.0.1:56464_solr]
[junit4] 2> 3334707 DEBUG (OverseerAutoScalingTriggerThread-99413779048300549-127.0.0.1:56424_solr-n_0000000001) [n:127.0.0.1:56424_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion -1
[junit4] 2> 3334707 DEBUG (OverseerAutoScalingTriggerThread-99413779048300549-127.0.0.1:56424_solr-n_0000000001) [n:127.0.0.1:56424_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
[junit4] 2> 3334710 DEBUG (OverseerAutoScalingTriggerThread-99413779048300549-127.0.0.1:56424_solr-n_0000000001) [n:127.0.0.1:56424_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 3334710 DEBUG (OverseerAutoScalingTriggerThread-99413779048300549-127.0.0.1:56424_solr-n_0000000001) [n:127.0.0.1:56424_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
[junit4] 2> 3334711 DEBUG (ScheduledTrigger-12143-thread-1) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3335722 DEBUG (ScheduledTrigger-12143-thread-1) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3336732 DEBUG (ScheduledTrigger-12143-thread-2) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3337743 DEBUG (ScheduledTrigger-12143-thread-1) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3338745 DEBUG (ScheduledTrigger-12143-thread-1) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3339707 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.c.a.TriggerIntegrationTest ====== ADD TRIGGERS
[junit4] 2> 3339719 INFO (qtp1139425151-28634) [n:127.0.0.1:56464_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3339722 INFO (qtp1346920357-28516) [n:127.0.0.1:56424_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3339728 INFO (qtp1149754475-28578) [n:127.0.0.1:56437_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3339728 DEBUG (qtp1346920357-28514) [n:127.0.0.1:56424_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3339730 INFO (qtp1346920357-28514) [n:127.0.0.1:56424_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=18
[junit4] 2> 3339731 DEBUG (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
[junit4] 2> 3339734 DEBUG (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:56424_solr, 127.0.0.1:56437_solr, 127.0.0.1:56464_solr]
[junit4] 2> 3339734 DEBUG (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerMR instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestEventMarkerAction}], enabled=true}
[junit4] 2> 3339735 DEBUG (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:56424_solr, 127.0.0.1:56437_solr, 127.0.0.1:56464_solr]
[junit4] 2> 3339735 DEBUG (OverseerAutoScalingTriggerThread-99413779048300549-127.0.0.1:56424_solr-n_0000000001) [n:127.0.0.1:56424_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
[junit4] 2> 3339737 INFO (qtp1346920357-28514) [n:127.0.0.1:56424_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3339740 INFO (qtp1149754475-28578) [n:127.0.0.1:56437_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3339742 INFO (OverseerAutoScalingTriggerThread-99413779048300549-127.0.0.1:56424_solr-n_0000000001) [n:127.0.0.1:56424_solr ] o.a.s.c.a.TriggerIntegrationTest TestEventMarkerAction init
[junit4] 2> 3339743 INFO (qtp1139425151-28639) [n:127.0.0.1:56464_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 3339744 DEBUG (qtp1139425151-28638) [n:127.0.0.1:56464_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 3339744 DEBUG (OverseerAutoScalingTriggerThread-99413779048300549-127.0.0.1:56424_solr-n_0000000001) [n:127.0.0.1:56424_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
[junit4] 2> 3339745 INFO (qtp1139425151-28638) [n:127.0.0.1:56464_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=11
[junit4] 2> 3339745 DEBUG (ScheduledTrigger-12143-thread-3) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 3339746 DEBUG (ScheduledTrigger-12143-thread-4) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_triggerMR
[junit4] 2> 3339746 DEBUG (ScheduledTrigger-12143-thread-4) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 3339746 INFO (qtp1346920357-28520) [n:127.0.0.1:56424_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 3339747 DEBUG (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 12
[junit4] 2> 3339748 DEBUG (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:56424_solr, 127.0.0.1:56437_solr, 127.0.0.1:56464_solr]
[junit4] 2> 3339750 DEBUG (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeAddedTrigger Initial livenodes: [127.0.0.1:56424_solr, 127.0.0.1:56437_solr, 127.0.0.1:56464_solr]
[junit4] 2> 3339750 DEBUG (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_triggerMR instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.TriggerIntegrationTest$TestEventMarkerAction}], enabled=true}
[junit4] 2> 3339750 DEBUG (zkCallback-6248-thread-1-processing-n:127.0.0.1:56424_solr) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:56424_solr, 127.0.0.1:56437_solr, 127.0.0.1:56464_solr]
[junit4] 2> 3339751 DEBUG (OverseerAutoScalingTriggerThread-99413779048300549-127.0.0.1:56424_solr-n_0000000001) [n:127.0.0.1:56424_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 12
[junit4] 2> 3339755 INFO (OverseerAutoScalingTriggerThread-99413779048300549-127.0.0.1:56424_solr-n_0000000001) [n:127.0.0.1:56424_solr ] o.a.s.c.a.TriggerIntegrationTest TestEventMarkerAction init
[junit4] 2> 3339757 DEBUG (OverseerAutoScalingTriggerThread-99413779048300549-127.0.0.1:56424_solr-n_0000000001) [n:127.0.0.1:56424_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion 12
[junit4] 2> 3339758 DEBUG (ScheduledTrigger-12143-thread-2) [n:127.0.0.1:56424_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_triggerMR with currently live nodes: 3
[junit4] 2> 3339759 INFO (qtp1346920357-28520) [n:127.0.0.1:56424_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=12
[junit4] 2> 3339760 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.a.s.c.a.TriggerIntegrationTest ====== ADD NODE 1
[junit4] 2> 3339760 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T01:27:37+04:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 3339761 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 3339761 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 3339761 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 3339762 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@77173e45{/solr,null,AVAILABLE}
[junit4] 2> 3339762 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistration-seed#[E9410546733603A2]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@27c89b54{HTTP/1.1,[http/1.1]}{127.0.0.1:56473}
[junit4] 2> 3339762 INFO (TEST-TriggerIntegrationTest.testNodeMarkersRegistr
[...truncated too long message...]
stFeatureExtractionFromMultipleSegments-seed#[63908A209C36A0E1]-worker) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@655943e9{/solr,null,UNAVAILABLE}
[junit4] 2> 8774 INFO (SUITE-TestFeatureExtractionFromMultipleSegments-seed#[63908A209C36A0E1]-worker) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 38793 WARN (SUITE-TestFeatureExtractionFromMultipleSegments-seed#[63908A209C36A0E1]-worker) [ ] o.e.j.u.t.QueuedThreadPool QueuedThreadPool@qtp1850466759{STOPPING,?<=?<=?????,i=?,q=?} Couldn't stop Thread[qtp1850466759-57,5,TGRP-TestFeatureExtractionFromMultipleSegments]
[junit4] 2> 38797 INFO (SUITE-TestFeatureExtractionFromMultipleSegments-seed#[63908A209C36A0E1]-worker) [ ] o.a.s.SolrTestCaseJ4 ###deleteCore
[junit4] 2> 38797 INFO (SUITE-TestFeatureExtractionFromMultipleSegments-seed#[63908A209C36A0E1]-worker) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=464542332
[junit4] 2> 38798 INFO (SUITE-TestFeatureExtractionFromMultipleSegments-seed#[63908A209C36A0E1]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 38798 INFO (SUITE-TestFeatureExtractionFromMultipleSegments-seed#[63908A209C36A0E1]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5dde446f: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / null
[junit4] 2> 38798 INFO (SUITE-TestFeatureExtractionFromMultipleSegments-seed#[63908A209C36A0E1]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 38798 INFO (SUITE-TestFeatureExtractionFromMultipleSegments-seed#[63908A209C36A0E1]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7eef9d8: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / null
[junit4] 2> 38798 INFO (SUITE-TestFeatureExtractionFromMultipleSegments-seed#[63908A209C36A0E1]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 38798 INFO (SUITE-TestFeatureExtractionFromMultipleSegments-seed#[63908A209C36A0E1]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5b957df4: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / null
[junit4] 2> 38801 INFO (coreCloseExecutor-51-thread-1) [ x:collection1] o.a.s.c.SolrCore [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@2d44c67f
[junit4] 2> 38801 INFO (coreCloseExecutor-51-thread-1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=759481983
[junit4] 2> 38812 INFO (coreCloseExecutor-51-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4e17df8a: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / null
[junit4] 2> 38818 INFO (SUITE-TestFeatureExtractionFromMultipleSegments-seed#[63908A209C36A0E1]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
[junit4] 2> Jan 26, 2018 2:48:34 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> Jan 26, 2018 2:48:54 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.ltr.feature.TestFeatureExtractionFromMultipleSegments:
[junit4] 2> 1) Thread[id=57, name=qtp1850466759-57, state=TIMED_WAITING, group=TGRP-TestFeatureExtractionFromMultipleSegments]
[junit4] 2> at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
[junit4] 2> at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
[junit4] 2> at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
[junit4] 2> at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] 2> at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] 2> at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] 2> at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] 2> at java.base@9/java.lang.Thread.run(Thread.java:844)
[junit4] 2> Jan 26, 2018 2:48:54 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=57, name=qtp1850466759-57, state=TIMED_WAITING, group=TGRP-TestFeatureExtractionFromMultipleSegments]
[junit4] 2> Jan 26, 2018 2:48:57 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> SEVERE: There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=57, name=qtp1850466759-57, state=TIMED_WAITING, group=TGRP-TestFeatureExtractionFromMultipleSegments]
[junit4] 2> at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
[junit4] 2> at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
[junit4] 2> at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
[junit4] 2> at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] 2> at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] 2> at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] 2> at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] 2> at java.base@9/java.lang.Thread.run(Thread.java:844)
[junit4] 2> NOTE: test params are: codec=Lucene70, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@35996ca2), locale=mzn, timezone=VST
[junit4] 2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 9 (64-bit)/cpus=3,threads=2,free=14310672,total=54853632
[junit4] 2> NOTE: All tests run in this JVM: [TestLTRQParserExplain, TestFeatureExtractionFromMultipleSegments]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestFeatureExtractionFromMultipleSegments -Dtests.seed=63908A209C36A0E1 -Dtests.slow=true -Dtests.locale=mzn -Dtests.timezone=VST -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
[junit4] ERROR 0.00s J1 | TestFeatureExtractionFromMultipleSegments (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.ltr.feature.TestFeatureExtractionFromMultipleSegments:
[junit4] > 1) Thread[id=57, name=qtp1850466759-57, state=TIMED_WAITING, group=TGRP-TestFeatureExtractionFromMultipleSegments]
[junit4] > at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
[junit4] > at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
[junit4] > at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
[junit4] > at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] > at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] > at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] > at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.base@9/java.lang.Thread.run(Thread.java:844)
[junit4] > at __randomizedtesting.SeedInfo.seed([63908A209C36A0E1]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=57, name=qtp1850466759-57, state=TIMED_WAITING, group=TGRP-TestFeatureExtractionFromMultipleSegments]
[junit4] > at java.base@9/jdk.internal.misc.Unsafe.park(Native Method)
[junit4] > at java.base@9/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
[junit4] > at java.base@9/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2192)
[junit4] > at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:308)
[junit4] > at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:373)
[junit4] > at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
[junit4] > at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
[junit4] > at java.base@9/java.lang.Thread.run(Thread.java:844)
[junit4] > at __randomizedtesting.SeedInfo.seed([63908A209C36A0E1]:0)
[junit4] Completed [28/35 (1!)] on J1 in 56.01s, 1 test, 2 errors <<< FAILURES!
[...truncated 33441 lines...]
Re: [JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build #
4407 - Failure!
Posted by Steve Rowe <sa...@gmail.com>.
I believe I’ve seen this kind of error under disk full conditions, but I don’t see any other indications of that in this Jenkins log. Maybe some other kind of disk problem?
FWIW (probably not much): there are 62 failures like this, but only 5 unique resources that aren’t accessible; each fails a different number times:
$ perl -ne 'if (/invalid api spec:/i) { s/^.*(?=invalid api spec:)//i; s/\s+$//; $c{$_}++; } END { for (sort {$c{$b}<=>$c{$a}} keys %c) { print "$c{$_}\t$_\n"; } }' consoleText
40 invalid API spec: apispec/emptySpec.json
12 invalid API spec: apispec/core.SchemaRead.json
5 invalid API spec: apispec/core.config.Commands.json
4 invalid API spec: apispec/core.SchemaRead.fields.json
1 invalid API spec: apispec/cluster.configs.delete.json
--
Steve
www.lucidworks.com
> On Jan 25, 2018, at 11:06 AM, Shalin Shekhar Mangar <sh...@gmail.com> wrote:
>
> This is a strange failure. The logs are full of messages "invalid API
> spec:" which is a misleading message. The message is logged if the
> call to ValidatingJsonMap.class.getClassLoader().getResourceAsStream(resourceName);
> returns a null input stream. I'll fix the message but I am not sure
> why the call failed. I looked through the last two mac os failures and
> I don't see this exception. Does anyone have any ideas?
>
> On Thu, Jan 25, 2018 at 7:53 PM, Policeman Jenkins Server
> <je...@thetaphi.de> wrote:
>> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4407/
>> Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC
>>
>> 8 tests failed.
>> FAILED: org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testSpecificConfigsets
>>
>> Error Message:
>> KeeperErrorCode = NoNode for /collections/withconfigset2
>>
>> Stack Trace:
>> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/withconfigset2
>> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:530732B00F155C13]:0)
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
>> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
>> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
>> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
>> at org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testSpecificConfigsets(CollectionsAPIDistributedZkTest.java:373)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> FAILED: org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCollectionReload
>>
>> Error Message:
>> Could not find collection : reloaded_collection
>>
>> Stack Trace:
>> org.apache.solr.common.SolrException: Could not find collection : reloaded_collection
>> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:7FC051D42DBE4980]:0)
>> at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:118)
>> at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:247)
>> at org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.collectStartTimes(CollectionsAPIDistributedZkTest.java:550)
>> at org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCollectionReload(CollectionsAPIDistributedZkTest.java:491)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> FAILED: org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection
>>
>> Error Message:
>> Error from server at http://127.0.0.1:63452/solr: ADDREPLICA failed to create replica
>>
>> Stack Trace:
>> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:63452/solr: ADDREPLICA failed to create replica
>> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:54F9110F4FEBC6D7]:0)
>> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
>> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
>> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
>> at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
>> at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
>> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
>> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
>> at org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection(CollectionsAPIDistributedZkTest.java:170)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> FAILED: org.apache.solr.cloud.api.collections.CustomCollectionTest.testRouteFieldForImplicitRouter
>>
>> Error Message:
>> Collection not found: withShardField
>>
>> Stack Trace:
>> org.apache.solr.common.SolrException: Collection not found: withShardField
>> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:2B29957854C829EF]:0)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:851)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
>> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
>> at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
>> at org.apache.solr.cloud.api.collections.CustomCollectionTest.testRouteFieldForImplicitRouter(CustomCollectionTest.java:142)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> FAILED: org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple
>>
>> Error Message:
>>
>>
>> Stack Trace:
>> java.lang.NullPointerException
>> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:46CA5914DFC232CE]:0)
>> at org.apache.solr.client.solrj.embedded.JettySolrRunner.getNodeName(JettySolrRunner.java:345)
>> at org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple(AutoAddReplicasPlanActionTest.java:65)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest
>>
>> Error Message:
>> 10 threads leaked from SUITE scope at org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest: 1) Thread[id=25475, name=Connection evictor, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=25489, name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) 3) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=25490, name=jetty-launcher-5573-thread-2-EventThread, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) 6) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=25470, name=Connection evictor, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 8) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 9) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 10) Thread[id=25521, name=zkCallback-5597-thread-2-processing-n:127.0.0.1:63757_solr, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:494) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:437) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:424) at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:788) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135) at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56) at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348) at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268) at org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$231/1095815815.run(Unknown Source) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$114/259134832.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
>>
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest:
>> 1) Thread[id=25475, name=Connection evictor, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>> at java.lang.Thread.run(Thread.java:748)
>> 2) Thread[id=25489, name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
>> 3) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> 4) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> 5) Thread[id=25490, name=jetty-launcher-5573-thread-2-EventThread, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
>> 6) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> 7) Thread[id=25470, name=Connection evictor, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
>> at java.lang.Thread.run(Thread.java:748)
>> 8) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> 9) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> 10) Thread[id=25521, name=zkCallback-5597-thread-2-processing-n:127.0.0.1:63757_solr, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)
>> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
>> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:494)
>> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:437)
>> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:424)
>> at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:788)
>> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
>> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
>> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)
>> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
>> at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268)
>> at org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$231/1095815815.run(Unknown Source)
>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
>> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$114/259134832.run(Unknown Source)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)
>>
>>
>> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest
>>
>> Error Message:
>> There are still zombie threads that couldn't be terminated: 1) Thread[id=25489, name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) 2) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=25521, name=zkCallback-5597-thread-2, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
>>
>> Stack Trace:
>> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
>> 1) Thread[id=25489, name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at java.lang.Thread.sleep(Native Method)
>> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
>> 2) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> 3) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> 4) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> 5) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> 6) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> 7) Thread[id=25521, name=zkCallback-5597-thread-2, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> at sun.misc.Unsafe.park(Native Method)
>> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> at java.lang.Thread.run(Thread.java:748)
>> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)
>>
>>
>> FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger
>>
>> Error Message:
>>
>>
>> Stack Trace:
>> java.lang.NullPointerException
>> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:C4754A65A7D93050]:0)
>> at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger(TriggerIntegrationTest.java:1565)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
>> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
>> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
>> at java.lang.Thread.run(Thread.java:748)
>>
>>
>>
>>
>> Build Log:
>> [...truncated 11654 lines...]
>> [junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
>> [junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/init-core-data-001
>> [junit4] 2> 16244 WARN (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
>> [junit4] 2> 16244 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
>> [junit4] 2> 16249 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
>> [junit4] 2> 16249 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
>> [junit4] 2> 16252 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/tempDir-001
>> [junit4] 2> 16267 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
>> [junit4] 2> 16270 INFO (Thread-14) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
>> [junit4] 2> 16270 INFO (Thread-14) [ ] o.a.s.c.ZkTestServer Starting server
>> [junit4] 2> 16481 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:53024
>> [junit4] 2> 16566 ERROR (Thread-14) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
>> [junit4] 2> 16670 INFO (zkConnectionManagerCallback-10-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
>> [junit4] 2> 16754 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
>> [junit4] 2> 16755 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
>> [junit4] 2> 16757 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
>> [junit4] 2> 16757 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
>> [junit4] 2> 16757 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
>> [junit4] 2> 16757 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.session Scavenging every 660000ms
>> [junit4] 2> 16757 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
>> [junit4] 2> 16757 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.session Scavenging every 660000ms
>> [junit4] 2> 16758 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@d63a739{/solr,null,AVAILABLE}
>> [junit4] 2> 16758 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1257398a{/solr,null,AVAILABLE}
>> [junit4] 2> 16760 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@356cfda{HTTP/1.1,[http/1.1]}{127.0.0.1:53029}
>> [junit4] 2> 16761 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.Server Started @20200ms
>> [junit4] 2> 16760 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@1e6e5b7a{HTTP/1.1,[http/1.1]}{127.0.0.1:53028}
>> [junit4] 2> 16761 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.Server Started @20200ms
>> [junit4] 2> 16761 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53029}
>> [junit4] 2> 16761 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53028}
>> [junit4] 2> 16761 ERROR (jetty-launcher-7-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
>> [junit4] 2> 16761 ERROR (jetty-launcher-7-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
>> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
>> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
>> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
>> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-25T13:19:30.715Z
>> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
>> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
>> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
>> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-25T13:19:30.715Z
>> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
>> [junit4] 2> 16766 INFO (zkConnectionManagerCallback-13-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
>> [junit4] 2> 16767 INFO (zkConnectionManagerCallback-14-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
>> [junit4] 2> 16769 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>> [junit4] 2> 16769 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>> [junit4] 2> 16796 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
>> [junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1612d7928310002, likely client has closed socket
>> [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
>> [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
>> [junit4] 2> at java.lang.Thread.run(Thread.java:748)
>> [junit4] 2> 16800 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53024/solr
>> [junit4] 2> 16800 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
>> [junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1612d7928310001, likely client has closed socket
>> [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
>> [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
>> [junit4] 2> at java.lang.Thread.run(Thread.java:748)
>> [junit4] 2> 16804 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53024/solr
>> [junit4] 2> 16842 INFO (zkConnectionManagerCallback-20-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
>> [junit4] 2> 16849 INFO (zkConnectionManagerCallback-22-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
>> [junit4] 2> 16853 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
>> [junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1612d7928310004, likely client has closed socket
>> [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
>> [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
>> [junit4] 2> at java.lang.Thread.run(Thread.java:748)
>> [junit4] 2> 16867 INFO (zkConnectionManagerCallback-24-thread-1-processing-n:127.0.0.1:53028_solr) [n:127.0.0.1:53028_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
>> [junit4] 2> 16868 INFO (zkConnectionManagerCallback-26-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
>> [junit4] 2> 17135 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.Overseer Overseer (id=null) closing
>> [junit4] 2> 17135 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.c.Overseer Overseer (id=null) closing
>> [junit4] 2> 17138 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:53029_solr
>> [junit4] 2> 17139 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53028_solr
>> [junit4] 2> 17142 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.Overseer Overseer (id=99410665167454214-127.0.0.1:53029_solr-n_0000000000) starting
>> [junit4] 2> 17173 INFO (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
>> [junit4] 2> 17173 INFO (zkCallback-23-thread-1-processing-n:127.0.0.1:53028_solr) [n:127.0.0.1:53028_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
>> [junit4] 2> 17508 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53029_solr
>> [junit4] 2> 17524 INFO (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
>> [junit4] 2> 17525 INFO (zkCallback-23-thread-1-processing-n:127.0.0.1:53028_solr) [n:127.0.0.1:53028_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
>> [junit4] 2> 17573 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
>> [junit4] 2> 17579 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
>> [junit4] 2> 18329 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:53029_solr, 127.0.0.1:53028_solr]
>> [junit4] 2> 18329 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
>> [junit4] 2> 18329 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
>> [junit4] 2> 18362 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
>> [junit4] 2> 18365 DEBUG (ScheduledTrigger-48-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
>> [junit4] 2> 18371 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53028.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>> [junit4] 2> 18380 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
>> [junit4] 2> 18428 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53029.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>> [junit4] 2> 18489 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53028.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>> [junit4] 2> 18489 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53028.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>> [junit4] 2> 18502 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53029.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>> [junit4] 2> 18502 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53029.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>> [junit4] 2> 18577 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/tempDir-001/node2/.
>> [junit4] 2> 18581 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/tempDir-001/node1/.
>> [junit4] 2> 18697 INFO (zkConnectionManagerCallback-32-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
>> [junit4] 2> 18704 INFO (zkConnectionManagerCallback-36-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
>> [junit4] 2> 18707 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
>> [junit4] 2> 18709 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:53024/solr ready
>> [junit4] 2> 18727 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTrigger
>> [junit4] 2> 18728 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 2
>> [junit4] 2> 18729 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
>> [junit4] 2> 18729 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
>> [junit4] 2> 18734 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
>> [junit4] 2> 18735 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
>> [junit4] 2> 18736 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
>> [junit4] 2> 18747 INFO (qtp657754392-72) [n:127.0.0.1:53028_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
>> [junit4] 2> 18777 INFO (qtp657754392-72) [n:127.0.0.1:53028_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=30
>> [junit4] 2> 18873 INFO (qtp657754392-76) [n:127.0.0.1:53028_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=10
>> [junit4] 2> 18876 INFO (qtp1778631180-70) [n:127.0.0.1:53029_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
>> [junit4] 2> 18877 DEBUG (qtp1778631180-66) [n:127.0.0.1:53029_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
>> [junit4] 2> 18878 INFO (qtp1778631180-66) [n:127.0.0.1:53029_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=65
>> [junit4] 2> 18879 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
>> [junit4] 2> 18880 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:53029_solr, 127.0.0.1:53028_solr]
>> [junit4] 2> 18880 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
>> [junit4] 2> 18883 INFO (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
>> [junit4] 2> 18883 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
>> [junit4] 2> 18884 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
>> [junit4] 2> 18884 DEBUG (ScheduledTrigger-48-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
>> [junit4] 2> 18884 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@1e6e5b7a{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
>> [junit4] 2> 18885 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=950881581
>> [junit4] 2> 18885 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
>> [junit4] 2> 18885 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1634fb6: rootName = solr_53028, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@1fb41f0a
>> [junit4] 2> 18895 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
>> [junit4] 2> 18895 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6b7253ba: rootName = solr_53028, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@43d06aba
>> [junit4] 2> 18900 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
>> [junit4] 2> 18901 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@59b22ff4: rootName = solr_53028, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@2116ae4f
>> [junit4] 2> 18903 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:53028_solr
>> [junit4] 2> 18904 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
>> [junit4] 2> 18904 INFO (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
>> [junit4] 2> 18905 INFO (zkCallback-35-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
>> [junit4] 2> 18904 INFO (zkCallback-23-thread-1-processing-n:127.0.0.1:53028_solr) [n:127.0.0.1:53028_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
>> [junit4] 2> 18905 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
>> [junit4] 2> 18907 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@d63a739{/solr,null,UNAVAILABLE}
>> [junit4] 2> 18908 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.session Stopped scavenging
>> [junit4] 2> 19897 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
>> [junit4] 2> 19897 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:53028_solr
>> [junit4] 2> 20771 INFO (OverseerCollectionConfigSetProcessor-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
>> [junit4] 2> 20900 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
>> [junit4] 2> 21901 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
>> [junit4] 2> 22902 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
>> [junit4] 2> 22902 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:53028_solr]
>> [junit4] 2> 22905 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
>> [junit4] 2> "id":"215c6a28112b6Tztlufkk2v8h7oh4p3og9272t",
>> [junit4] 2> "source":"node_lost_trigger",
>> [junit4] 2> "eventTime":586892827497142,
>> [junit4] 2> "eventType":"NODELOST",
>> [junit4] 2> "properties":{
>> [junit4] 2> "eventTimes":[586892827497142],
>> [junit4] 2> "nodeNames":["127.0.0.1:53028_solr"]}}
>> [junit4] 2> 22905 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_lost_trigger]
>> [junit4] 2> 22910 INFO (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
>> [junit4] 2> 22912 DEBUG (AutoscalingActionExecutor-49-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
>> [junit4] 2> "id":"215c6a28112b6Tztlufkk2v8h7oh4p3og9272t",
>> [junit4] 2> "source":"node_lost_trigger",
>> [junit4] 2> "eventTime":586892827497142,
>> [junit4] 2> "eventType":"NODELOST",
>> [junit4] 2> "properties":{
>> [junit4] 2> "eventTimes":[586892827497142],
>> [junit4] 2> "_enqueue_time_":586895836278162,
>> [junit4] 2> "nodeNames":["127.0.0.1:53028_solr"]}}
>> [junit4] 2> 22917 DEBUG (AutoscalingActionExecutor-49-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger after 5000ms
>> [junit4] 2> 22917 DEBUG (AutoscalingActionExecutor-49-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.ScheduledTriggers -- processing took 6 ms for event id=215c6a28112b6Tztlufkk2v8h7oh4p3og9272t
>> [junit4] 2> 22928 INFO (qtp1778631180-70) [n:127.0.0.1:53029_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=7
>> [junit4] 2> 22929 DEBUG (qtp1778631180-68) [n:127.0.0.1:53029_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
>> [junit4] 2> 22930 INFO (qtp1778631180-68) [n:127.0.0.1:53029_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=12
>> [junit4] 2> 22931 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
>> [junit4] 2> 22931 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:53029_solr]
>> [junit4] 2> 22932 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
>> [junit4] 2> 22932 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
>> [junit4] 2> 22933 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
>> [junit4] 2> 24933 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTrigger
>> [junit4] 2> 24941 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testContinueTriggersOnOverseerRestart
>> [junit4] 2> 24943 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 5
>> [junit4] 2> 24944 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
>> [junit4] 2> 24945 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
>> [junit4] 2> 24945 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
>> [junit4] 2> 24946 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
>> [junit4] 2> 24946 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
>> [junit4] 2> 24946 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.session Scavenging every 600000ms
>> [junit4] 2> 24947 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@338dbf4b{/solr,null,AVAILABLE}
>> [junit4] 2> 24948 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@12a9a774{HTTP/1.1,[http/1.1]}{127.0.0.1:53065}
>> [junit4] 2> 24948 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.Server Started @28387ms
>> [junit4] 2> 24948 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53065}
>> [junit4] 2> 24948 ERROR (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
>> [junit4] 2> 24948 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
>> [junit4] 2> 24949 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
>> [junit4] 2> 24949 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
>> [junit4] 2> 24949 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-25T13:19:38.902Z
>> [junit4] 2> 24950 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
>> [junit4] 2> 24951 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
>> [junit4] 2> 24951 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
>> [junit4] 2> 24952 INFO (zkConnectionManagerCallback-38-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
>> [junit4] 2> 24953 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
>> [junit4] 2> 24964 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53024/solr
>> [junit4] 2> 24967 INFO (zkConnectionManagerCallback-42-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
>> [junit4] 2> 24971 INFO (zkConnectionManagerCallback-44-thread-1-processing-n:127.0.0.1:53065_solr) [n:127.0.0.1:53065_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
>> [junit4] 2> 24984 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
>> [junit4] 2> 24988 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.c.Overseer Overseer (id=null) closing
>> [junit4] 2> 24992 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53065_solr
>> [junit4] 2> 24993 INFO (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
>> [junit4] 2> 24993 INFO (zkCallback-35-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
>> [junit4] 2> 24994 INFO (zkCallback-43-thread-1-processing-n:127.0.0.1:53065_solr) [n:127.0.0.1:53065_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
>> [junit4] 2> 25107 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53065.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>> [junit4] 2> 25123 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53065.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>> [junit4] 2> 25123 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53065.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
>> [junit4] 2> 25125 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a
>>
>> [...truncated too long message...]
>>
>> pache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
>> [junit4] > at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268)
>> [junit4] > at org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$231/1095815815.run(Unknown Source)
>> [junit4] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> [junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> [junit4] > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
>> [junit4] > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$114/259134832.run(Unknown Source)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> [junit4] > at java.lang.Thread.run(Thread.java:748)
>> [junit4] > at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
>> [junit4] > 1) Thread[id=25489, name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> [junit4] > at java.lang.Thread.sleep(Native Method)
>> [junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
>> [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
>> [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
>> [junit4] > 2) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> [junit4] > at java.lang.Thread.run(Thread.java:748)
>> [junit4] > 3) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>> [junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>> [junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> [junit4] > at java.lang.Thread.run(Thread.java:748)
>> [junit4] > 4) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> [junit4] > at java.lang.Thread.run(Thread.java:748)
>> [junit4] > 5) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> [junit4] > at java.lang.Thread.run(Thread.java:748)
>> [junit4] > 6) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> [junit4] > at java.lang.Thread.run(Thread.java:748)
>> [junit4] > 7) Thread[id=25521, name=zkCallback-5597-thread-2, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
>> [junit4] > at sun.misc.Unsafe.park(Native Method)
>> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
>> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
>> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
>> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> [junit4] > at java.lang.Thread.run(Thread.java:748)
>> [junit4] > at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)
>> [junit4] Completed [561/768 (4!)] on J1 in 24.26s, 1 test, 3 errors <<< FAILURES!
>>
>> [...truncated 37138 lines...]
>> -ecj-javadoc-lint-tests:
>> [mkdir] Created dir: /var/folders/qg/h2dfw5s161s51l2bn79mrb7r0000gn/T/ecj1722876386
>> [ecj-lint] Compiling 19 source files to /var/folders/qg/h2dfw5s161s51l2bn79mrb7r0000gn/T/ecj1722876386
>> [ecj-lint] ----------
>> [ecj-lint] 1. ERROR in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/RandomPlaneTest.java (at line 20)
>> [ecj-lint] import java.util.ArrayList;
>> [ecj-lint] ^^^^^^^^^^^^^^^^^^^
>> [ecj-lint] The import java.util.ArrayList is never used
>> [ecj-lint] ----------
>> [ecj-lint] 2. ERROR in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/RandomPlaneTest.java (at line 21)
>> [ecj-lint] import java.util.List;
>> [ecj-lint] ^^^^^^^^^^^^^^
>> [ecj-lint] The import java.util.List is never used
>> [ecj-lint] ----------
>> [ecj-lint] 2 problems (2 errors)
>>
>> BUILD FAILED
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:836: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:101: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build.xml:208: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:2281: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:2112: The following error occurred while executing this line:
>> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:2145: Compile failed; see the compiler error output for details.
>>
>> Total time: 81 minutes 26 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Archiving artifacts
>> [WARNINGS] Skipping publisher since build result is FAILURE
>> Recording test results
>> Email was triggered for: Failure - Any
>> Sending email for trigger: Failure - Any
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: [JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build #
4407 - Failure!
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
This is a strange failure. The logs are full of messages "invalid API
spec:" which is a misleading message. The message is logged if the
call to ValidatingJsonMap.class.getClassLoader().getResourceAsStream(resourceName);
returns a null input stream. I'll fix the message but I am not sure
why the call failed. I looked through the last two mac os failures and
I don't see this exception. Does anyone have any ideas?
On Thu, Jan 25, 2018 at 7:53 PM, Policeman Jenkins Server
<je...@thetaphi.de> wrote:
> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4407/
> Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC
>
> 8 tests failed.
> FAILED: org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testSpecificConfigsets
>
> Error Message:
> KeeperErrorCode = NoNode for /collections/withconfigset2
>
> Stack Trace:
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/withconfigset2
> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:530732B00F155C13]:0)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
> at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
> at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
> at org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testSpecificConfigsets(CollectionsAPIDistributedZkTest.java:373)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at java.lang.Thread.run(Thread.java:748)
>
>
> FAILED: org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCollectionReload
>
> Error Message:
> Could not find collection : reloaded_collection
>
> Stack Trace:
> org.apache.solr.common.SolrException: Could not find collection : reloaded_collection
> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:7FC051D42DBE4980]:0)
> at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:118)
> at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:247)
> at org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.collectStartTimes(CollectionsAPIDistributedZkTest.java:550)
> at org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCollectionReload(CollectionsAPIDistributedZkTest.java:491)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at java.lang.Thread.run(Thread.java:748)
>
>
> FAILED: org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection
>
> Error Message:
> Error from server at http://127.0.0.1:63452/solr: ADDREPLICA failed to create replica
>
> Stack Trace:
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:63452/solr: ADDREPLICA failed to create replica
> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:54F9110F4FEBC6D7]:0)
> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
> at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
> at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
> at org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection(CollectionsAPIDistributedZkTest.java:170)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at java.lang.Thread.run(Thread.java:748)
>
>
> FAILED: org.apache.solr.cloud.api.collections.CustomCollectionTest.testRouteFieldForImplicitRouter
>
> Error Message:
> Collection not found: withShardField
>
> Stack Trace:
> org.apache.solr.common.SolrException: Collection not found: withShardField
> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:2B29957854C829EF]:0)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:851)
> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
> at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
> at org.apache.solr.cloud.api.collections.CustomCollectionTest.testRouteFieldForImplicitRouter(CustomCollectionTest.java:142)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at java.lang.Thread.run(Thread.java:748)
>
>
> FAILED: org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple
>
> Error Message:
>
>
> Stack Trace:
> java.lang.NullPointerException
> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:46CA5914DFC232CE]:0)
> at org.apache.solr.client.solrj.embedded.JettySolrRunner.getNodeName(JettySolrRunner.java:345)
> at org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple(AutoAddReplicasPlanActionTest.java:65)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at java.lang.Thread.run(Thread.java:748)
>
>
> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest
>
> Error Message:
> 10 threads leaked from SUITE scope at org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest: 1) Thread[id=25475, name=Connection evictor, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=25489, name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) 3) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=25490, name=jetty-launcher-5573-thread-2-EventThread, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) 6) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=25470, name=Connection evictor, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 8) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 9) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 10) Thread[id=25521, name=zkCallback-5597-thread-2-processing-n:127.0.0.1:63757_solr, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:494) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:437) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:424) at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:788) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135) at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56) at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348) at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268) at org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$231/1095815815.run(Unknown Source) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$114/259134832.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest:
> 1) Thread[id=25475, name=Connection evictor, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at java.lang.Thread.sleep(Native Method)
> at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
> at java.lang.Thread.run(Thread.java:748)
> 2) Thread[id=25489, name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at java.lang.Thread.sleep(Native Method)
> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
> 3) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 4) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 5) Thread[id=25490, name=jetty-launcher-5573-thread-2-EventThread, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
> 6) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 7) Thread[id=25470, name=Connection evictor, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at java.lang.Thread.sleep(Native Method)
> at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
> at java.lang.Thread.run(Thread.java:748)
> 8) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 9) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 10) Thread[id=25521, name=zkCallback-5597-thread-2-processing-n:127.0.0.1:63757_solr, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at java.lang.Thread.sleep(Native Method)
> at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)
> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:494)
> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:437)
> at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:424)
> at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:788)
> at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
> at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)
> at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
> at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268)
> at org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$231/1095815815.run(Unknown Source)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$114/259134832.run(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)
>
>
> FAILED: junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest
>
> Error Message:
> There are still zombie threads that couldn't be terminated: 1) Thread[id=25489, name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) 2) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=25521, name=zkCallback-5597-thread-2, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
>
> Stack Trace:
> com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
> 1) Thread[id=25489, name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at java.lang.Thread.sleep(Native Method)
> at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
> at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
> 2) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 3) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 4) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 5) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 6) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 7) Thread[id=25521, name=zkCallback-5597-thread-2, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)
>
>
> FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger
>
> Error Message:
>
>
> Stack Trace:
> java.lang.NullPointerException
> at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:C4754A65A7D93050]:0)
> at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger(TriggerIntegrationTest.java:1565)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
> at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
> at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
> at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
> at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
> at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
> at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
> at java.lang.Thread.run(Thread.java:748)
>
>
>
>
> Build Log:
> [...truncated 11654 lines...]
> [junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
> [junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/init-core-data-001
> [junit4] 2> 16244 WARN (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
> [junit4] 2> 16244 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
> [junit4] 2> 16249 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
> [junit4] 2> 16249 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
> [junit4] 2> 16252 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/tempDir-001
> [junit4] 2> 16267 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
> [junit4] 2> 16270 INFO (Thread-14) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
> [junit4] 2> 16270 INFO (Thread-14) [ ] o.a.s.c.ZkTestServer Starting server
> [junit4] 2> 16481 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:53024
> [junit4] 2> 16566 ERROR (Thread-14) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
> [junit4] 2> 16670 INFO (zkConnectionManagerCallback-10-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
> [junit4] 2> 16754 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
> [junit4] 2> 16755 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
> [junit4] 2> 16757 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
> [junit4] 2> 16757 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
> [junit4] 2> 16757 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
> [junit4] 2> 16757 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.session Scavenging every 660000ms
> [junit4] 2> 16757 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
> [junit4] 2> 16757 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.session Scavenging every 660000ms
> [junit4] 2> 16758 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@d63a739{/solr,null,AVAILABLE}
> [junit4] 2> 16758 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1257398a{/solr,null,AVAILABLE}
> [junit4] 2> 16760 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@356cfda{HTTP/1.1,[http/1.1]}{127.0.0.1:53029}
> [junit4] 2> 16761 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.Server Started @20200ms
> [junit4] 2> 16760 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@1e6e5b7a{HTTP/1.1,[http/1.1]}{127.0.0.1:53028}
> [junit4] 2> 16761 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.Server Started @20200ms
> [junit4] 2> 16761 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53029}
> [junit4] 2> 16761 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53028}
> [junit4] 2> 16761 ERROR (jetty-launcher-7-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
> [junit4] 2> 16761 ERROR (jetty-launcher-7-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-25T13:19:30.715Z
> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-25T13:19:30.715Z
> [junit4] 2> 16762 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
> [junit4] 2> 16766 INFO (zkConnectionManagerCallback-13-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
> [junit4] 2> 16767 INFO (zkConnectionManagerCallback-14-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
> [junit4] 2> 16769 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
> [junit4] 2> 16769 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
> [junit4] 2> 16796 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
> [junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1612d7928310002, likely client has closed socket
> [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
> [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
> [junit4] 2> at java.lang.Thread.run(Thread.java:748)
> [junit4] 2> 16800 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53024/solr
> [junit4] 2> 16800 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
> [junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1612d7928310001, likely client has closed socket
> [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
> [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
> [junit4] 2> at java.lang.Thread.run(Thread.java:748)
> [junit4] 2> 16804 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53024/solr
> [junit4] 2> 16842 INFO (zkConnectionManagerCallback-20-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
> [junit4] 2> 16849 INFO (zkConnectionManagerCallback-22-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
> [junit4] 2> 16853 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
> [junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1612d7928310004, likely client has closed socket
> [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
> [junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
> [junit4] 2> at java.lang.Thread.run(Thread.java:748)
> [junit4] 2> 16867 INFO (zkConnectionManagerCallback-24-thread-1-processing-n:127.0.0.1:53028_solr) [n:127.0.0.1:53028_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
> [junit4] 2> 16868 INFO (zkConnectionManagerCallback-26-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
> [junit4] 2> 17135 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.Overseer Overseer (id=null) closing
> [junit4] 2> 17135 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.c.Overseer Overseer (id=null) closing
> [junit4] 2> 17138 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:53029_solr
> [junit4] 2> 17139 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53028_solr
> [junit4] 2> 17142 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.Overseer Overseer (id=99410665167454214-127.0.0.1:53029_solr-n_0000000000) starting
> [junit4] 2> 17173 INFO (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
> [junit4] 2> 17173 INFO (zkCallback-23-thread-1-processing-n:127.0.0.1:53028_solr) [n:127.0.0.1:53028_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
> [junit4] 2> 17508 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53029_solr
> [junit4] 2> 17524 INFO (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
> [junit4] 2> 17525 INFO (zkCallback-23-thread-1-processing-n:127.0.0.1:53028_solr) [n:127.0.0.1:53028_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
> [junit4] 2> 17573 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
> [junit4] 2> 17579 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
> [junit4] 2> 18329 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:53029_solr, 127.0.0.1:53028_solr]
> [junit4] 2> 18329 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
> [junit4] 2> 18329 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
> [junit4] 2> 18362 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
> [junit4] 2> 18365 DEBUG (ScheduledTrigger-48-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
> [junit4] 2> 18371 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53028.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
> [junit4] 2> 18380 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
> [junit4] 2> 18428 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53029.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
> [junit4] 2> 18489 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53028.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
> [junit4] 2> 18489 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53028.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
> [junit4] 2> 18502 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53029.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
> [junit4] 2> 18502 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53029.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
> [junit4] 2> 18577 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/tempDir-001/node2/.
> [junit4] 2> 18581 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/tempDir-001/node1/.
> [junit4] 2> 18697 INFO (zkConnectionManagerCallback-32-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
> [junit4] 2> 18704 INFO (zkConnectionManagerCallback-36-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
> [junit4] 2> 18707 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
> [junit4] 2> 18709 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:53024/solr ready
> [junit4] 2> 18727 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTrigger
> [junit4] 2> 18728 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 2
> [junit4] 2> 18729 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
> [junit4] 2> 18729 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
> [junit4] 2> 18734 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
> [junit4] 2> 18735 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
> [junit4] 2> 18736 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
> [junit4] 2> 18747 INFO (qtp657754392-72) [n:127.0.0.1:53028_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
> [junit4] 2> 18777 INFO (qtp657754392-72) [n:127.0.0.1:53028_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=30
> [junit4] 2> 18873 INFO (qtp657754392-76) [n:127.0.0.1:53028_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=10
> [junit4] 2> 18876 INFO (qtp1778631180-70) [n:127.0.0.1:53029_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
> [junit4] 2> 18877 DEBUG (qtp1778631180-66) [n:127.0.0.1:53029_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
> [junit4] 2> 18878 INFO (qtp1778631180-66) [n:127.0.0.1:53029_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=65
> [junit4] 2> 18879 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
> [junit4] 2> 18880 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:53029_solr, 127.0.0.1:53028_solr]
> [junit4] 2> 18880 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
> [junit4] 2> 18883 INFO (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
> [junit4] 2> 18883 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
> [junit4] 2> 18884 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
> [junit4] 2> 18884 DEBUG (ScheduledTrigger-48-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
> [junit4] 2> 18884 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@1e6e5b7a{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
> [junit4] 2> 18885 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=950881581
> [junit4] 2> 18885 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
> [junit4] 2> 18885 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1634fb6: rootName = solr_53028, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@1fb41f0a
> [junit4] 2> 18895 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
> [junit4] 2> 18895 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6b7253ba: rootName = solr_53028, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@43d06aba
> [junit4] 2> 18900 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
> [junit4] 2> 18901 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@59b22ff4: rootName = solr_53028, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@2116ae4f
> [junit4] 2> 18903 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:53028_solr
> [junit4] 2> 18904 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
> [junit4] 2> 18904 INFO (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
> [junit4] 2> 18905 INFO (zkCallback-35-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
> [junit4] 2> 18904 INFO (zkCallback-23-thread-1-processing-n:127.0.0.1:53028_solr) [n:127.0.0.1:53028_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
> [junit4] 2> 18905 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
> [junit4] 2> 18907 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@d63a739{/solr,null,UNAVAILABLE}
> [junit4] 2> 18908 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.session Stopped scavenging
> [junit4] 2> 19897 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
> [junit4] 2> 19897 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:53028_solr
> [junit4] 2> 20771 INFO (OverseerCollectionConfigSetProcessor-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
> [junit4] 2> 20900 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
> [junit4] 2> 21901 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
> [junit4] 2> 22902 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
> [junit4] 2> 22902 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:53028_solr]
> [junit4] 2> 22905 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
> [junit4] 2> "id":"215c6a28112b6Tztlufkk2v8h7oh4p3og9272t",
> [junit4] 2> "source":"node_lost_trigger",
> [junit4] 2> "eventTime":586892827497142,
> [junit4] 2> "eventType":"NODELOST",
> [junit4] 2> "properties":{
> [junit4] 2> "eventTimes":[586892827497142],
> [junit4] 2> "nodeNames":["127.0.0.1:53028_solr"]}}
> [junit4] 2> 22905 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_lost_trigger]
> [junit4] 2> 22910 INFO (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
> [junit4] 2> 22912 DEBUG (AutoscalingActionExecutor-49-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
> [junit4] 2> "id":"215c6a28112b6Tztlufkk2v8h7oh4p3og9272t",
> [junit4] 2> "source":"node_lost_trigger",
> [junit4] 2> "eventTime":586892827497142,
> [junit4] 2> "eventType":"NODELOST",
> [junit4] 2> "properties":{
> [junit4] 2> "eventTimes":[586892827497142],
> [junit4] 2> "_enqueue_time_":586895836278162,
> [junit4] 2> "nodeNames":["127.0.0.1:53028_solr"]}}
> [junit4] 2> 22917 DEBUG (AutoscalingActionExecutor-49-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger after 5000ms
> [junit4] 2> 22917 DEBUG (AutoscalingActionExecutor-49-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.ScheduledTriggers -- processing took 6 ms for event id=215c6a28112b6Tztlufkk2v8h7oh4p3og9272t
> [junit4] 2> 22928 INFO (qtp1778631180-70) [n:127.0.0.1:53029_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=7
> [junit4] 2> 22929 DEBUG (qtp1778631180-68) [n:127.0.0.1:53029_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
> [junit4] 2> 22930 INFO (qtp1778631180-68) [n:127.0.0.1:53029_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=12
> [junit4] 2> 22931 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
> [junit4] 2> 22931 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:53029_solr]
> [junit4] 2> 22932 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
> [junit4] 2> 22932 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
> [junit4] 2> 22933 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
> [junit4] 2> 24933 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTrigger
> [junit4] 2> 24941 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testContinueTriggersOnOverseerRestart
> [junit4] 2> 24943 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 5
> [junit4] 2> 24944 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
> [junit4] 2> 24945 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
> [junit4] 2> 24945 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
> [junit4] 2> 24946 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
> [junit4] 2> 24946 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
> [junit4] 2> 24946 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.session Scavenging every 600000ms
> [junit4] 2> 24947 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@338dbf4b{/solr,null,AVAILABLE}
> [junit4] 2> 24948 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@12a9a774{HTTP/1.1,[http/1.1]}{127.0.0.1:53065}
> [junit4] 2> 24948 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.Server Started @28387ms
> [junit4] 2> 24948 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53065}
> [junit4] 2> 24948 ERROR (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
> [junit4] 2> 24948 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
> [junit4] 2> 24949 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
> [junit4] 2> 24949 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
> [junit4] 2> 24949 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-25T13:19:38.902Z
> [junit4] 2> 24950 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
> [junit4] 2> 24951 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
> [junit4] 2> 24951 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
> [junit4] 2> 24952 INFO (zkConnectionManagerCallback-38-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
> [junit4] 2> 24953 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
> [junit4] 2> 24964 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53024/solr
> [junit4] 2> 24967 INFO (zkConnectionManagerCallback-42-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
> [junit4] 2> 24971 INFO (zkConnectionManagerCallback-44-thread-1-processing-n:127.0.0.1:53065_solr) [n:127.0.0.1:53065_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
> [junit4] 2> 24984 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
> [junit4] 2> 24988 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.c.Overseer Overseer (id=null) closing
> [junit4] 2> 24992 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53065_solr
> [junit4] 2> 24993 INFO (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
> [junit4] 2> 24993 INFO (zkCallback-35-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
> [junit4] 2> 24994 INFO (zkCallback-43-thread-1-processing-n:127.0.0.1:53065_solr) [n:127.0.0.1:53065_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
> [junit4] 2> 25107 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53065.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
> [junit4] 2> 25123 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53065.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
> [junit4] 2> 25123 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53065.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
> [junit4] 2> 25125 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a
>
> [...truncated too long message...]
>
> pache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
> [junit4] > at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268)
> [junit4] > at org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$231/1095815815.run(Unknown Source)
> [junit4] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [junit4] > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
> [junit4] > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$114/259134832.run(Unknown Source)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [junit4] > at java.lang.Thread.run(Thread.java:748)
> [junit4] > at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
> [junit4] > 1) Thread[id=25489, name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> [junit4] > at java.lang.Thread.sleep(Native Method)
> [junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
> [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
> [junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
> [junit4] > 2) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> [junit4] > at sun.misc.Unsafe.park(Native Method)
> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [junit4] > at java.lang.Thread.run(Thread.java:748)
> [junit4] > 3) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> [junit4] > at sun.misc.Unsafe.park(Native Method)
> [junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> [junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> [junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [junit4] > at java.lang.Thread.run(Thread.java:748)
> [junit4] > 4) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> [junit4] > at sun.misc.Unsafe.park(Native Method)
> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [junit4] > at java.lang.Thread.run(Thread.java:748)
> [junit4] > 5) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> [junit4] > at sun.misc.Unsafe.park(Native Method)
> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [junit4] > at java.lang.Thread.run(Thread.java:748)
> [junit4] > 6) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> [junit4] > at sun.misc.Unsafe.park(Native Method)
> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [junit4] > at java.lang.Thread.run(Thread.java:748)
> [junit4] > 7) Thread[id=25521, name=zkCallback-5597-thread-2, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
> [junit4] > at sun.misc.Unsafe.park(Native Method)
> [junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
> [junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
> [junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> [junit4] > at java.lang.Thread.run(Thread.java:748)
> [junit4] > at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)
> [junit4] Completed [561/768 (4!)] on J1 in 24.26s, 1 test, 3 errors <<< FAILURES!
>
> [...truncated 37138 lines...]
> -ecj-javadoc-lint-tests:
> [mkdir] Created dir: /var/folders/qg/h2dfw5s161s51l2bn79mrb7r0000gn/T/ecj1722876386
> [ecj-lint] Compiling 19 source files to /var/folders/qg/h2dfw5s161s51l2bn79mrb7r0000gn/T/ecj1722876386
> [ecj-lint] ----------
> [ecj-lint] 1. ERROR in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/RandomPlaneTest.java (at line 20)
> [ecj-lint] import java.util.ArrayList;
> [ecj-lint] ^^^^^^^^^^^^^^^^^^^
> [ecj-lint] The import java.util.ArrayList is never used
> [ecj-lint] ----------
> [ecj-lint] 2. ERROR in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/RandomPlaneTest.java (at line 21)
> [ecj-lint] import java.util.List;
> [ecj-lint] ^^^^^^^^^^^^^^
> [ecj-lint] The import java.util.List is never used
> [ecj-lint] ----------
> [ecj-lint] 2 problems (2 errors)
>
> BUILD FAILED
> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:836: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:101: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build.xml:208: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:2281: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:2112: The following error occurred while executing this line:
> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:2145: Compile failed; see the compiler error output for details.
>
> Total time: 81 minutes 26 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> [WARNINGS] Skipping publisher since build result is FAILURE
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
--
Regards,
Shalin Shekhar Mangar.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4407
- Failure!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4407/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC
8 tests failed.
FAILED: org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testSpecificConfigsets
Error Message:
KeeperErrorCode = NoNode for /collections/withconfigset2
Stack Trace:
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/withconfigset2
at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:530732B00F155C13]:0)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1212)
at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:339)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:339)
at org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testSpecificConfigsets(CollectionsAPIDistributedZkTest.java:373)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCollectionReload
Error Message:
Could not find collection : reloaded_collection
Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : reloaded_collection
at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:7FC051D42DBE4980]:0)
at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:118)
at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:247)
at org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.collectStartTimes(CollectionsAPIDistributedZkTest.java:550)
at org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.testCollectionReload(CollectionsAPIDistributedZkTest.java:491)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection
Error Message:
Error from server at http://127.0.0.1:63452/solr: ADDREPLICA failed to create replica
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:63452/solr: ADDREPLICA failed to create replica
at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:54F9110F4FEBC6D7]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1104)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.cloud.api.collections.CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection(CollectionsAPIDistributedZkTest.java:170)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.api.collections.CustomCollectionTest.testRouteFieldForImplicitRouter
Error Message:
Collection not found: withShardField
Stack Trace:
org.apache.solr.common.SolrException: Collection not found: withShardField
at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:2B29957854C829EF]:0)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:851)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:233)
at org.apache.solr.cloud.api.collections.CustomCollectionTest.testRouteFieldForImplicitRouter(CustomCollectionTest.java:142)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple
Error Message:
Stack Trace:
java.lang.NullPointerException
at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:46CA5914DFC232CE]:0)
at org.apache.solr.client.solrj.embedded.JettySolrRunner.getNodeName(JettySolrRunner.java:345)
at org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest.testSimple(AutoAddReplicasPlanActionTest.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest
Error Message:
10 threads leaked from SUITE scope at org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest: 1) Thread[id=25475, name=Connection evictor, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=25489, name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) 3) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=25490, name=jetty-launcher-5573-thread-2-EventThread, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501) 6) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=25470, name=Connection evictor, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 8) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 9) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 10) Thread[id=25521, name=zkCallback-5597-thread-2-processing-n:127.0.0.1:63757_solr, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:494) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:437) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:424) at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:788) at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170) at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135) at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56) at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348) at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268) at org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$231/1095815815.run(Unknown Source) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$114/259134832.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest:
1) Thread[id=25475, name=Connection evictor, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
2) Thread[id=25489, name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
3) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
4) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
5) Thread[id=25490, name=jetty-launcher-5573-thread-2-EventThread, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:501)
6) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
7) Thread[id=25470, name=Connection evictor, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
8) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
9) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
10) Thread[id=25521, name=zkCallback-5597-thread-2-processing-n:127.0.0.1:63757_solr, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:494)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:437)
at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:424)
at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:788)
at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:56)
at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268)
at org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$231/1095815815.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$114/259134832.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.AutoAddReplicasPlanActionTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=25489, name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) 2) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 7) Thread[id=25521, name=zkCallback-5597-thread-2, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=25489, name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
2) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
3) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
4) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
5) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
6) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
7) Thread[id=25521, name=zkCallback-5597-thread-2, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)
FAILED: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger
Error Message:
Stack Trace:
java.lang.NullPointerException
at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F:C4754A65A7D93050]:0)
at org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testMetricTrigger(TriggerIntegrationTest.java:1565)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 11654 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.TriggerIntegrationTest
[junit4] 2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/init-core-data-001
[junit4] 2> 16244 WARN (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
[junit4] 2> 16244 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 16249 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
[junit4] 2> 16249 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 16252 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/tempDir-001
[junit4] 2> 16267 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 16270 INFO (Thread-14) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 16270 INFO (Thread-14) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 16481 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:53024
[junit4] 2> 16566 ERROR (Thread-14) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 16670 INFO (zkConnectionManagerCallback-10-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 16754 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 16755 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 16757 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 16757 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 16757 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 16757 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 16757 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 16757 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 16758 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@d63a739{/solr,null,AVAILABLE}
[junit4] 2> 16758 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1257398a{/solr,null,AVAILABLE}
[junit4] 2> 16760 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@356cfda{HTTP/1.1,[http/1.1]}{127.0.0.1:53029}
[junit4] 2> 16761 INFO (jetty-launcher-7-thread-1) [ ] o.e.j.s.Server Started @20200ms
[junit4] 2> 16760 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@1e6e5b7a{HTTP/1.1,[http/1.1]}{127.0.0.1:53028}
[junit4] 2> 16761 INFO (jetty-launcher-7-thread-2) [ ] o.e.j.s.Server Started @20200ms
[junit4] 2> 16761 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53029}
[junit4] 2> 16761 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53028}
[junit4] 2> 16761 ERROR (jetty-launcher-7-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 16761 ERROR (jetty-launcher-7-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 16762 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 16762 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 16762 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 16762 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-25T13:19:30.715Z
[junit4] 2> 16762 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 16762 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 16762 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 16762 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-25T13:19:30.715Z
[junit4] 2> 16762 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
[junit4] 2> 16766 INFO (zkConnectionManagerCallback-13-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 16767 INFO (zkConnectionManagerCallback-14-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 16769 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 16769 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 16796 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1612d7928310002, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 16800 INFO (jetty-launcher-7-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53024/solr
[junit4] 2> 16800 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1612d7928310001, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 16804 INFO (jetty-launcher-7-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53024/solr
[junit4] 2> 16842 INFO (zkConnectionManagerCallback-20-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 16849 INFO (zkConnectionManagerCallback-22-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 16853 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn caught end of stream exception
[junit4] 2> EndOfStreamException: Unable to read additional data from client sessionid 0x1612d7928310004, likely client has closed socket
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:239)
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:203)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 16867 INFO (zkConnectionManagerCallback-24-thread-1-processing-n:127.0.0.1:53028_solr) [n:127.0.0.1:53028_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 16868 INFO (zkConnectionManagerCallback-26-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 17135 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 17135 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 17138 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:53029_solr
[junit4] 2> 17139 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53028_solr
[junit4] 2> 17142 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.Overseer Overseer (id=99410665167454214-127.0.0.1:53029_solr-n_0000000000) starting
[junit4] 2> 17173 INFO (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 17173 INFO (zkCallback-23-thread-1-processing-n:127.0.0.1:53028_solr) [n:127.0.0.1:53028_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 17508 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53029_solr
[junit4] 2> 17524 INFO (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 17525 INFO (zkCallback-23-thread-1-processing-n:127.0.0.1:53028_solr) [n:127.0.0.1:53028_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 17573 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Adding .autoAddReplicas trigger
[junit4] 2> 17579 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 18329 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:53029_solr, 127.0.0.1:53028_solr]
[junit4] 2> 18329 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
[junit4] 2> 18329 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 18362 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 18365 DEBUG (ScheduledTrigger-48-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 18371 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53028.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
[junit4] 2> 18380 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 18428 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53029.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
[junit4] 2> 18489 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53028.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
[junit4] 2> 18489 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53028.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
[junit4] 2> 18502 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53029.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
[junit4] 2> 18502 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53029.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
[junit4] 2> 18577 INFO (jetty-launcher-7-thread-2) [n:127.0.0.1:53028_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/tempDir-001/node2/.
[junit4] 2> 18581 INFO (jetty-launcher-7-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.TriggerIntegrationTest_7E797DEAF831E61F-001/tempDir-001/node1/.
[junit4] 2> 18697 INFO (zkConnectionManagerCallback-32-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 18704 INFO (zkConnectionManagerCallback-36-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 18707 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 18709 INFO (SUITE-TriggerIntegrationTest-seed#[7E797DEAF831E61F]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:53024/solr ready
[junit4] 2> 18727 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTrigger
[junit4] 2> 18728 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 2
[junit4] 2> 18729 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 18729 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 18734 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 18735 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 18736 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 18747 INFO (qtp657754392-72) [n:127.0.0.1:53028_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 18777 INFO (qtp657754392-72) [n:127.0.0.1:53028_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=30
[junit4] 2> 18873 INFO (qtp657754392-76) [n:127.0.0.1:53028_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=10
[junit4] 2> 18876 INFO (qtp1778631180-70) [n:127.0.0.1:53029_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
[junit4] 2> 18877 DEBUG (qtp1778631180-66) [n:127.0.0.1:53029_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 18878 INFO (qtp1778631180-66) [n:127.0.0.1:53029_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=65
[junit4] 2> 18879 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 18880 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:53029_solr, 127.0.0.1:53028_solr]
[junit4] 2> 18880 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 18883 INFO (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.TriggerIntegrationTest TestTriggerAction init
[junit4] 2> 18883 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 18884 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 18884 DEBUG (ScheduledTrigger-48-thread-1) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
[junit4] 2> 18884 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@1e6e5b7a{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 18885 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=950881581
[junit4] 2> 18885 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 18885 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1634fb6: rootName = solr_53028, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@1fb41f0a
[junit4] 2> 18895 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 18895 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6b7253ba: rootName = solr_53028, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@43d06aba
[junit4] 2> 18900 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 18901 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@59b22ff4: rootName = solr_53028, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@2116ae4f
[junit4] 2> 18903 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:53028_solr
[junit4] 2> 18904 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 18904 INFO (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 18905 INFO (zkCallback-35-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 18904 INFO (zkCallback-23-thread-1-processing-n:127.0.0.1:53028_solr) [n:127.0.0.1:53028_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 18905 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 18907 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@d63a739{/solr,null,UNAVAILABLE}
[junit4] 2> 18908 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 19897 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 19897 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:53028_solr
[junit4] 2> 20771 INFO (OverseerCollectionConfigSetProcessor-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 20900 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 21901 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 22902 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 22902 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:53028_solr]
[junit4] 2> 22905 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"215c6a28112b6Tztlufkk2v8h7oh4p3og9272t",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":586892827497142,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[586892827497142],
[junit4] 2> "nodeNames":["127.0.0.1:53028_solr"]}}
[junit4] 2> 22905 DEBUG (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [node_lost_trigger]
[junit4] 2> 22910 INFO (ScheduledTrigger-48-thread-2) [n:127.0.0.1:53029_solr ] o.a.s.c.a.SystemLogListener Collection .system does not exist, disabling logging.
[junit4] 2> 22912 DEBUG (AutoscalingActionExecutor-49-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"215c6a28112b6Tztlufkk2v8h7oh4p3og9272t",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":586892827497142,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[586892827497142],
[junit4] 2> "_enqueue_time_":586895836278162,
[junit4] 2> "nodeNames":["127.0.0.1:53028_solr"]}}
[junit4] 2> 22917 DEBUG (AutoscalingActionExecutor-49-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger after 5000ms
[junit4] 2> 22917 DEBUG (AutoscalingActionExecutor-49-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.ScheduledTriggers -- processing took 6 ms for event id=215c6a28112b6Tztlufkk2v8h7oh4p3og9272t
[junit4] 2> 22928 INFO (qtp1778631180-70) [n:127.0.0.1:53029_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=7
[junit4] 2> 22929 DEBUG (qtp1778631180-68) [n:127.0.0.1:53029_solr ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 22930 INFO (qtp1778631180-68) [n:127.0.0.1:53029_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=12
[junit4] 2> 22931 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 22931 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:53029_solr]
[junit4] 2> 22932 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 22932 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 22933 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 24933 INFO (TEST-TriggerIntegrationTest.testNodeLostTrigger-seed#[7E797DEAF831E61F]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTrigger
[junit4] 2> 24941 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testContinueTriggersOnOverseerRestart
[junit4] 2> 24943 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.a.TriggerIntegrationTest /autoscaling.json reset, new znode version 5
[junit4] 2> 24944 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-22T05:27:37+08:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 24945 DEBUG (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
[junit4] 2> 24945 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 24946 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 24946 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 24946 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.session Scavenging every 600000ms
[junit4] 2> 24947 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@338dbf4b{/solr,null,AVAILABLE}
[junit4] 2> 24948 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@12a9a774{HTTP/1.1,[http/1.1]}{127.0.0.1:53065}
[junit4] 2> 24948 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.e.j.s.Server Started @28387ms
[junit4] 2> 24948 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53065}
[junit4] 2> 24948 ERROR (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 24948 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 24949 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 24949 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 24949 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-01-25T13:19:38.902Z
[junit4] 2> 24950 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 24951 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 24951 DEBUG (OverseerAutoScalingTriggerThread-99410665167454214-127.0.0.1:53029_solr-n_0000000000) [n:127.0.0.1:53029_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
[junit4] 2> 24952 INFO (zkConnectionManagerCallback-38-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 24953 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 24964 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53024/solr
[junit4] 2> 24967 INFO (zkConnectionManagerCallback-42-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 24971 INFO (zkConnectionManagerCallback-44-thread-1-processing-n:127.0.0.1:53065_solr) [n:127.0.0.1:53065_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 24984 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 24988 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 24992 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53065_solr
[junit4] 2> 24993 INFO (zkCallback-25-thread-1-processing-n:127.0.0.1:53029_solr) [n:127.0.0.1:53029_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 24993 INFO (zkCallback-35-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 24994 INFO (zkCallback-43-thread-1-processing-n:127.0.0.1:53065_solr) [n:127.0.0.1:53065_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 25107 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53065.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
[junit4] 2> 25123 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53065.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
[junit4] 2> 25123 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53065.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@752cda5c
[junit4] 2> 25125 INFO (TEST-TriggerIntegrationTest.testContinueTriggersOnOverseerRestart-seed#[7E797DEAF831E61F]) [n:127.0.0.1:53065_solr ] o.a
[...truncated too long message...]
pache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient$1.lambda$process$1(SolrZkClient.java:268)
[junit4] > at org.apache.solr.common.cloud.SolrZkClient$1$$Lambda$231/1095815815.run(Unknown Source)
[junit4] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[junit4] > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[junit4] > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
[junit4] > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$114/259134832.run(Unknown Source)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=25489, name=jetty-launcher-5573-thread-2-SendThread(127.0.0.1:63752), state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
[junit4] > at java.lang.Thread.sleep(Native Method)
[junit4] > at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
[junit4] > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
[junit4] > 2) Thread[id=25523, name=zkCallback-5597-thread-4, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 3) Thread[id=25493, name=zkConnectionManagerCallback-5598-thread-1, state=WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
[junit4] > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
[junit4] > at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 4) Thread[id=25495, name=zkCallback-5597-thread-1, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 5) Thread[id=25522, name=zkCallback-5597-thread-3, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 6) Thread[id=25524, name=zkCallback-5597-thread-5, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > 7) Thread[id=25521, name=zkCallback-5597-thread-2, state=TIMED_WAITING, group=TGRP-AutoAddReplicasPlanActionTest]
[junit4] > at sun.misc.Unsafe.park(Native Method)
[junit4] > at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4] > at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
[junit4] > at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
[junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
[junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] > at __randomizedtesting.SeedInfo.seed([7E797DEAF831E61F]:0)
[junit4] Completed [561/768 (4!)] on J1 in 24.26s, 1 test, 3 errors <<< FAILURES!
[...truncated 37138 lines...]
-ecj-javadoc-lint-tests:
[mkdir] Created dir: /var/folders/qg/h2dfw5s161s51l2bn79mrb7r0000gn/T/ecj1722876386
[ecj-lint] Compiling 19 source files to /var/folders/qg/h2dfw5s161s51l2bn79mrb7r0000gn/T/ecj1722876386
[ecj-lint] ----------
[ecj-lint] 1. ERROR in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/RandomPlaneTest.java (at line 20)
[ecj-lint] import java.util.ArrayList;
[ecj-lint] ^^^^^^^^^^^^^^^^^^^
[ecj-lint] The import java.util.ArrayList is never used
[ecj-lint] ----------
[ecj-lint] 2. ERROR in /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/spatial3d/src/test/org/apache/lucene/spatial3d/geom/RandomPlaneTest.java (at line 21)
[ecj-lint] import java.util.List;
[ecj-lint] ^^^^^^^^^^^^^^
[ecj-lint] The import java.util.List is never used
[ecj-lint] ----------
[ecj-lint] 2 problems (2 errors)
BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:836: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:101: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build.xml:208: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:2281: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:2112: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:2145: Compile failed; see the compiler error output for details.
Total time: 81 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any