You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2018/03/11 01:10:00 UTC
[JENKINS] Lucene-Solr-master-Linux (32bit/jdk1.8.0_162) - Build #
21613 - Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/21613/
Java: 32bit/jdk1.8.0_162 -client -XX:+UseConcMarkSweepGC
3 tests failed.
FAILED: org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup
Error Message:
should be at least one inactive event
Stack Trace:
java.lang.AssertionError: should be at least one inactive event
at __randomizedtesting.SeedInfo.seed([7FE64CD257D2AC02:62CA8CA036918B09]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup(ScheduledMaintenanceTriggerTest.java:218)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup
Error Message:
should be at least one inactive event
Stack Trace:
java.lang.AssertionError: should be at least one inactive event
at __randomizedtesting.SeedInfo.seed([7FE64CD257D2AC02:62CA8CA036918B09]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup(ScheduledMaintenanceTriggerTest.java:218)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup
Error Message:
cleanup action didn't run
Stack Trace:
java.lang.AssertionError: cleanup action didn't run
at __randomizedtesting.SeedInfo.seed([7FE64CD257D2AC02:62CA8CA036918B09]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup(ScheduledMaintenanceTriggerTest.java:197)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 14072 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest
[junit4] 2> 2197696 INFO (SUITE-ScheduledMaintenanceTriggerTest-seed#[7FE64CD257D2AC02]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest_7FE64CD257D2AC02-001/init-core-data-001
[junit4] 2> 2197697 WARN (SUITE-ScheduledMaintenanceTriggerTest-seed#[7FE64CD257D2AC02]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=10 numCloses=10
[junit4] 2> 2197697 INFO (SUITE-ScheduledMaintenanceTriggerTest-seed#[7FE64CD257D2AC02]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 2197699 INFO (SUITE-ScheduledMaintenanceTriggerTest-seed#[7FE64CD257D2AC02]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 2197700 INFO (SUITE-ScheduledMaintenanceTriggerTest-seed#[7FE64CD257D2AC02]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest_7FE64CD257D2AC02-001/tempDir-001
[junit4] 2> 2197700 INFO (SUITE-ScheduledMaintenanceTriggerTest-seed#[7FE64CD257D2AC02]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 2197701 INFO (Thread-6719) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2197701 INFO (Thread-6719) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 2197704 ERROR (Thread-6719) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2197801 INFO (SUITE-ScheduledMaintenanceTriggerTest-seed#[7FE64CD257D2AC02]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:33747
[junit4] 2> 2197805 INFO (zkConnectionManagerCallback-9917-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2197812 INFO (jetty-launcher-9914-thread-1) [ ] o.e.j.s.Server jetty-9.4.8.v20171121, build timestamp: 2017-11-21T22:27:37+01:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8
[junit4] 2> 2197813 INFO (jetty-launcher-9914-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2197813 INFO (jetty-launcher-9914-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2197813 INFO (jetty-launcher-9914-thread-1) [ ] o.e.j.s.session Scavenging every 660000ms
[junit4] 2> 2197813 INFO (jetty-launcher-9914-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@14cabca{/solr,null,AVAILABLE}
[junit4] 2> 2197814 INFO (jetty-launcher-9914-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@73b801{SSL,[ssl, http/1.1]}{127.0.0.1:42071}
[junit4] 2> 2197814 INFO (jetty-launcher-9914-thread-1) [ ] o.e.j.s.Server Started @2199401ms
[junit4] 2> 2197814 INFO (jetty-launcher-9914-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42071}
[junit4] 2> 2197815 ERROR (jetty-launcher-9914-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2197815 INFO (jetty-launcher-9914-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 2197815 INFO (jetty-launcher-9914-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2197815 INFO (jetty-launcher-9914-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2197815 INFO (jetty-launcher-9914-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-03-11T00:46:17.572Z
[junit4] 2> 2197817 INFO (zkConnectionManagerCallback-9919-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2197817 INFO (jetty-launcher-9914-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 2197829 INFO (jetty-launcher-9914-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33747/solr
[junit4] 2> 2197831 INFO (zkConnectionManagerCallback-9923-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2197833 INFO (zkConnectionManagerCallback-9925-thread-1-processing-n:127.0.0.1:42071_solr) [n:127.0.0.1:42071_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2197888 INFO (jetty-launcher-9914-thread-1) [n:127.0.0.1:42071_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2197889 INFO (jetty-launcher-9914-thread-1) [n:127.0.0.1:42071_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:42071_solr
[junit4] 2> 2197889 INFO (jetty-launcher-9914-thread-1) [n:127.0.0.1:42071_solr ] o.a.s.c.Overseer Overseer (id=72114399630393347-127.0.0.1:42071_solr-n_0000000000) starting
[junit4] 2> 2197895 INFO (zkConnectionManagerCallback-9930-thread-1-processing-n:127.0.0.1:42071_solr) [n:127.0.0.1:42071_solr ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2197897 INFO (jetty-launcher-9914-thread-1) [n:127.0.0.1:42071_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33747/solr ready
[junit4] 2> 2197897 DEBUG (OverseerAutoScalingTriggerThread-72114399630393347-127.0.0.1:42071_solr-n_0000000000) [n:127.0.0.1:42071_solr ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
[junit4] 2> 2197898 INFO (jetty-launcher-9914-thread-1) [n:127.0.0.1:42071_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42071_solr
[junit4] 2> 2197898 DEBUG (OverseerAutoScalingTriggerThread-72114399630393347-127.0.0.1:42071_solr-n_0000000000) [n:127.0.0.1:42071_solr ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 2197899 DEBUG (OverseerAutoScalingTriggerThread-72114399630393347-127.0.0.1:42071_solr-n_0000000000) [n:127.0.0.1:42071_solr ] o.a.s.c.a.NodeLostTrigger Initial livenodes: []
[junit4] 2> 2197899 DEBUG (OverseerAutoScalingTriggerThread-72114399630393347-127.0.0.1:42071_solr-n_0000000000) [n:127.0.0.1:42071_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
[junit4] 2> 2197899 DEBUG (OverseerAutoScalingTriggerThread-72114399630393347-127.0.0.1:42071_solr-n_0000000000) [n:127.0.0.1:42071_solr ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 2197900 INFO (zkCallback-9924-thread-1-processing-n:127.0.0.1:42071_solr) [n:127.0.0.1:42071_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2197901 INFO (zkCallback-9929-thread-1-processing-n:127.0.0.1:42071_solr) [n:127.0.0.1:42071_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2197902 DEBUG (OverseerAutoScalingTriggerThread-72114399630393347-127.0.0.1:42071_solr-n_0000000000) [n:127.0.0.1:42071_solr ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 2197902 DEBUG (OverseerAutoScalingTriggerThread-72114399630393347-127.0.0.1:42071_solr-n_0000000000) [n:127.0.0.1:42071_solr ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 2197902 DEBUG (ScheduledTrigger-8487-thread-2) [n:127.0.0.1:42071_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198292 INFO (jetty-launcher-9914-thread-1) [n:127.0.0.1:42071_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42071.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a9f21e
[junit4] 2> 2198308 INFO (jetty-launcher-9914-thread-1) [n:127.0.0.1:42071_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42071.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a9f21e
[junit4] 2> 2198309 INFO (jetty-launcher-9914-thread-1) [n:127.0.0.1:42071_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_42071.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@a9f21e
[junit4] 2> 2198310 INFO (jetty-launcher-9914-thread-1) [n:127.0.0.1:42071_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest_7FE64CD257D2AC02-001/tempDir-001/node1/.
[junit4] 2> 2198327 INFO (zkConnectionManagerCallback-9932-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2198331 INFO (zkConnectionManagerCallback-9936-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2198332 INFO (SUITE-ScheduledMaintenanceTriggerTest-seed#[7FE64CD257D2AC02]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2198332 INFO (SUITE-ScheduledMaintenanceTriggerTest-seed#[7FE64CD257D2AC02]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:33747/solr ready
[junit4] 2> 2198336 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
[junit4] 2> 2198336 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
[junit4] 2> 2198336 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10009_solr]
[junit4] 2> 2198336 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
[junit4] 2> 2198337 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
[junit4] 2> 2198345 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 2198346 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198346 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
[junit4] 2> 2198370 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198392 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198395 INFO (TEST-ScheduledMaintenanceTriggerTest.testTriggerDefaults-seed#[7FE64CD257D2AC02]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testTriggerDefaults
[junit4] 2> 2198396 INFO (TEST-ScheduledMaintenanceTriggerTest.testTriggerDefaults-seed#[7FE64CD257D2AC02]) [ ] o.a.s.c.a.ScheduledMaintenanceTriggerTest {
[junit4] 2> "cluster-preferences":[
[junit4] 2> {
[junit4] 2> "minimize":"cores",
[junit4] 2> "precision":1},
[junit4] 2> {"maximize":"freedisk"}],
[junit4] 2> "triggers":{
[junit4] 2> ".scheduled_maintenance":{
[junit4] 2> "name":".scheduled_maintenance",
[junit4] 2> "startTime":"NOW",
[junit4] 2> "event":"scheduled",
[junit4] 2> "every":"+1DAY",
[junit4] 2> "actions":[
[junit4] 2> {
[junit4] 2> "name":"inactive_shard_plan",
[junit4] 2> "class":"solr.InactiveShardPlanAction"},
[junit4] 2> {
[junit4] 2> "name":"execute_plan",
[junit4] 2> "class":"solr.ExecutePlanAction"}],
[junit4] 2> "enabled":true},
[junit4] 2> ".auto_add_replicas":{
[junit4] 2> "name":".auto_add_replicas",
[junit4] 2> "event":"nodeLost",
[junit4] 2> "waitFor":120,
[junit4] 2> "actions":[
[junit4] 2> {
[junit4] 2> "name":"auto_add_replicas_plan",
[junit4] 2> "class":"solr.AutoAddReplicasPlanAction"},
[junit4] 2> {
[junit4] 2> "name":"execute_plan",
[junit4] 2> "class":"solr.ExecutePlanAction"}],
[junit4] 2> "enabled":true}},
[junit4] 2> "listeners":{
[junit4] 2> ".scheduled_maintenance.system":{
[junit4] 2> "trigger":".scheduled_maintenance",
[junit4] 2> "afterAction":[],
[junit4] 2> "stage":[
[junit4] 2> "STARTED",
[junit4] 2> "ABORTED",
[junit4] 2> "SUCCEEDED",
[junit4] 2> "FAILED",
[junit4] 2> "BEFORE_ACTION",
[junit4] 2> "AFTER_ACTION",
[junit4] 2> "IGNORED"],
[junit4] 2> "class":"org.apache.solr.cloud.autoscaling.SystemLogListener",
[junit4] 2> "beforeAction":[]},
[junit4] 2> ".auto_add_replicas.system":{
[junit4] 2> "trigger":".auto_add_replicas",
[junit4] 2> "afterAction":[],
[junit4] 2> "stage":[
[junit4] 2> "STARTED",
[junit4] 2> "ABORTED",
[junit4] 2> "SUCCEEDED",
[junit4] 2> "FAILED",
[junit4] 2> "BEFORE_ACTION",
[junit4] 2> "AFTER_ACTION",
[junit4] 2> "IGNORED"],
[junit4] 2> "class":"org.apache.solr.cloud.autoscaling.SystemLogListener",
[junit4] 2> "beforeAction":[]}},
[junit4] 2> "properties":{}}
[junit4] 2> 2198402 DEBUG (simCloudManagerPool-8493-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 2198402 DEBUG (simCloudManagerPool-8493-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 2198404 DEBUG (simCloudManagerPool-8493-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10009_solr]
[junit4] 2> 2198404 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 2198404 INFO (TEST-ScheduledMaintenanceTriggerTest.testTriggerDefaults-seed#[7FE64CD257D2AC02]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testTriggerDefaults
[junit4] 2> 2198406 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 2198406 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 2198414 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198437 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198459 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198470 INFO (TEST-ScheduledMaintenanceTriggerTest.testInactiveShardCleanup-seed#[7FE64CD257D2AC02]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testInactiveShardCleanup
[junit4] 2> 2198480 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198500 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198521 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198541 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198561 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198582 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198602 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198622 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198633 DEBUG (simCloudManagerPool-8493-thread-6) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for ScheduledMaintenanceTriggerTest_collection1 / shard1_1: {"core_node5":{
[junit4] 2> "node_name":"127.0.0.1:10009_solr",
[junit4] 2> "base_url":"http://127.0.0.1:10009/solr",
[junit4] 2> "leader":"true",
[junit4] 2> "state":"active",
[junit4] 2> "type":"NRT",
[junit4] 2> "INDEX.sizeInBytes":1,
[junit4] 2> "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_1_replica0",
[junit4] 2> "shard":"shard1_1",
[junit4] 2> "collection":"ScheduledMaintenanceTriggerTest_collection1"}}
[junit4] 2> 2198633 DEBUG (simCloudManagerPool-8493-thread-5) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for ScheduledMaintenanceTriggerTest_collection1 / shard1: {"core_node1":{
[junit4] 2> "node_name":"127.0.0.1:10009_solr",
[junit4] 2> "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_replica_n1",
[junit4] 2> "leader":"true",
[junit4] 2> "state":"active",
[junit4] 2> "type":"NRT",
[junit4] 2> "INDEX.sizeInBytes":1,
[junit4] 2> "shard":"shard1",
[junit4] 2> "collection":"ScheduledMaintenanceTriggerTest_collection1"}}
[junit4] 2> 2198633 DEBUG (simCloudManagerPool-8493-thread-7) [ ] o.a.s.c.a.s.SimClusterStateProvider -- elected new leader for ScheduledMaintenanceTriggerTest_collection1 / shard1_0: {"core_node4":{
[junit4] 2> "node_name":"127.0.0.1:10009_solr",
[junit4] 2> "base_url":"http://127.0.0.1:10009/solr",
[junit4] 2> "leader":"true",
[junit4] 2> "state":"active",
[junit4] 2> "type":"NRT",
[junit4] 2> "INDEX.sizeInBytes":1,
[junit4] 2> "core":"ScheduledMaintenanceTriggerTest_collection1_shard1_0_replica0",
[junit4] 2> "shard":"shard1_0",
[junit4] 2> "collection":"ScheduledMaintenanceTriggerTest_collection1"}}
[junit4] 2> 2198639 DEBUG (simCloudManagerPool-8493-thread-8) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 2198640 DEBUG (simCloudManagerPool-8493-thread-8) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 2198640 DEBUG (simCloudManagerPool-8493-thread-8) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10009_solr]
[junit4] 2> 2198640 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 2198640 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 2198640 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 2198641 DEBUG (simCloudManagerPool-8493-thread-9) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 2198641 DEBUG (simCloudManagerPool-8493-thread-9) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 2198642 DEBUG (simCloudManagerPool-8493-thread-9) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10009_solr]
[junit4] 2> 2198642 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 2198642 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 2198642 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 2198642 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:20.093Z. Now: 2018-03-11T23:37:32.949Z
[junit4] 2> 2198642 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198643 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621285c7bdTeimwawk0fbrwg137co0gl9neu",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729180093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811452949},
[junit4] 2> "ignored":true}
[junit4] 2> 2198663 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198666 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:22.093Z. Now: 2018-03-11T23:37:34.157Z
[junit4] 2> 2198667 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621285cf8dTeimwawk0fbrwg137co0gl9new",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729182093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811454157},
[junit4] 2> "ignored":true}
[junit4] 2> 2198683 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198688 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:24.093Z. Now: 2018-03-11T23:37:35.237Z
[junit4] 2> 2198688 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621285d75dTeimwawk0fbrwg137co0gl9ney",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729184093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811455237},
[junit4] 2> "ignored":true}
[junit4] 2> 2198703 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198709 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:26.093Z. Now: 2018-03-11T23:37:36.285Z
[junit4] 2> 2198709 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621285df2dTeimwawk0fbrwg137co0gl9nf0",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729186093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811456285},
[junit4] 2> "ignored":true}
[junit4] 2> 2198723 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198730 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:28.093Z. Now: 2018-03-11T23:37:37.333Z
[junit4] 2> 2198730 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621285e6fdTeimwawk0fbrwg137co0gl9nf2",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729188093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811457333},
[junit4] 2> "ignored":true}
[junit4] 2> 2198744 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198751 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:30.093Z. Now: 2018-03-11T23:37:38.386Z
[junit4] 2> 2198751 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621285eecdTeimwawk0fbrwg137co0gl9nf4",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729190093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811458386},
[junit4] 2> "ignored":true}
[junit4] 2> 2198764 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198772 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:32.093Z. Now: 2018-03-11T23:37:39.440Z
[junit4] 2> 2198772 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621285f69dTeimwawk0fbrwg137co0gl9nf6",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729192093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811459440},
[junit4] 2> "ignored":true}
[junit4] 2> 2198784 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198793 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:34.093Z. Now: 2018-03-11T23:37:40.493Z
[junit4] 2> 2198793 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621285fe6dTeimwawk0fbrwg137co0gl9nf8",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729194093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811460493},
[junit4] 2> "ignored":true}
[junit4] 2> 2198804 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198814 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:36.093Z. Now: 2018-03-11T23:37:41.549Z
[junit4] 2> 2198814 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286063dTeimwawk0fbrwg137co0gl9nfa",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729196093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811461549},
[junit4] 2> "ignored":true}
[junit4] 2> 2198825 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198835 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:38.093Z. Now: 2018-03-11T23:37:42.602Z
[junit4] 2> 2198835 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212860e0dTeimwawk0fbrwg137co0gl9nfc",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729198093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811462602},
[junit4] 2> "ignored":true}
[junit4] 2> 2198845 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198856 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:40.093Z. Now: 2018-03-11T23:37:43.655Z
[junit4] 2> 2198857 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"162128615ddTeimwawk0fbrwg137co0gl9nfe",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729200093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811463655},
[junit4] 2> "ignored":true}
[junit4] 2> 2198865 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198877 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:42.093Z. Now: 2018-03-11T23:37:44.710Z
[junit4] 2> 2198878 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212861dadTeimwawk0fbrwg137co0gl9nfg",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729202093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811464710},
[junit4] 2> "ignored":true}
[junit4] 2> 2198885 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198898 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:44.093Z. Now: 2018-03-11T23:37:45.768Z
[junit4] 2> 2198899 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286257dTeimwawk0fbrwg137co0gl9nfi",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729204093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811465768},
[junit4] 2> "ignored":true}
[junit4] 2> 2198905 DEBUG (ScheduledTrigger-8487-thread-3) [n:127.0.0.1:42071_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198906 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198919 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:46.093Z. Now: 2018-03-11T23:37:46.817Z
[junit4] 2> 2198920 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212862d4dTeimwawk0fbrwg137co0gl9nfk",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729206093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811466817},
[junit4] 2> "ignored":true}
[junit4] 2> 2198926 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198940 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:48.093Z. Now: 2018-03-11T23:37:47.862Z
[junit4] 2> 2198941 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286351dTeimwawk0fbrwg137co0gl9nfm",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729208093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811467862},
[junit4] 2> "ignored":true}
[junit4] 2> 2198946 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198961 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:50.093Z. Now: 2018-03-11T23:37:48.911Z
[junit4] 2> 2198962 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212863cedTeimwawk0fbrwg137co0gl9nfo",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729210093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811468911},
[junit4] 2> "ignored":true}
[junit4] 2> 2198966 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2198982 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:52.093Z. Now: 2018-03-11T23:37:49.955Z
[junit4] 2> 2198982 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"162128644bdTeimwawk0fbrwg137co0gl9nfq",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729212093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811469955},
[junit4] 2> "ignored":true}
[junit4] 2> 2198987 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199003 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:54.093Z. Now: 2018-03-11T23:37:50.998Z
[junit4] 2> 2199003 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212864c8dTeimwawk0fbrwg137co0gl9nfs",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729214093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811470998},
[junit4] 2> "ignored":true}
[junit4] 2> 2199007 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199024 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:56.093Z. Now: 2018-03-11T23:37:52.056Z
[junit4] 2> 2199025 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286545dTeimwawk0fbrwg137co0gl9nfu",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729216093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811472056},
[junit4] 2> "ignored":true}
[junit4] 2> 2199027 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199045 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:46:58.093Z. Now: 2018-03-11T23:37:53.110Z
[junit4] 2> 2199046 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212865c2dTeimwawk0fbrwg137co0gl9nfw",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729218093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811473110},
[junit4] 2> "ignored":true}
[junit4] 2> 2199047 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199066 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:00.093Z. Now: 2018-03-11T23:37:54.158Z
[junit4] 2> 2199067 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"162128663fdTeimwawk0fbrwg137co0gl9nfy",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729220093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811474158},
[junit4] 2> "ignored":true}
[junit4] 2> 2199067 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199087 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:02.093Z. Now: 2018-03-11T23:37:55.212Z
[junit4] 2> 2199088 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199088 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212866bcdTeimwawk0fbrwg137co0gl9ng0",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729222093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811475212},
[junit4] 2> "ignored":true}
[junit4] 2> 2199108 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199108 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:04.093Z. Now: 2018-03-11T23:37:56.252Z
[junit4] 2> 2199108 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286739dTeimwawk0fbrwg137co0gl9ng2",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729224093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811476252},
[junit4] 2> "ignored":true}
[junit4] 2> 2199128 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199129 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:06.093Z. Now: 2018-03-11T23:37:57.300Z
[junit4] 2> 2199129 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212867b6dTeimwawk0fbrwg137co0gl9ng4",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729226093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811477300},
[junit4] 2> "ignored":true}
[junit4] 2> 2199148 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199150 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:08.093Z. Now: 2018-03-11T23:37:58.351Z
[junit4] 2> 2199150 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286833dTeimwawk0fbrwg137co0gl9ng6",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729228093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811478351},
[junit4] 2> "ignored":true}
[junit4] 2> 2199169 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199171 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:10.093Z. Now: 2018-03-11T23:37:59.401Z
[junit4] 2> 2199171 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212868b0dTeimwawk0fbrwg137co0gl9ng8",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729230093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811479401},
[junit4] 2> "ignored":true}
[junit4] 2> 2199189 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199192 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:12.093Z. Now: 2018-03-11T23:38:00.448Z
[junit4] 2> 2199192 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"162128692ddTeimwawk0fbrwg137co0gl9nga",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729232093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811480448},
[junit4] 2> "ignored":true}
[junit4] 2> 2199209 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199213 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:14.093Z. Now: 2018-03-11T23:38:01.505Z
[junit4] 2> 2199213 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212869aadTeimwawk0fbrwg137co0gl9ngc",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729234093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811481505},
[junit4] 2> "ignored":true}
[junit4] 2> 2199229 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199234 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:16.093Z. Now: 2018-03-11T23:38:02.546Z
[junit4] 2> 2199234 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286a27dTeimwawk0fbrwg137co0gl9nge",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729236093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811482546},
[junit4] 2> "ignored":true}
[junit4] 2> 2199250 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199255 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:18.093Z. Now: 2018-03-11T23:38:03.596Z
[junit4] 2> 2199255 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286aa4dTeimwawk0fbrwg137co0gl9ngg",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729238093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811483596},
[junit4] 2> "ignored":true}
[junit4] 2> 2199270 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199276 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:20.093Z. Now: 2018-03-11T23:38:04.641Z
[junit4] 2> 2199276 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286b21dTeimwawk0fbrwg137co0gl9ngi",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729240093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811484641},
[junit4] 2> "ignored":true}
[junit4] 2> 2199290 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199297 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:22.093Z. Now: 2018-03-11T23:38:05.681Z
[junit4] 2> 2199297 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286b9edTeimwawk0fbrwg137co0gl9ngk",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729242093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811485681},
[junit4] 2> "ignored":true}
[junit4] 2> 2199310 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199318 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:24.093Z. Now: 2018-03-11T23:38:06.724Z
[junit4] 2> 2199318 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286c1bdTeimwawk0fbrwg137co0gl9ngm",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729244093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811486724},
[junit4] 2> "ignored":true}
[junit4] 2> 2199331 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199339 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:26.093Z. Now: 2018-03-11T23:38:07.778Z
[junit4] 2> 2199339 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286c98dTeimwawk0fbrwg137co0gl9ngo",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729246093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811487778},
[junit4] 2> "ignored":true}
[junit4] 2> 2199351 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199360 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:28.093Z. Now: 2018-03-11T23:38:08.828Z
[junit4] 2> 2199360 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286d15dTeimwawk0fbrwg137co0gl9ngq",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729248093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811488828},
[junit4] 2> "ignored":true}
[junit4] 2> 2199371 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199381 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:30.093Z. Now: 2018-03-11T23:38:09.875Z
[junit4] 2> 2199381 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286d92dTeimwawk0fbrwg137co0gl9ngs",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729250093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811489875},
[junit4] 2> "ignored":true}
[junit4] 2> 2199391 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199401 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:32.093Z. Now: 2018-03-11T23:38:10.920Z
[junit4] 2> 2199402 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286e0fdTeimwawk0fbrwg137co0gl9ngu",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729252093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811490920},
[junit4] 2> "ignored":true}
[junit4] 2> 2199412 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199422 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:34.093Z. Now: 2018-03-11T23:38:11.967Z
[junit4] 2> 2199423 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286e8cdTeimwawk0fbrwg137co0gl9ngw",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729254093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811491967},
[junit4] 2> "ignored":true}
[junit4] 2> 2199432 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199444 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:36.093Z. Now: 2018-03-11T23:38:13.023Z
[junit4] 2> 2199444 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286f09dTeimwawk0fbrwg137co0gl9ngy",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729256093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811493023},
[junit4] 2> "ignored":true}
[junit4] 2> 2199452 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199465 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:38.093Z. Now: 2018-03-11T23:38:14.085Z
[junit4] 2> 2199465 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621286f86dTeimwawk0fbrwg137co0gl9nh0",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729258093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811494085},
[junit4] 2> "ignored":true}
[junit4] 2> 2199473 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199486 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:40.093Z. Now: 2018-03-11T23:38:15.139Z
[junit4] 2> 2199486 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621287003dTeimwawk0fbrwg137co0gl9nh2",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729260093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811495139},
[junit4] 2> "ignored":true}
[junit4] 2> 2199493 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199507 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:42.093Z. Now: 2018-03-11T23:38:16.193Z
[junit4] 2> 2199507 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621287080dTeimwawk0fbrwg137co0gl9nh4",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729262093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811496193},
[junit4] 2> "ignored":true}
[junit4] 2> 2199513 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199528 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:44.093Z. Now: 2018-03-11T23:38:17.246Z
[junit4] 2> 2199528 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212870fddTeimwawk0fbrwg137co0gl9nh6",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729264093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811497246},
[junit4] 2> "ignored":true}
[junit4] 2> 2199533 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199549 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:46.093Z. Now: 2018-03-11T23:38:18.300Z
[junit4] 2> 2199549 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"162128717adTeimwawk0fbrwg137co0gl9nh8",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729266093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811498300},
[junit4] 2> "ignored":true}
[junit4] 2> 2199554 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199570 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:48.093Z. Now: 2018-03-11T23:38:19.353Z
[junit4] 2> 2199571 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212871f7dTeimwawk0fbrwg137co0gl9nha",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729268093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811499353},
[junit4] 2> "ignored":true}
[junit4] 2> 2199574 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199591 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:50.093Z. Now: 2018-03-11T23:38:20.406Z
[junit4] 2> 2199592 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621287274dTeimwawk0fbrwg137co0gl9nhc",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729270093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811500406},
[junit4] 2> "ignored":true}
[junit4] 2> 2199594 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199612 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:52.093Z. Now: 2018-03-11T23:38:21.461Z
[junit4] 2> 2199613 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212872f1dTeimwawk0fbrwg137co0gl9nhe",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729272093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811501461},
[junit4] 2> "ignored":true}
[junit4] 2> 2199614 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199633 WARN (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:54.093Z. Now: 2018-03-11T23:38:22.512Z
[junit4] 2> 2199634 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"162128736edTeimwawk0fbrwg137co0gl9nhg",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729274093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811502512},
[junit4] 2> "ignored":true}
[junit4] 2> 2199634 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199654 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:56.093Z. Now: 2018-03-11T23:38:23.564Z
[junit4] 2> 2199655 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199655 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212873ebdTeimwawk0fbrwg137co0gl9nhi",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729276093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811503564},
[junit4] 2> "ignored":true}
[junit4] 2> 2199675 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199675 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:47:58.093Z. Now: 2018-03-11T23:38:24.614Z
[junit4] 2> 2199676 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621287468dTeimwawk0fbrwg137co0gl9nhk",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729278093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811504614},
[junit4] 2> "ignored":true}
[junit4] 2> 2199695 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199696 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:00.093Z. Now: 2018-03-11T23:38:25.659Z
[junit4] 2> 2199697 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212874e5dTeimwawk0fbrwg137co0gl9nhm",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729280093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811505659},
[junit4] 2> "ignored":true}
[junit4] 2> 2199715 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199717 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:02.093Z. Now: 2018-03-11T23:38:26.714Z
[junit4] 2> 2199718 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621287562dTeimwawk0fbrwg137co0gl9nho",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729282093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811506714},
[junit4] 2> "ignored":true}
[junit4] 2> 2199736 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199738 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:04.093Z. Now: 2018-03-11T23:38:27.751Z
[junit4] 2> 2199738 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212875dfdTeimwawk0fbrwg137co0gl9nhq",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729284093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811507751},
[junit4] 2> "ignored":true}
[junit4] 2> 2199756 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199759 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:06.093Z. Now: 2018-03-11T23:38:28.803Z
[junit4] 2> 2199759 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"162128765cdTeimwawk0fbrwg137co0gl9nhs",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729286093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811508803},
[junit4] 2> "ignored":true}
[junit4] 2> 2199776 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199780 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:08.093Z. Now: 2018-03-11T23:38:29.846Z
[junit4] 2> 2199780 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212876d9dTeimwawk0fbrwg137co0gl9nhu",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729288093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811509846},
[junit4] 2> "ignored":true}
[junit4] 2> 2199797 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199801 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:10.093Z. Now: 2018-03-11T23:38:30.899Z
[junit4] 2> 2199801 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621287756dTeimwawk0fbrwg137co0gl9nhw",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729290093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811510899},
[junit4] 2> "ignored":true}
[junit4] 2> 2199817 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199822 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:12.093Z. Now: 2018-03-11T23:38:31.942Z
[junit4] 2> 2199822 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212877d3dTeimwawk0fbrwg137co0gl9nhy",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729292093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811511942},
[junit4] 2> "ignored":true}
[junit4] 2> 2199837 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199843 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:14.093Z. Now: 2018-03-11T23:38:32.994Z
[junit4] 2> 2199843 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621287850dTeimwawk0fbrwg137co0gl9ni0",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729294093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811512994},
[junit4] 2> "ignored":true}
[junit4] 2> 2199858 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199864 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:16.093Z. Now: 2018-03-11T23:38:34.031Z
[junit4] 2> 2199864 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212878cddTeimwawk0fbrwg137co0gl9ni2",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729296093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811514031},
[junit4] 2> "ignored":true}
[junit4] 2> 2199878 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199885 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:18.093Z. Now: 2018-03-11T23:38:35.076Z
[junit4] 2> 2199885 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"162128794adTeimwawk0fbrwg137co0gl9ni4",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729298093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811515076},
[junit4] 2> "ignored":true}
[junit4] 2> 2199898 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199905 DEBUG (ScheduledTrigger-8487-thread-4) [n:127.0.0.1:42071_solr ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199905 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:20.093Z. Now: 2018-03-11T23:38:36.112Z
[junit4] 2> 2199906 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"16212879c7dTeimwawk0fbrwg137co0gl9ni6",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729300093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811516112},
[junit4] 2> "ignored":true}
[junit4] 2> 2199918 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199926 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:22.093Z. Now: 2018-03-11T23:38:37.160Z
[junit4] 2> 2199927 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621287a44dTeimwawk0fbrwg137co0gl9ni8",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729302093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811517160},
[junit4] 2> "ignored":true}
[junit4] 2> 2199939 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199947 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:24.093Z. Now: 2018-03-11T23:38:38.200Z
[junit4] 2> 2199947 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621287ac1dTeimwawk0fbrwg137co0gl9nia",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729304093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811518200},
[junit4] 2> "ignored":true}
[junit4] 2> 2199959 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199968 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:26.093Z. Now: 2018-03-11T23:38:39.239Z
[junit4] 2> 2199968 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621287b3edTeimwawk0fbrwg137co0gl9nic",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729306093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811519239},
[junit4] 2> "ignored":true}
[junit4] 2> 2199979 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2199989 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:28.093Z. Now: 2018-03-11T23:38:40.287Z
[junit4] 2> 2199989 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621287bbbdTeimwawk0fbrwg137co0gl9nie",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729308093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811520287},
[junit4] 2> "ignored":true}
[junit4] 2> 2200000 DEBUG (ScheduledTrigger-8494-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2200010 WARN (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:30.093Z. Now: 2018-03-11T23:38:41.334Z
[junit4] 2> 2200010 DEBUG (ScheduledTrigger-8494-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event: {
[junit4] 2> "id":"1621287c38dTeimwawk0fbrwg137co0gl9nig",
[junit4] 2> "source":".scheduled_maintenance",
[junit4] 2> "eventTime":1520729310093,
[junit4] 2> "eventType":"SCHEDULED",
[junit4] 2> "properties":{"actualEventTime":1520811521334},
[junit4] 2> "ignored":true}
[junit4] 2> 2200020 DEBUG (ScheduledTrigger-8494-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 2200031 WARN (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTrigger ScheduledTrigger was not able to run event at scheduled time: 2018-03-11T00:48:32.093Z. Now: 2018-03-11T23:38:42.383Z
[junit4] 2> 2200031 DEBUG (ScheduledTrigger-8494-thread-2) [ ] o.a.s.c.a.ScheduledTriggers -------- Ignoring event
[...truncated too long message...]
utorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[junit4] 2> at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 5696 DEBUG (AutoscalingActionExecutor-29-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 5696 DEBUG (simCloudManagerPool-27-thread-12) [ ] o.a.s.c.a.NodeLostTrigger Initial livenodes: [127.0.0.1:10001_solr]
[junit4] 2> 5696 DEBUG (AutoscalingActionExecutor-29-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 86 ms for event id=315bad2fcf333Teimwawk0fbrwg137co0gl9fez
[junit4] 2> 5696 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 5696 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 5696 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
[junit4] 2> 5696 INFO (TEST-ScheduledMaintenanceTriggerTest.testInactiveShardCleanup-seed#[7FE64CD257D2AC02]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testInactiveShardCleanup
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=ScheduledMaintenanceTriggerTest -Dtests.method=testInactiveShardCleanup -Dtests.seed=7FE64CD257D2AC02 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=mt-MT -Dtests.timezone=America/Nipigon -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] FAILURE 0.18s J2 | ScheduledMaintenanceTriggerTest.testInactiveShardCleanup <<<
[junit4] > Throwable #1: java.lang.AssertionError: should be at least one inactive event
[junit4] > at __randomizedtesting.SeedInfo.seed([7FE64CD257D2AC02:62CA8CA036918B09]:0)
[junit4] > at org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup(ScheduledMaintenanceTriggerTest.java:218)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 5700 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 5701 DEBUG (SUITE-ScheduledMaintenanceTriggerTest-seed#[7FE64CD257D2AC02]-worker) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 5704 INFO (jetty-closer-25-thread-1) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@1b70154{SSL,[ssl, http/1.1]}{127.0.0.1:0}
[junit4] 2> 5705 INFO (jetty-closer-25-thread-1) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=7380891
[junit4] 2> 5706 INFO (jetty-closer-25-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 5706 INFO (jetty-closer-25-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@93c8cb: rootName = solr_37961, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@1075cc4
[junit4] 2> 5722 INFO (jetty-closer-25-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 5723 INFO (jetty-closer-25-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@10e3347: rootName = solr_37961, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@40c68b
[junit4] 2> 5739 INFO (jetty-closer-25-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 5739 INFO (jetty-closer-25-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@b4fb10: rootName = solr_37961, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@e2c2ea
[junit4] 2> 5741 INFO (jetty-closer-25-thread-1) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:37961_solr
[junit4] 2> 5742 INFO (jetty-closer-25-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 5742 INFO (jetty-closer-25-thread-1) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 5742 INFO (zkCallback-34-thread-1-processing-n:127.0.0.1:37961_solr) [n:127.0.0.1:37961_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 5742 INFO (jetty-closer-25-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72114490196295683-127.0.0.1:37961_solr-n_0000000000) closing
[junit4] 2> 5742 WARN (OverseerAutoScalingTriggerThread-72114490196295683-127.0.0.1:37961_solr-n_0000000000) [n:127.0.0.1:37961_solr ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 5742 INFO (zkCallback-39-thread-1-processing-n:127.0.0.1:37961_solr) [n:127.0.0.1:37961_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 5742 INFO (OverseerStateUpdate-72114490196295683-127.0.0.1:37961_solr-n_0000000000) [n:127.0.0.1:37961_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:37961_solr
[junit4] 2> 5743 DEBUG (jetty-closer-25-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 5744 INFO (jetty-closer-25-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@259448{/solr,null,UNAVAILABLE}
[junit4] 2> 5745 INFO (jetty-closer-25-thread-1) [ ] o.e.j.s.session Stopped scavenging
[junit4] 2> 5745 ERROR (SUITE-ScheduledMaintenanceTriggerTest-seed#[7FE64CD257D2AC02]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 5745 INFO (SUITE-ScheduledMaintenanceTriggerTest-seed#[7FE64CD257D2AC02]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:33925 33925
[junit4] 2> 6949 INFO (Thread-12) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:33925 33925
[junit4] 2> 6949 WARN (Thread-12) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/aliases.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/clusterstate.json
[junit4] 2> 3 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 3 /solr/live_nodes
[junit4] 2> 3 /solr/collections
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest_7FE64CD257D2AC02-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=57, maxMBSortInHeap=6.855484510279051, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@16b11a7), locale=mt-MT, timezone=America/Nipigon
[junit4] 2> NOTE: Linux 4.13.0-36-generic i386/Oracle Corporation 1.8.0_162 (32-bit)/cpus=8,threads=1,free=13335616,total=29474816
[junit4] 2> NOTE: All tests run in this JVM: [ScheduledMaintenanceTriggerTest, ScheduledMaintenanceTriggerTest]
[junit4] Completed [5/5 (2!)] on J2 in 2.24s, 2 tests, 1 failure <<< FAILURES!
[...truncated 12 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:1561: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:1085: There were test failures: 5 suites, 10 tests, 2 failures [seed: 7FE64CD257D2AC02]
Total time: 9 seconds
[repro] Setting last failure code to 256
[repro] Failures:
[repro] 2/5 failed: org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=f1f929b15a4bc0876e8c241bbc184fa870662939, workspace=/var/lib/jenkins/workspace/Lucene-Solr-master-Linux
[WARNINGS] Computing warning deltas based on reference build #21612
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2