You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2018/06/08 12:56:40 UTC
[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 74 - Still
Unstable
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/74/
3 tests failed.
FAILED: org.apache.solr.cloud.MoveReplicaHDFSTest.testNormalMove
Error Message:
Collection not found: MoveReplicaHDFSTest_coll_false
Stack Trace:
org.apache.solr.common.SolrException: Collection not found: MoveReplicaHDFSTest_coll_false
at __randomizedtesting.SeedInfo.seed([EA2FB78042EDA802:4CF70027C70C6A18]:0)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:853)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
at org.apache.solr.cloud.MoveReplicaTest.addDocs(MoveReplicaTest.java:374)
at org.apache.solr.cloud.MoveReplicaTest.test(MoveReplicaTest.java:111)
at org.apache.solr.cloud.MoveReplicaHDFSTest.testNormalMove(MoveReplicaHDFSTest.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration
Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:10016_solr should have been deleted
Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:10016_solr should have been deleted
at __randomizedtesting.SeedInfo.seed([EA2FB78042EDA802:F2953F8C4CD865ED]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertFalse(Assert.java:68)
at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration(TestTriggerIntegration.java:863)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate
Error Message:
The trigger did not fire at all
Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
at __randomizedtesting.SeedInfo.seed([EA2FB78042EDA802:B767A9098D2B0E4D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testSearchRate(TestTriggerIntegration.java:1201)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12576 lines...]
[junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestTriggerIntegration_EA2FB78042EDA802-001/init-core-data-001
[junit4] 2> 454375 INFO (SUITE-TestTriggerIntegration-seed#[EA2FB78042EDA802]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 454376 INFO (SUITE-TestTriggerIntegration-seed#[EA2FB78042EDA802]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 454377 INFO (SUITE-TestTriggerIntegration-seed#[EA2FB78042EDA802]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> 454411 DEBUG (SUITE-TestTriggerIntegration-seed#[EA2FB78042EDA802]-worker) [ ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10009_solr
[junit4] 2> 454411 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
[junit4] 2> 454411 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 0
[junit4] 2> 454414 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
[junit4] 2> 454415 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
[junit4] 2> 454475 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10009_solr]
[junit4] 2> 454475 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 454475 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion 0
[junit4] 2> 454476 DEBUG (ScheduledTrigger-1624-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 454487 WARN (SUITE-TestTriggerIntegration-seed#[EA2FB78042EDA802]-worker) [ ] o.a.s.h.a.MetricsHistoryHandler Missing .system, keeping metrics history in memory
[junit4] 2> 454490 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeLostTrigger
[junit4] 2> 454490 DEBUG (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 454491 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 454491 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 454495 DEBUG (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
[junit4] 2> 454495 DEBUG (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[junit4] 2> 454495 DEBUG (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[junit4] 2> 454495 DEBUG (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
[junit4] 2> 454495 DEBUG (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[junit4] 2> 454495 DEBUG (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 454508 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
[junit4] 2> 454513 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 2
[junit4] 2> 454514 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion -1
[junit4] 2> 454514 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 2
[junit4] 2> 454527 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10009_solr]
[junit4] 2> 454527 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 454527 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 2, lastZnodeVersion 2
[junit4] 2> 454527 DEBUG (ScheduledTrigger-1629-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 454548 DEBUG (ScheduledTrigger-1629-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 454568 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 454589 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 454652 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 454672 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 454692 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 454712 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 454733 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 454746 DEBUG (simCloudManagerPool-1628-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 454747 DEBUG (simCloudManagerPool-1628-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 3
[junit4] 2> 454747 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 3
[junit4] 2> 454748 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 454748 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 3, lastZnodeVersion 3
[junit4] 2> 454753 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 454772 DEBUG (simCloudManagerPool-1628-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 454772 DEBUG (simCloudManagerPool-1628-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 4
[junit4] 2> 454773 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 4
[junit4] 2> 454773 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 454773 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10009_solr]
[junit4] 2> 454774 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 454774 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
[junit4] 2> 454774 DEBUG (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10010_solr
[junit4] 2> 454775 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 454775 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 4, lastZnodeVersion 4
[junit4] 2> 454794 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 454794 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10009_solr
[junit4] 2> 454794 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 454794 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10009_solr
[junit4] 2> 454814 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 454814 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 454834 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1
[junit4] 2> 454834 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 1
[junit4] 2> 454834 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger firing registered processor for lost nodes: [127.0.0.1:10009_solr]
[junit4] 2> 454837 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"65fe5b78cbcd8cTalsd34ee26hypk3sj8jfryl2w",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":28708641469877644,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[28708641469877644],
[junit4] 2> "nodeNames":["127.0.0.1:10009_solr"]}}
[junit4] 2> 454837 DEBUG (ScheduledTrigger-1629-thread-4) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_lost_trigger]
[junit4] 2> 454935 DEBUG (AutoscalingActionExecutor-1630-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"65fe5b78cbcd8cTalsd34ee26hypk3sj8jfryl2w",
[junit4] 2> "source":"node_lost_trigger",
[junit4] 2> "eventTime":28708641469877644,
[junit4] 2> "eventType":"NODELOST",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[28708641469877644],
[junit4] 2> "_enqueue_time_":28708643655608244,
[junit4] 2> "nodeNames":["127.0.0.1:10009_solr"]}}
[junit4] 2> 454958 DEBUG (AutoscalingActionExecutor-1630-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 454958 DEBUG (AutoscalingActionExecutor-1630-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_lost_trigger after 100ms
[junit4] 2> 454958 DEBUG (AutoscalingActionExecutor-1630-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 1158 ms for event id=65fe5b78cbcd8cTalsd34ee26hypk3sj8jfryl2w
[junit4] 2> 454959 DEBUG (simCloudManagerPool-1628-thread-6) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 454959 DEBUG (simCloudManagerPool-1628-thread-6) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 5
[junit4] 2> 454960 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 5
[junit4] 2> 454960 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 454960 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 5, lastZnodeVersion 5
[junit4] 2> 455001 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeLostTrigger
[junit4] 2> 455005 INFO (TEST-TestTriggerIntegration.testNodeLostTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
[junit4] 2> ############ CLUSTER STATE ############
[junit4] 2> #######################################
[junit4] 2> ## Live nodes: 1
[junit4] 2> ## Empty nodes: 0
[junit4] 2> ## Dead nodes: 1
[junit4] 2> ## 127.0.0.1:10009_solr
[junit4] 2> ## Collections: [.system]
[junit4] 2> ## Max replicas per node: 1
[junit4] 2> ## Min replicas per node: 1
[junit4] 2> ## Total replicas: 1
[junit4] 2> ## * .system 1
[junit4] 2> ## - active 1
[junit4] 2> ######### Solr op counts ##########
[junit4] 2> ## - autoscaling 3
[junit4] 2> ## - update 2
[junit4] 2> ######### Autoscaling event counts ###########
[junit4] 2> ## * Trigger: node_lost_trigger
[junit4] 2> ## - STARTED 1
[junit4] 2> ## - SUCCEEDED 1
[junit4] 2>
[junit4] 2> 455009 INFO (TEST-TestTriggerIntegration.testListeners-seed#[EA2FB78042EDA802]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testListeners
[junit4] 2> 455009 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 6
[junit4] 2> 455009 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 6
[junit4] 2> 455009 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 455009 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 455009 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 6, lastZnodeVersion 6
[junit4] 2> 455009 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10009_solr
[junit4] 2> 455010 INFO (TEST-TestTriggerIntegration.testListeners-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 455010 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException: null
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_172]
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_172]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
[junit4] 2> 455010 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
[junit4] 2> 455010 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[junit4] 2> 455011 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[junit4] 2> 455011 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
[junit4] 2> 455011 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[junit4] 2> 455011 DEBUG (TEST-TestTriggerIntegration.testListeners-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 455251 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
[junit4] 2> 455251 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 7
[junit4] 2> 455259 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion -1
[junit4] 2> 455259 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 7
[junit4] 2> 455266 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10011_solr]
[junit4] 2> 455279 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 455279 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 7, lastZnodeVersion 7
[junit4] 2> 455279 DEBUG (ScheduledTrigger-1632-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 455303 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 455323 DEBUG (ScheduledTrigger-1632-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 455344 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 455364 DEBUG (ScheduledTrigger-1632-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 455365 DEBUG (simCloudManagerPool-1631-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 455365 DEBUG (simCloudManagerPool-1631-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 8
[junit4] 2> 455373 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 8
[junit4] 2> 455374 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 455374 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 8, lastZnodeVersion 8
[junit4] 2> 455384 DEBUG (ScheduledTrigger-1632-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 455395 DEBUG (simCloudManagerPool-1631-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 455395 DEBUG (simCloudManagerPool-1631-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 9
[junit4] 2> 455396 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 9
[junit4] 2> 455396 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 455397 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10011_solr]
[junit4] 2> 455397 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=3, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}, {name=test1, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction}], enabled=true}
[junit4] 2> 455397 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 9, lastZnodeVersion 9
[junit4] 2> 455398 DEBUG (ScheduledTrigger-1632-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 455398 DEBUG (ScheduledTrigger-1632-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 455404 DEBUG (ScheduledTrigger-1632-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 455418 DEBUG (simCloudManagerPool-1631-thread-3) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 455418 DEBUG (simCloudManagerPool-1631-thread-3) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 10
[junit4] 2> 455418 DEBUG (ScheduledTrigger-1632-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 455419 DEBUG (ScheduledTrigger-1632-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 455419 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 10
[junit4] 2> 455419 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 10, lastZnodeVersion 10
[junit4] 2> 455422 DEBUG (simCloudManagerPool-1631-thread-4) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 455422 DEBUG (simCloudManagerPool-1631-thread-4) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 11
[junit4] 2> 455424 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 455430 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 11
[junit4] 2> 455430 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 11, lastZnodeVersion 11
[junit4] 2> 455439 DEBUG (ScheduledTrigger-1632-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 455439 DEBUG (ScheduledTrigger-1632-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 455439 DEBUG (ScheduledTrigger-1632-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10012_solr at time 28708673725769394
[junit4] 2> 455445 DEBUG (ScheduledTrigger-1632-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 455459 DEBUG (ScheduledTrigger-1632-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 455459 DEBUG (ScheduledTrigger-1632-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 455465 DEBUG (ScheduledTrigger-1632-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 455480 DEBUG (ScheduledTrigger-1632-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 455480 DEBUG (ScheduledTrigger-1632-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 455485 DEBUG (ScheduledTrigger-1632-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 455526 DEBUG (ScheduledTrigger-1632-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 455526 DEBUG (ScheduledTrigger-1632-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 455526 DEBUG (ScheduledTrigger-1632-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10012_solr] added at times [28708673725769394], now=28708678082960094
[junit4] 2> 455527 DEBUG (ScheduledTrigger-1632-thread-4) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"65fe62fb65a6b2Talsd34ee26hypk3sj8jfryl2z",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":28708673725769394,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[28708673725769394],
[junit4] 2> "nodeNames":["127.0.0.1:10012_solr"]}}
[junit4] 2> 455527 DEBUG (ScheduledTrigger-1632-thread-4) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger]
[junit4] 2> 455591 DEBUG (ScheduledTrigger-1632-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - removing marker path: /autoscaling/nodeAdded/127.0.0.1:10012_solr
[junit4] 2> 455592 DEBUG (AutoscalingActionExecutor-1633-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"65fe62fb65a6b2Talsd34ee26hypk3sj8jfryl2z",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":28708673725769394,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[28708673725769394],
[junit4] 2> "_enqueue_time_":28708678135965294,
[junit4] 2> "nodeNames":["127.0.0.1:10012_solr"]}}
[junit4] 2> 455603 DEBUG (AutoscalingActionExecutor-1633-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 455603 DEBUG (AutoscalingActionExecutor-1633-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
[junit4] 2> 455603 DEBUG (AutoscalingActionExecutor-1633-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 598 ms for event id=65fe62fb65a6b2Talsd34ee26hypk3sj8jfryl2z
[junit4] 2> 455704 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 455704 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 455704 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 455704 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10013_solr at time 28708686969176794
[junit4] 2> 455724 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 455724 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 455724 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 455744 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 455745 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 455745 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 455764 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 455765 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 455765 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 4
[junit4] 2> 455765 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10013_solr] added at times [28708686969176794], now=28708690011969294
[junit4] 2> 455765 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"65fe6610c402daTalsd34ee26hypk3sj8jfryl32",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":28708686969176794,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[28708686969176794],
[junit4] 2> "nodeNames":["127.0.0.1:10013_solr"]}}
[junit4] 2> 455765 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger]
[junit4] 2> 455892 DEBUG (ScheduledTrigger-1632-thread-1) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - removing marker path: /autoscaling/nodeAdded/127.0.0.1:10013_solr
[junit4] 2> 455892 DEBUG (AutoscalingActionExecutor-1633-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"65fe6610c402daTalsd34ee26hypk3sj8jfryl32",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":28708686969176794,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[28708686969176794],
[junit4] 2> "_enqueue_time_":28708690032917844,
[junit4] 2> "nodeNames":["127.0.0.1:10013_solr"]}}
[junit4] 2> 455898 WARN (AutoscalingActionExecutor-1633-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Exception executing actions
[junit4] 2> java.lang.Exception: Error executing action: test1 for trigger event: {
[junit4] 2> "id":"65fe6610c402daTalsd34ee26hypk3sj8jfryl32",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":28708686969176794,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[28708686969176794],
[junit4] 2> "_enqueue_time_":28708690032917844,
[junit4] 2> "nodeNames":["127.0.0.1:10013_solr"]}}
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:311) ~[java/:?]
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_172]
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
[junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
[junit4] 2> Caused by: java.lang.RuntimeException: failure
[junit4] 2> at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestDummyAction.process(TestTriggerIntegration.java:911) ~[test/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:308) ~[java/:?]
[junit4] 2> ... 6 more
[junit4] 2> 455899 DEBUG (AutoscalingActionExecutor-1633-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 455899 DEBUG (AutoscalingActionExecutor-1633-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
[junit4] 2> 455899 DEBUG (AutoscalingActionExecutor-1633-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 359 ms for event id=65fe6610c402daTalsd34ee26hypk3sj8jfryl32
[junit4] 2> 455933 INFO (TEST-TestTriggerIntegration.testListeners-seed#[EA2FB78042EDA802]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testListeners
[junit4] 2> 455933 INFO (TEST-TestTriggerIntegration.testListeners-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
[junit4] 2> ############ CLUSTER STATE ############
[junit4] 2> #######################################
[junit4] 2> ## Live nodes: 4
[junit4] 2> ## Empty nodes: 3
[junit4] 2> ## Dead nodes: 0
[junit4] 2> ## Collections: [.system]
[junit4] 2> ## Max replicas per node: 1
[junit4] 2> ## Min replicas per node: 0
[junit4] 2> ## Total replicas: 1
[junit4] 2> ## * .system 1
[junit4] 2> ## - active 1
[junit4] 2> ######### Solr op counts ##########
[junit4] 2> ## - autoscaling 4
[junit4] 2> ## - update 4
[junit4] 2> ######### Autoscaling event counts ###########
[junit4] 2> ## * Trigger: node_added_trigger
[junit4] 2> ## - FAILED 1
[junit4] 2> ## - STARTED 2
[junit4] 2> ## - SUCCEEDED 1
[junit4] 2>
[junit4] 2> 455946 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeAddedTrigger
[junit4] 2> 455946 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 12
[junit4] 2> 455946 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 12
[junit4] 2> 455946 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 455946 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 455946 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 12, lastZnodeVersion 12
[junit4] 2> 455947 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimNodeStateProvider --removing value for 127.0.0.1:10011_solr
[junit4] 2> 455947 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimNodeStateProvider --removing value for 127.0.0.1:10012_solr
[junit4] 2> 455947 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 455947 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException: null
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_172]
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_172]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
[junit4] 2> 455947 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
[junit4] 2> 455948 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[junit4] 2> 455948 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[junit4] 2> 455948 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
[junit4] 2> 455948 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[junit4] 2> 455948 DEBUG (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 455963 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
[junit4] 2> 455963 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 13
[junit4] 2> 455963 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion -1
[junit4] 2> 455963 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 13
[junit4] 2> 455975 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr]
[junit4] 2> 455978 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 455978 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 13, lastZnodeVersion 13
[junit4] 2> 455978 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 455999 DEBUG (ScheduledTrigger-1635-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456019 DEBUG (ScheduledTrigger-1635-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456039 DEBUG (ScheduledTrigger-1635-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456059 DEBUG (ScheduledTrigger-1635-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456079 DEBUG (simCloudManagerPool-1634-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 456079 DEBUG (simCloudManagerPool-1634-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 14
[junit4] 2> 456080 DEBUG (ScheduledTrigger-1635-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456081 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 14
[junit4] 2> 456081 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 456081 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 14, lastZnodeVersion 14
[junit4] 2> 456098 DEBUG (simCloudManagerPool-1634-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 456107 DEBUG (ScheduledTrigger-1635-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456108 DEBUG (simCloudManagerPool-1634-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 15
[junit4] 2> 456109 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 15
[junit4] 2> 456109 INFO (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.s.TestTriggerIntegration TestTriggerAction init
[junit4] 2> 456109 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr]
[junit4] 2> 456109 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=2, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestTriggerAction}], enabled=true}
[junit4] 2> 456109 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 15, lastZnodeVersion 15
[junit4] 2> 456109 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 456109 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 456127 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 456130 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 456130 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 456130 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10014_solr at time 28708708256458744
[junit4] 2> 456147 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 456150 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 456150 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 456207 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 456207 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 456207 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 456207 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10014_solr] added at times [28708708256458744], now=28708712137760494
[junit4] 2> 456208 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"65fe6b059627f8Talsd34ee26hypk3sj8jfryl35",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":28708708256458744,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[28708708256458744],
[junit4] 2> "nodeNames":["127.0.0.1:10014_solr"]}}
[junit4] 2> 456211 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_added_trigger]
[junit4] 2> 456267 DEBUG (ScheduledTrigger-1635-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - removing marker path: /autoscaling/nodeAdded/127.0.0.1:10014_solr
[junit4] 2> 456267 DEBUG (AutoscalingActionExecutor-1636-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"65fe6b059627f8Talsd34ee26hypk3sj8jfryl35",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":28708708256458744,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[28708708256458744],
[junit4] 2> "_enqueue_time_":28708712329140594,
[junit4] 2> "nodeNames":["127.0.0.1:10014_solr"]}}
[junit4] 2> 456311 DEBUG (AutoscalingActionExecutor-1636-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: .auto_add_replicas after 100ms
[junit4] 2> 456311 DEBUG (AutoscalingActionExecutor-1636-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Resuming trigger: node_added_trigger after 100ms
[junit4] 2> 456311 DEBUG (AutoscalingActionExecutor-1636-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 2216 ms for event id=65fe6b059627f8Talsd34ee26hypk3sj8jfryl35
[junit4] 2> 456312 DEBUG (simCloudManagerPool-1634-thread-6) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 456312 DEBUG (simCloudManagerPool-1634-thread-6) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 16
[junit4] 2> 456318 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 16
[junit4] 2> 456319 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 16, lastZnodeVersion 16
[junit4] 2> 456361 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeAddedTrigger
[junit4] 2> 456362 INFO (TEST-TestTriggerIntegration.testNodeAddedTrigger-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
[junit4] 2> ############ CLUSTER STATE ############
[junit4] 2> #######################################
[junit4] 2> ## Live nodes: 3
[junit4] 2> ## Empty nodes: 2
[junit4] 2> ## Dead nodes: 0
[junit4] 2> ## Collections: [.system]
[junit4] 2> ## Max replicas per node: 1
[junit4] 2> ## Min replicas per node: 0
[junit4] 2> ## Total replicas: 1
[junit4] 2> ## * .system 1
[junit4] 2> ## - active 1
[junit4] 2> ######### Solr op counts ##########
[junit4] 2> ## - autoscaling 3
[junit4] 2> ## - update 2
[junit4] 2> ######### Autoscaling event counts ###########
[junit4] 2> ## * Trigger: node_added_trigger
[junit4] 2> ## - STARTED 1
[junit4] 2> ## - SUCCEEDED 1
[junit4] 2>
[junit4] 2> 456366 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNodeMarkersRegistration
[junit4] 2> 456366 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 17
[junit4] 2> 456366 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimNodeStateProvider --removing value for 127.0.0.1:10014_solr
[junit4] 2> 456366 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 456366 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 456366 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
[junit4] 2> 456367 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[junit4] 2> 456367 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[junit4] 2> 456367 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
[junit4] 2> 456367 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[junit4] 2> 456367 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 456383 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
[junit4] 2> 456384 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 18
[junit4] 2> 456421 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 18, lastZnodeVersion -1
[junit4] 2> 456421 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 18
[junit4] 2> 456439 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10010_solr, 127.0.0.1:10013_solr]
[junit4] 2> 456448 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 456448 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 18, lastZnodeVersion 18
[junit4] 2> 456449 DEBUG (ScheduledTrigger-1638-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456494 DEBUG (ScheduledTrigger-1638-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456515 DEBUG (ScheduledTrigger-1638-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456521 DEBUG (simCloudManagerPool-1637-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 456522 DEBUG (simCloudManagerPool-1637-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 19
[junit4] 2> 456527 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 19
[junit4] 2> 456528 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.TestTriggerIntegration ====== KILL OVERSEER 1
[junit4] 2> 456528 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 456530 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 456530 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException: null
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireInterruptibly(AbstractQueuedSynchronizer.java:1220) ~[?:1.8.0_172]
[junit4] 2> at java.util.concurrent.locks.ReentrantLock.lockInterruptibly(ReentrantLock.java:335) ~[?:1.8.0_172]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:177) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
[junit4] 2> 456531 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
[junit4] 2> 456531 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[junit4] 2> 456531 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[junit4] 2> 456531 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
[junit4] 2> 456531 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[junit4] 2> 456531 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 456531 DEBUG (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimClusterStateProvider --- new Overseer leader: 127.0.0.1:10013_solr
[junit4] 2> 456547 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 19
[junit4] 2> 456548 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 19, lastZnodeVersion -1
[junit4] 2> 456548 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 19
[junit4] 2> 456552 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10013_solr, 127.0.0.1:10015_solr]
[junit4] 2> 456552 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger Adding lost node from marker path: 127.0.0.1:10010_solr
[junit4] 2> 456558 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 456558 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 19, lastZnodeVersion 19
[junit4] 2> 456558 DEBUG (ScheduledTrigger-1641-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456558 DEBUG (ScheduledTrigger-1641-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Tracking lost node: 127.0.0.1:10010_solr
[junit4] 2> 456578 DEBUG (ScheduledTrigger-1641-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456599 DEBUG (ScheduledTrigger-1641-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456619 DEBUG (ScheduledTrigger-1641-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456640 DEBUG (ScheduledTrigger-1641-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456647 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.TestTriggerIntegration ====== ADD TRIGGERS
[junit4] 2> 456660 DEBUG (ScheduledTrigger-1641-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456668 INFO (simCloudManagerPool-1640-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
[junit4] 2> 456668 DEBUG (simCloudManagerPool-1640-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 456668 DEBUG (simCloudManagerPool-1640-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 20
[junit4] 2> 456671 INFO (simCloudManagerPool-1640-thread-1) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
[junit4] 2> 456671 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 20
[junit4] 2> 456672 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - Initial livenodes: [127.0.0.1:10013_solr, 127.0.0.1:10015_solr]
[junit4] 2> 456672 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger instantiated with properties: {event=nodeAdded, waitFor=1, actions=[{name=test, class=org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration$TestEventMarkerAction}], enabled=true}
[junit4] 2> 456672 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 20, lastZnodeVersion 20
[junit4] 2> 456672 DEBUG (ScheduledTrigger-1641-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 456672 DEBUG (ScheduledTrigger-1641-thread-3) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 456680 DEBUG (ScheduledTrigger-1641-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456689 INFO (simCloudManagerPool-1640-thread-2) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
[junit4] 2> 456690 DEBUG (simCloudManagerPool-1640-thread-2) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 456690 DEBUG (simCloudManagerPool-1640-thread-2) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 21
[junit4] 2> 456691 INFO (simCloudManagerPool-1640-thread-2) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
[junit4] 2> 456691 INFO (simCloudManagerPool-1640-thread-2) [ ] o.a.s.c.a.s.TestTriggerIntegration TestEventMarkerAction init
[junit4] 2> 456691 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 21
[junit4] 2> 456691 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger node_lost_trigger - Initial livenodes: [127.0.0.1:10013_solr, 127.0.0.1:10015_solr]
[junit4] 2> 456691 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 21, lastZnodeVersion 21
[junit4] 2> 456691 DEBUG (ScheduledTrigger-1641-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 2
[junit4] 2> 456700 DEBUG (ScheduledTrigger-1641-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 456700 DEBUG (ScheduledTrigger-1641-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 456700 DEBUG (ScheduledTrigger-1641-thread-4) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 2
[junit4] 2> 456701 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.TestTriggerIntegration ====== ADD NODE 1
[junit4] 2> 456712 DEBUG (ScheduledTrigger-1641-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 3
[junit4] 2> 456720 DEBUG (ScheduledTrigger-1641-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 456721 DEBUG (ScheduledTrigger-1641-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 456721 DEBUG (ScheduledTrigger-1641-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 456721 DEBUG (ScheduledTrigger-1641-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Tracking new node: 127.0.0.1:10016_solr at time 28708737808235594
[junit4] 2> 456732 DEBUG (ScheduledTrigger-1641-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: node_lost_trigger with currently live nodes: 3
[junit4] 2> 456741 DEBUG (ScheduledTrigger-1641-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 3
[junit4] 2> 456741 DEBUG (ScheduledTrigger-1641-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Running NodeAddedTrigger node_added_trigger
[junit4] 2> 456741 DEBUG (ScheduledTrigger-1641-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger Found livenodes: 3
[junit4] 2> 456741 DEBUG (ScheduledTrigger-1641-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger firing registered processor for nodes: [127.0.0.1:10016_solr] added at times [28708737808235594], now=28708738826808194
[junit4] 2> 456741 DEBUG (ScheduledTrigger-1641-thread-2) [ ] o.a.s.c.a.ScheduledTriggers ++++++++ Cooldown inactive - processing event: {
[junit4] 2> "id":"65fe71e7027c4aTalsd34ee26hypk3sj8jfryl38",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":28708737808235594,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[28708737808235594],
[junit4] 2> "nodeNames":["127.0.0.1:10016_solr"]}}
[junit4] 2> 456742 DEBUG (ScheduledTrigger-1641-thread-2) [ ] o.a.s.c.a.ScheduledTriggers Pausing all triggers: [.auto_add_replicas, node_lost_trigger, node_added_trigger]
[junit4] 2> 456803 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.SolrTestCaseJ4 ###Ending testNodeMarkersRegistration
[junit4] 2> 456804 INFO (TEST-TestTriggerIntegration.testNodeMarkersRegistration-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimSolrCloudTestCase #######################################
[junit4] 2> ############ CLUSTER STATE ############
[junit4] 2> #######################################
[junit4] 2> ## Live nodes: 3
[junit4] 2> ## Empty nodes: 2
[junit4] 2> ## Dead nodes: 1
[junit4] 2> ## 127.0.0.1:10010_solr
[junit4] 2> ## Collections: [.system]
[junit4] 2> ## Max replicas per node: 1
[junit4] 2> ## Min replicas per node: 0
[junit4] 2> ## Total replicas: 1
[junit4] 2> ## * .system 1
[junit4] 2> ## - active 1
[junit4] 2> ######### Solr op counts ##########
[junit4] 2> ## - autoscaling 3
[junit4] 2> ## - update 1
[junit4] 2> ######### Autoscaling event counts ###########
[junit4] 2> ## * Trigger: node_added_trigger
[junit4] 2> ## - STARTED 1
[junit4] 2>
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestTriggerIntegration -Dtests.method=testNodeMarkersRegistration -Dtests.seed=EA2FB78042EDA802 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=hr-HR -Dtests.timezone=Pacific/Noumea -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] FAILURE 0.44s J1 | TestTriggerIntegration.testNodeMarkersRegistration <<<
[junit4] > Throwable #1: java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:10016_solr should have been deleted
[junit4] > at __randomizedtesting.SeedInfo.seed([EA2FB78042EDA802:F2953F8C4CD865ED]:0)
[junit4] > at org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration.testNodeMarkersRegistration(TestTriggerIntegration.java:863)
[junit4] > at java.lang.Thread.run(Thread.java:748)
[junit4] 2> 456819 DEBUG (ScheduledTrigger-1641-thread-2) [ ] o.a.s.c.a.NodeAddedTrigger NodeAddedTrigger node_added_trigger - removing marker path: /autoscaling/nodeAdded/127.0.0.1:10016_solr
[junit4] 2> 456827 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[EA2FB78042EDA802]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testSearchRate
[junit4] 2> 456827 DEBUG (TEST-TestTriggerIntegration.testSearchRate-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 22
[junit4] 2> 456828 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 22
[junit4] 2> 456828 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeLost markers
[junit4] 2> 456828 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 456828 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 22, lastZnodeVersion 22
[junit4] 2> 456828 DEBUG (TEST-TestTriggerIntegration.testSearchRate-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimNodeStateProvider - removing dead node values: 127.0.0.1:10010_solr
[junit4] 2> 456828 DEBUG (TEST-TestTriggerIntegration.testSearchRate-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimNodeStateProvider --removing value for 127.0.0.1:10015_solr
[junit4] 2> 456828 INFO (TEST-TestTriggerIntegration.testSearchRate-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing object cache...
[junit4] 2> 456828 WARN (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Interrupted
[junit4] 2> java.lang.InterruptedException: null
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_172]
[junit4] 2> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_172]
[junit4] 2> at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189) [java/:?]
[junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
[junit4] 2> 456831 DEBUG (TEST-TestTriggerIntegration.testSearchRate-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
[junit4] 2> 456831 DEBUG (TEST-TestTriggerIntegration.testSearchRate-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[junit4] 2> 456831 DEBUG (TEST-TestTriggerIntegration.testSearchRate-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[junit4] 2> 456832 DEBUG (AutoscalingActionExecutor-1642-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing actions for {
[junit4] 2> "id":"65fe71e7027c4aTalsd34ee26hypk3sj8jfryl38",
[junit4] 2> "source":"node_added_trigger",
[junit4] 2> "eventTime":28708737808235594,
[junit4] 2> "eventType":"NODEADDED",
[junit4] 2> "properties":{
[junit4] 2> "eventTimes":[28708737808235594],
[junit4] 2> "_enqueue_time_":28708738856584544,
[junit4] 2> "nodeNames":["127.0.0.1:10016_solr"]}}
[junit4] 2> 456832 WARN (AutoscalingActionExecutor-1642-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Exception executing actions
[junit4] 2> org.apache.lucene.store.AlreadyClosedException: ScheduledTrigger node_added_trigger has been closed.
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.dequeue(ScheduledTriggers.java:545) ~[java/:?]
[junit4] 2> at org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$3(ScheduledTriggers.java:318) ~[java/:?]
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_172]
[junit4] 2> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_172]
[junit4] 2> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[java/:?]
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_172]
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_172]
[junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
[junit4] 2> 456832 DEBUG (AutoscalingActionExecutor-1642-thread-1) [ ] o.a.s.c.a.ScheduledTriggers -- processing took 35 ms for event id=65fe71e7027c4aTalsd34ee26hypk3sj8jfryl38
[junit4] 2> 456839 DEBUG (TEST-TestTriggerIntegration.testSearchRate-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
[junit4] 2> 456839 DEBUG (TEST-TestTriggerIntegration.testSearchRate-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[junit4] 2> 456839 DEBUG (TEST-TestTriggerIntegration.testSearchRate-seed#[EA2FB78042EDA802]) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 456845 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
[junit4] 2> 456845 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 23
[junit4] 2> 456846 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 23, lastZnodeVersion -1
[junit4] 2> 456846 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 23
[junit4] 2> 456859 WARN (MetricsHistoryHandler-1627-thread-1) [ ] o.a.s.h.a.MetricsHistoryHandler Missing .system, keeping metrics history in memory
[junit4] 2> 456867 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:10016_solr, 127.0.0.1:10013_solr]
[junit4] 2> 456997 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 456997 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 23, lastZnodeVersion 23
[junit4] 2> 456997 DEBUG (ScheduledTrigger-1644-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 457003 DEBUG (simCloudManagerPool-1643-thread-1) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 457003 DEBUG (simCloudManagerPool-1643-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 24
[junit4] 2> 457020 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 24
[junit4] 2> 457027 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 457027 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 24, lastZnodeVersion 24
[junit4] 2> 457028 DEBUG (ScheduledTrigger-1644-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 457049 DEBUG (ScheduledTrigger-1644-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 457069 DEBUG (ScheduledTrigger-1644-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 457089 DEBUG (ScheduledTrigger-1644-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 457101 DEBUG (simCloudManagerPool-1643-thread-6) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 457101 DEBUG (simCloudManagerPool-1643-thread-6) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 25
[junit4] 2> 457110 DEBUG (ScheduledTrigger-1644-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 457110 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 25
[junit4] 2> 457110 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 457110 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 25, lastZnodeVersion 25
[junit4] 2> 457130 DEBUG (ScheduledTrigger-1644-thread-3) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 457150 DEBUG (ScheduledTrigger-1644-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 457159 DEBUG (simCloudManagerPool-1643-thread-7) [ ] o.a.s.c.a.AutoScalingHandler Verified autoscaling configuration
[junit4] 2> 457160 DEBUG (simCloudManagerPool-1643-thread-7) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 26
[junit4] 2> 457161 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 26
[junit4] 2> 457161 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 457161 DEBUG (Simulated OverseerAutoScalingTriggerThread) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 26, lastZnodeVersion 26
[...truncated too long message...]
file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
jar-checksums:
[mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/null2011564224
[copy] Copying 39 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/null2011564224
[delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/null2011564224
resolve-example:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
resolve-server:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
jar-checksums:
[mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/null725530840
[copy] Copying 247 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/null725530840
[delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/null725530840
check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 101ms :: artifacts dl 2ms
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 8 | 0 | 0 | 0 || 8 | 0 |
---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...
-jenkins-base:
BUILD SUCCESSFUL
Total time: 172 minutes 0 seconds
Archiving artifacts
WARN: No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
WARN: java.lang.InterruptedException: no matches found within 10000
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 76 - Still
Unstable
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/76/
6 tests failed.
FAILED: org.apache.solr.cloud.CollectionsAPISolrJTest.testAddAndDeleteReplica
Error Message:
Could not load collection from ZK: solrj_replicatests
Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: solrj_replicatests
at __randomizedtesting.SeedInfo.seed([F64145A395D086EC:18F13CF6921C9156]:0)
at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1316)
at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:732)
at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:148)
at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:131)
at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:117)
at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:256)
at org.apache.solr.cloud.CollectionsAPISolrJTest.testAddAndDeleteReplica(CollectionsAPISolrJTest.java:350)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/solrj_replicatests/state.json
at org.apache.zookeeper.KeeperException.create(KeeperException.java:130)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1215)
at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:341)
at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:341)
at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1328)
at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1314)
... 45 more
FAILED: org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup
Error Message:
should be at least one inactive event
Stack Trace:
java.lang.AssertionError: should be at least one inactive event
at __randomizedtesting.SeedInfo.seed([F64145A395D086EC:EB6D85D1F493A1E7]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup(ScheduledMaintenanceTriggerTest.java:224)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testAddNode
Error Message:
trigger did not fire
Stack Trace:
java.lang.AssertionError: trigger did not fire
at __randomizedtesting.SeedInfo.seed([F64145A395D086EC:51AE58005A9D09F4]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testAddNode(TestLargeCluster.java:301)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testSearchRate
Error Message:
The trigger did not fire at all
Stack Trace:
java.lang.AssertionError: The trigger did not fire at all
at __randomizedtesting.SeedInfo.seed([F64145A395D086EC:AB095B2A5A1620A3]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testSearchRate(TestLargeCluster.java:636)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.autoscaling.sim.TestLargeCluster.testNodeLost
Error Message:
last state: DocCollection(testNodeLost//clusterstate.json/463)={ "replicationFactor":"10", "pullReplicas":"0", "router":{"name":"compositeId"}, "maxShardsPerNode":"5", "autoAddReplicas":"false", "nrtReplicas":"10", "tlogReplicas":"0", "autoCreated":"true", "shards":{ "shard20":{ "replicas":{ "core_node191":{ "core":"testNodeLost_shard20_replica_n191", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10110_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node200":{ "core":"testNodeLost_shard20_replica_n200", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10119_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node199":{ "core":"testNodeLost_shard20_replica_n199", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10020_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node198":{ "core":"testNodeLost_shard20_replica_n198", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10037_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node197":{ "core":"testNodeLost_shard20_replica_n197", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10038_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node196":{ "core":"testNodeLost_shard20_replica_n196", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10051_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node195":{ "core":"testNodeLost_shard20_replica_n195", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10085_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node194":{ "core":"testNodeLost_shard20_replica_n194", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10098_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node193":{ "core":"testNodeLost_shard20_replica_n193", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10040_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node192":{ "core":"testNodeLost_shard20_replica_n192", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10022_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"73330000-7fffffff", "state":"active"}, "shard10":{ "replicas":{ "core_node92":{ "core":"testNodeLost_shard10_replica_n92", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10022_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node91":{ "core":"testNodeLost_shard10_replica_n91", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10110_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node94":{ "core":"testNodeLost_shard10_replica_n94", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10098_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node93":{ "core":"testNodeLost_shard10_replica_n93", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10040_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node96":{ "core":"testNodeLost_shard10_replica_n96", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10051_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node95":{ "core":"testNodeLost_shard10_replica_n95", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10085_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node100":{ "core":"testNodeLost_shard10_replica_n100", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10119_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node98":{ "core":"testNodeLost_shard10_replica_n98", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10037_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node97":{ "core":"testNodeLost_shard10_replica_n97", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10038_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node99":{ "core":"testNodeLost_shard10_replica_n99", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10020_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"f3330000-ffffffff", "state":"active"}, "shard11":{ "replicas":{ "core_node108":{ "core":"testNodeLost_shard11_replica_n108", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10099_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node107":{ "core":"testNodeLost_shard11_replica_n107", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10114_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node106":{ "core":"testNodeLost_shard11_replica_n106", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10015_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node105":{ "core":"testNodeLost_shard11_replica_n105", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10068_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node104":{ "core":"testNodeLost_shard11_replica_n104", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10081_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node103":{ "core":"testNodeLost_shard11_replica_n103", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10047_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node102":{ "core":"testNodeLost_shard11_replica_n102", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10011_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node101":{ "core":"testNodeLost_shard11_replica_n101", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10045_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node110":{ "core":"testNodeLost_shard11_replica_n110", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10078_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node109":{ "core":"testNodeLost_shard11_replica_n109", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10097_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"0-ccbffff", "state":"active"}, "shard12":{ "replicas":{ "core_node119":{ "core":"testNodeLost_shard12_replica_n119", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10005_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node118":{ "core":"testNodeLost_shard12_replica_n118", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10043_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node117":{ "core":"testNodeLost_shard12_replica_n117", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10059_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node116":{ "core":"testNodeLost_shard12_replica_n116", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10079_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node115":{ "core":"testNodeLost_shard12_replica_n115", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10120_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node114":{ "core":"testNodeLost_shard12_replica_n114", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10127_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node113":{ "core":"testNodeLost_shard12_replica_n113", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10100_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node112":{ "core":"testNodeLost_shard12_replica_n112", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10061_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node111":{ "core":"testNodeLost_shard12_replica_n111", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10016_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node120":{ "core":"testNodeLost_shard12_replica_n120", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10093_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"ccc0000-1998ffff", "state":"active"}, "shard13":{ "replicas":{ "core_node129":{ "core":"testNodeLost_shard13_replica_n129", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10065_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node128":{ "core":"testNodeLost_shard13_replica_n128", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10121_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node127":{ "core":"testNodeLost_shard13_replica_n127", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10024_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node126":{ "core":"testNodeLost_shard13_replica_n126", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10089_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node125":{ "core":"testNodeLost_shard13_replica_n125", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10076_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node124":{ "core":"testNodeLost_shard13_replica_n124", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10123_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node123":{ "core":"testNodeLost_shard13_replica_n123", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10112_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node122":{ "core":"testNodeLost_shard13_replica_n122", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10094_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node121":{ "core":"testNodeLost_shard13_replica_n121", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10083_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node130":{ "core":"testNodeLost_shard13_replica_n130", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10088_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"19990000-2665ffff", "state":"active"}, "shard2":{ "replicas":{ "core_node18":{ "core":"testNodeLost_shard2_replica_n18", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10043_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node17":{ "core":"testNodeLost_shard2_replica_n17", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10059_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node19":{ "core":"testNodeLost_shard2_replica_n19", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10005_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node20":{ "core":"testNodeLost_shard2_replica_n20", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10093_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node12":{ "core":"testNodeLost_shard2_replica_n12", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10061_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node11":{ "core":"testNodeLost_shard2_replica_n11", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10016_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node14":{ "core":"testNodeLost_shard2_replica_n14", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10127_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node13":{ "core":"testNodeLost_shard2_replica_n13", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10100_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node16":{ "core":"testNodeLost_shard2_replica_n16", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10079_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node15":{ "core":"testNodeLost_shard2_replica_n15", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10120_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"8ccc0000-9998ffff", "state":"active"}, "shard3":{ "replicas":{ "core_node29":{ "core":"testNodeLost_shard3_replica_n29", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10065_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node28":{ "core":"testNodeLost_shard3_replica_n28", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10121_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node30":{ "core":"testNodeLost_shard3_replica_n30", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10088_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node21":{ "core":"testNodeLost_shard3_replica_n21", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10083_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node23":{ "core":"testNodeLost_shard3_replica_n23", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10112_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node22":{ "core":"testNodeLost_shard3_replica_n22", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10094_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node25":{ "core":"testNodeLost_shard3_replica_n25", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10076_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node24":{ "core":"testNodeLost_shard3_replica_n24", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10123_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node27":{ "core":"testNodeLost_shard3_replica_n27", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10024_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node26":{ "core":"testNodeLost_shard3_replica_n26", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10089_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"99990000-a665ffff", "state":"active"}, "shard14":{ "replicas":{ "core_node139":{ "core":"testNodeLost_shard14_replica_n139", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10006_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node138":{ "core":"testNodeLost_shard14_replica_n138", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10026_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node137":{ "core":"testNodeLost_shard14_replica_n137", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10014_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node136":{ "core":"testNodeLost_shard14_replica_n136", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10109_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node202":{ "node_name":"127.0.0.1:10072_solr", "core":"testNodeLost_shard14_replica_n201", "state":"active", "INDEX.sizeInBytes":1, "type":"NRT"}, "core_node135":{ "core":"testNodeLost_shard14_replica_n135", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10115_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node134":{ "core":"testNodeLost_shard14_replica_n134", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10018_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node133":{ "core":"testNodeLost_shard14_replica_n133", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10105_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node132":{ "core":"testNodeLost_shard14_replica_n132", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10117_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node140":{ "core":"testNodeLost_shard14_replica_n140", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10035_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"26660000-3332ffff", "state":"active"}, "shard15":{ "replicas":{ "core_node206":{ "node_name":"127.0.0.1:10059_solr", "core":"testNodeLost_shard15_replica_n205", "state":"active", "INDEX.sizeInBytes":1, "type":"NRT"}, "core_node149":{ "core":"testNodeLost_shard15_replica_n149", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10049_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node148":{ "core":"testNodeLost_shard15_replica_n148", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10067_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node146":{ "core":"testNodeLost_shard15_replica_n146", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10031_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node145":{ "core":"testNodeLost_shard15_replica_n145", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10050_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node144":{ "core":"testNodeLost_shard15_replica_n144", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10034_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node143":{ "core":"testNodeLost_shard15_replica_n143", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10072_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node142":{ "core":"testNodeLost_shard15_replica_n142", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10064_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node141":{ "core":"testNodeLost_shard15_replica_n141", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10008_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node150":{ "core":"testNodeLost_shard15_replica_n150", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10058_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"33330000-3fffffff", "state":"active"}, "shard4":{ "replicas":{ "core_node39":{ "core":"testNodeLost_shard4_replica_n39", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10006_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node204":{ "node_name":"127.0.0.1:10103_solr", "core":"testNodeLost_shard4_replica_n203", "state":"active", "INDEX.sizeInBytes":1, "type":"NRT"}, "core_node40":{ "core":"testNodeLost_shard4_replica_n40", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10035_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node32":{ "core":"testNodeLost_shard4_replica_n32", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10117_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node34":{ "core":"testNodeLost_shard4_replica_n34", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10018_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node33":{ "core":"testNodeLost_shard4_replica_n33", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10105_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node36":{ "core":"testNodeLost_shard4_replica_n36", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10109_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node35":{ "core":"testNodeLost_shard4_replica_n35", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10115_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node38":{ "core":"testNodeLost_shard4_replica_n38", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10026_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node37":{ "core":"testNodeLost_shard4_replica_n37", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10014_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"a6660000-b332ffff", "state":"active"}, "shard16":{ "replicas":{ "core_node159":{ "core":"testNodeLost_shard16_replica_n159", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10095_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node158":{ "core":"testNodeLost_shard16_replica_n158", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10074_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node157":{ "core":"testNodeLost_shard16_replica_n157", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10087_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node156":{ "core":"testNodeLost_shard16_replica_n156", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10066_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node155":{ "core":"testNodeLost_shard16_replica_n155", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10107_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node154":{ "core":"testNodeLost_shard16_replica_n154", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10086_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node153":{ "core":"testNodeLost_shard16_replica_n153", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10012_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node152":{ "core":"testNodeLost_shard16_replica_n152", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10000_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node151":{ "core":"testNodeLost_shard16_replica_n151", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10039_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node160":{ "core":"testNodeLost_shard16_replica_n160", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10122_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"40000000-4ccbffff", "state":"active"}, "shard5":{ "replicas":{ "core_node50":{ "core":"testNodeLost_shard5_replica_n50", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10058_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node41":{ "core":"testNodeLost_shard5_replica_n41", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10008_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node43":{ "core":"testNodeLost_shard5_replica_n43", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10072_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node42":{ "core":"testNodeLost_shard5_replica_n42", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10064_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node45":{ "core":"testNodeLost_shard5_replica_n45", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10050_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node44":{ "core":"testNodeLost_shard5_replica_n44", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10034_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node46":{ "core":"testNodeLost_shard5_replica_n46", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10031_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node49":{ "core":"testNodeLost_shard5_replica_n49", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10049_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node48":{ "core":"testNodeLost_shard5_replica_n48", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10067_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node208":{ "node_name":"127.0.0.1:10053_solr", "core":"testNodeLost_shard5_replica_n207", "state":"active", "INDEX.sizeInBytes":1, "type":"NRT"}}, "range":"b3330000-bfffffff", "state":"active"}, "shard17":{ "replicas":{ "core_node169":{ "core":"testNodeLost_shard17_replica_n169", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10054_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node168":{ "core":"testNodeLost_shard17_replica_n168", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10041_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node167":{ "core":"testNodeLost_shard17_replica_n167", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10071_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node166":{ "core":"testNodeLost_shard17_replica_n166", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10017_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node165":{ "core":"testNodeLost_shard17_replica_n165", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10007_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node164":{ "core":"testNodeLost_shard17_replica_n164", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10125_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node163":{ "core":"testNodeLost_shard17_replica_n163", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10103_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node162":{ "core":"testNodeLost_shard17_replica_n162", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10091_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node161":{ "core":"testNodeLost_shard17_replica_n161", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10116_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node170":{ "core":"testNodeLost_shard17_replica_n170", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10023_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"4ccc0000-5998ffff", "state":"active"}, "shard18":{ "replicas":{ "core_node180":{ "core":"testNodeLost_shard18_replica_n180", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10070_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node179":{ "core":"testNodeLost_shard18_replica_n179", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10048_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node178":{ "core":"testNodeLost_shard18_replica_n178", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10032_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node177":{ "core":"testNodeLost_shard18_replica_n177", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10002_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node176":{ "core":"testNodeLost_shard18_replica_n176", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10075_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node175":{ "core":"testNodeLost_shard18_replica_n175", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10036_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node174":{ "core":"testNodeLost_shard18_replica_n174", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10101_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node173":{ "core":"testNodeLost_shard18_replica_n173", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10033_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node172":{ "core":"testNodeLost_shard18_replica_n172", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10042_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node171":{ "core":"testNodeLost_shard18_replica_n171", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10063_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"59990000-6665ffff", "state":"active"}, "shard19":{ "replicas":{ "core_node190":{ "core":"testNodeLost_shard19_replica_n190", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10090_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node189":{ "core":"testNodeLost_shard19_replica_n189", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10124_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node188":{ "core":"testNodeLost_shard19_replica_n188", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10019_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node187":{ "core":"testNodeLost_shard19_replica_n187", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10013_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node186":{ "core":"testNodeLost_shard19_replica_n186", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10053_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node185":{ "core":"testNodeLost_shard19_replica_n185", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10113_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node184":{ "core":"testNodeLost_shard19_replica_n184", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10106_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node183":{ "core":"testNodeLost_shard19_replica_n183", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10104_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node182":{ "core":"testNodeLost_shard19_replica_n182", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10084_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node181":{ "core":"testNodeLost_shard19_replica_n181", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10062_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"66660000-7332ffff", "state":"active"}, "shard1":{ "replicas":{ "core_node1":{ "core":"testNodeLost_shard1_replica_n1", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10045_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node6":{ "core":"testNodeLost_shard1_replica_n6", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10015_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node10":{ "core":"testNodeLost_shard1_replica_n10", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10078_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node7":{ "core":"testNodeLost_shard1_replica_n7", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10114_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node8":{ "core":"testNodeLost_shard1_replica_n8", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10099_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node9":{ "core":"testNodeLost_shard1_replica_n9", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10097_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node2":{ "core":"testNodeLost_shard1_replica_n2", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10011_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node3":{ "core":"testNodeLost_shard1_replica_n3", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10047_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node4":{ "core":"testNodeLost_shard1_replica_n4", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10081_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node5":{ "core":"testNodeLost_shard1_replica_n5", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10068_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"80000000-8ccbffff", "state":"active"}, "shard6":{ "replicas":{ "core_node60":{ "core":"testNodeLost_shard6_replica_n60", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10122_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node52":{ "core":"testNodeLost_shard6_replica_n52", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10000_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node51":{ "core":"testNodeLost_shard6_replica_n51", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10039_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node54":{ "core":"testNodeLost_shard6_replica_n54", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10086_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node53":{ "core":"testNodeLost_shard6_replica_n53", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10012_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node56":{ "core":"testNodeLost_shard6_replica_n56", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10066_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node55":{ "core":"testNodeLost_shard6_replica_n55", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10107_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node58":{ "core":"testNodeLost_shard6_replica_n58", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10074_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node57":{ "core":"testNodeLost_shard6_replica_n57", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10087_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node59":{ "core":"testNodeLost_shard6_replica_n59", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10095_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"c0000000-cccbffff", "state":"active"}, "shard7":{ "replicas":{ "core_node70":{ "core":"testNodeLost_shard7_replica_n70", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10023_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node61":{ "core":"testNodeLost_shard7_replica_n61", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10116_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node63":{ "core":"testNodeLost_shard7_replica_n63", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10103_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node62":{ "core":"testNodeLost_shard7_replica_n62", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10091_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node65":{ "core":"testNodeLost_shard7_replica_n65", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10007_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node64":{ "core":"testNodeLost_shard7_replica_n64", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10125_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node67":{ "core":"testNodeLost_shard7_replica_n67", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10071_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node66":{ "core":"testNodeLost_shard7_replica_n66", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10017_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node69":{ "core":"testNodeLost_shard7_replica_n69", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10054_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node68":{ "core":"testNodeLost_shard7_replica_n68", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10041_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"cccc0000-d998ffff", "state":"active"}, "shard8":{ "replicas":{ "core_node80":{ "core":"testNodeLost_shard8_replica_n80", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10070_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node72":{ "core":"testNodeLost_shard8_replica_n72", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10042_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node71":{ "core":"testNodeLost_shard8_replica_n71", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10063_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node74":{ "core":"testNodeLost_shard8_replica_n74", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10101_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node73":{ "core":"testNodeLost_shard8_replica_n73", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10033_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node76":{ "core":"testNodeLost_shard8_replica_n76", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10075_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node75":{ "core":"testNodeLost_shard8_replica_n75", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10036_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node78":{ "core":"testNodeLost_shard8_replica_n78", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10032_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node77":{ "core":"testNodeLost_shard8_replica_n77", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10002_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node79":{ "core":"testNodeLost_shard8_replica_n79", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10048_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"d9990000-e665ffff", "state":"active"}, "shard9":{ "replicas":{ "core_node90":{ "core":"testNodeLost_shard9_replica_n90", "leader":"true", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10090_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node81":{ "core":"testNodeLost_shard9_replica_n81", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10062_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node83":{ "core":"testNodeLost_shard9_replica_n83", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10104_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node82":{ "core":"testNodeLost_shard9_replica_n82", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10084_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node85":{ "core":"testNodeLost_shard9_replica_n85", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10113_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node84":{ "core":"testNodeLost_shard9_replica_n84", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10106_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node87":{ "core":"testNodeLost_shard9_replica_n87", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10013_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node86":{ "core":"testNodeLost_shard9_replica_n86", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10053_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node89":{ "core":"testNodeLost_shard9_replica_n89", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10124_solr", "state":"active", "type":"NRT", "SEARCHER.searcher.numDocs":0}, "core_node88":{ "core":"testNodeLost_shard9_replica_n88", "SEARCHER.searcher.maxDoc":0, "SEARCHER.searcher.deletedDocs":0, "INDEX.sizeInBytes":1, "node_name":"127.0.0.1:10019_solr", "state":"down", "type":"NRT", "SEARCHER.searcher.numDocs":0}}, "range":"e6660000-f332ffff", "state":"active"}}}
Stack Trace:
java.util.concurrent.TimeoutException: last state: DocCollection(testNodeLost//clusterstate.json/463)={
"replicationFactor":"10",
"pullReplicas":"0",
"router":{"name":"compositeId"},
"maxShardsPerNode":"5",
"autoAddReplicas":"false",
"nrtReplicas":"10",
"tlogReplicas":"0",
"autoCreated":"true",
"shards":{
"shard20":{
"replicas":{
"core_node191":{
"core":"testNodeLost_shard20_replica_n191",
"leader":"true",
"SEARCHER.searcher.maxDoc":0,
"SEARCHER.searcher.deletedDocs":0,
"INDEX.sizeInBytes":1,
"node_name":"127.0.0.1:10110_solr",
"state":"active",
"type":"NRT",
"SEARCHER.searcher.numDocs":0},
"core_node200":{
"core":"testNodeLost_shard20_replica_n200",
"SEARCHER.searcher.maxDoc":0,
"SEARCHER.searcher.deletedDocs":0,
"INDEX.sizeInBytes":1,
"node_name":"127.0.0.1:10119_solr",
"state":"down",
"type":"NRT",
"SEARCHER.searcher.numDocs":0},
"core_node199":{
"core":"testNodeLost_shard20_replica_n199",
"SEARCHER.searcher.maxDoc":0,
"SEARCHER.searcher.deletedDocs":0,
"INDEX.sizeInBytes":1,
"node_name":"127.0.0.1:10020_solr",
"state":"down",
"type":"NRT",
"SEARCHER.searcher.numDocs":0},
"core_node198":{
"core":"testNodeLost_shard20_replica_n198",
"SEARCHER.searcher.maxDoc":0,
"SEARCHER.searcher.deletedDocs":0,
"INDEX.sizeInBytes":1,
"node_name":"127.0.0.1:10037_solr",
"state":"active",
"type":"NRT",
"SEARCHER.searcher.numDocs":0},
"core_node197":{
"core":"testNodeLost_shard20_replica_n197",
"SEARCHER.searcher.maxDoc":0,
"SEARCHER.searcher.deletedDocs":0,
"INDEX.sizeInBytes":1,
"node_name":"127.0.0.1:10038_solr",
"state":"active",
"type":"NRT",
"SEARCHER.searcher.numDocs":0},
"core_node196":{
"core":"testNodeLost_shard20_replica_n196",
"SEARCHER.searcher.maxDoc":0,
"SEARCHER.searcher.deletedDocs":0,
"INDEX.si
[...truncated too long message...]
dApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
jar-checksums:
[mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/null1137020262
[copy] Copying 39 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/null1137020262
[delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/null1137020262
resolve-example:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
resolve-server:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
jar-checksums:
[mkdir] Created dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/null1304741452
[copy] Copying 247 files to /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/null1304741452
[delete] Deleting directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/null1304741452
check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 78ms :: artifacts dl 1ms
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 8 | 0 | 0 | 0 || 8 | 0 |
---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...
-jenkins-base:
BUILD SUCCESSFUL
Total time: 340 minutes 23 seconds
Archiving artifacts
WARN: No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
WARN: java.lang.InterruptedException: no matches found within 10000
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 75 - Still
Unstable
Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/75/
4 tests failed.
FAILED: org.apache.solr.cloud.MoveReplicaHDFSTest.testFailedMove
Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:33429/solr/MoveReplicaHDFSTest_failed_coll_true, https://127.0.0.1:38009/solr/MoveReplicaHDFSTest_failed_coll_true]
Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:33429/solr/MoveReplicaHDFSTest_failed_coll_true, https://127.0.0.1:38009/solr/MoveReplicaHDFSTest_failed_coll_true]
at __randomizedtesting.SeedInfo.seed([FB959E4A6DBE7BE5:51584DB8DA6DAE35]:0)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:462)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1106)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:993)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
at org.apache.solr.cloud.MoveReplicaTest.testFailedMove(MoveReplicaTest.java:290)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:33429/solr/MoveReplicaHDFSTest_failed_coll_true: no servers hosting shard: shard2
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:436)
... 46 more
FAILED: org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue.testDistributedQueue
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([FB959E4A6DBE7BE5]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestGenericDistributedQueue
Error Message:
Suite timeout exceeded (>= 7200000 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
at __randomizedtesting.SeedInfo.seed([FB959E4A6DBE7BE5]:0)
FAILED: org.apache.solr.metrics.rrd.SolrRrdBackendFactoryTest.testBasic
Error Message:
one expected:<101> but was:<100>
Stack Trace:
java.lang.AssertionError: one expected:<101> but was:<100>
at __randomizedtesting.SeedInfo.seed([FB959E4A6DBE7BE5:506F835FB262FDCB]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.apache.solr.metrics.rrd.SolrRrdBackendFactoryTest.testBasic(SolrRrdBackendFactoryTest.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 14213 lines...]
[junit4] Suite: org.apache.solr.cloud.MoveReplicaHDFSTest
[junit4] 2> 2983691 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_FB959E4A6DBE7BE5-001/init-core-data-001
[junit4] 2> 2983693 WARN (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=12 numCloses=12
[junit4] 2> 2983693 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
[junit4] 2> 2983694 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 2983696 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_FB959E4A6DBE7BE5-001/tempDir-001
[junit4] 2> 2983696 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 2983696 INFO (Thread-4248) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 2983696 INFO (Thread-4248) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 2983705 ERROR (Thread-4248) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 2983796 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:35017
[junit4] 2> 2983801 INFO (zkConnectionManagerCallback-5324-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2983812 INFO (jetty-launcher-5321-thread-2) [ ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
[junit4] 2> 2983812 INFO (jetty-launcher-5321-thread-1) [ ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
[junit4] 2> 2983813 INFO (jetty-launcher-5321-thread-3) [ ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
[junit4] 2> 2983814 INFO (jetty-launcher-5321-thread-4) [ ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
[junit4] 2> 2983814 INFO (jetty-launcher-5321-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2983814 INFO (jetty-launcher-5321-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2983814 INFO (jetty-launcher-5321-thread-2) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 2983815 INFO (jetty-launcher-5321-thread-4) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2983815 INFO (jetty-launcher-5321-thread-4) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2983815 INFO (jetty-launcher-5321-thread-4) [ ] o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 2983816 INFO (jetty-launcher-5321-thread-4) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@64ba8308{/solr,null,AVAILABLE}
[junit4] 2> 2983817 INFO (jetty-launcher-5321-thread-3) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2983817 INFO (jetty-launcher-5321-thread-3) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2983817 INFO (jetty-launcher-5321-thread-3) [ ] o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 2983818 INFO (jetty-launcher-5321-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@12b635aa{/solr,null,AVAILABLE}
[junit4] 2> 2983818 INFO (jetty-launcher-5321-thread-3) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6639a99{/solr,null,AVAILABLE}
[junit4] 2> 2983818 INFO (jetty-launcher-5321-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@b00d0ed{SSL,[ssl, http/1.1]}{127.0.0.1:38009}
[junit4] 2> 2983818 INFO (jetty-launcher-5321-thread-2) [ ] o.e.j.s.Server Started @2983943ms
[junit4] 2> 2983818 INFO (jetty-launcher-5321-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38009}
[junit4] 2> 2983819 INFO (jetty-launcher-5321-thread-3) [ ] o.e.j.s.AbstractConnector Started ServerConnector@66bc256{SSL,[ssl, http/1.1]}{127.0.0.1:53219}
[junit4] 2> 2983826 INFO (jetty-launcher-5321-thread-3) [ ] o.e.j.s.Server Started @2983950ms
[junit4] 2> 2983832 INFO (jetty-launcher-5321-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53219}
[junit4] 2> 2983833 INFO (jetty-launcher-5321-thread-4) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4b67866a{SSL,[ssl, http/1.1]}{127.0.0.1:33429}
[junit4] 2> 2983833 INFO (jetty-launcher-5321-thread-4) [ ] o.e.j.s.Server Started @2983957ms
[junit4] 2> 2983833 INFO (jetty-launcher-5321-thread-4) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33429}
[junit4] 2> 2983833 ERROR (jetty-launcher-5321-thread-3) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2983833 INFO (jetty-launcher-5321-thread-3) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 2983833 INFO (jetty-launcher-5321-thread-3) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 2983833 ERROR (jetty-launcher-5321-thread-4) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2983833 INFO (jetty-launcher-5321-thread-4) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 2983833 INFO (jetty-launcher-5321-thread-3) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2983833 INFO (jetty-launcher-5321-thread-3) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2983833 INFO (jetty-launcher-5321-thread-3) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-06-10T06:38:10.613Z
[junit4] 2> 2983826 ERROR (jetty-launcher-5321-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2983836 INFO (jetty-launcher-5321-thread-2) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 2983836 INFO (jetty-launcher-5321-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 2983836 INFO (jetty-launcher-5321-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2983836 INFO (jetty-launcher-5321-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2983836 INFO (jetty-launcher-5321-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-06-10T06:38:10.616Z
[junit4] 2> 2983825 INFO (jetty-launcher-5321-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2983837 INFO (jetty-launcher-5321-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2983837 INFO (jetty-launcher-5321-thread-1) [ ] o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 2983833 INFO (jetty-launcher-5321-thread-4) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 2983837 INFO (jetty-launcher-5321-thread-4) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2983837 INFO (jetty-launcher-5321-thread-4) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2983837 INFO (jetty-launcher-5321-thread-4) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-06-10T06:38:10.617Z
[junit4] 2> 2983845 INFO (zkConnectionManagerCallback-5328-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2983845 INFO (jetty-launcher-5321-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@16e9ea27{/solr,null,AVAILABLE}
[junit4] 2> 2983845 INFO (jetty-launcher-5321-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@48c07531{SSL,[ssl, http/1.1]}{127.0.0.1:57697}
[junit4] 2> 2983852 INFO (jetty-launcher-5321-thread-1) [ ] o.e.j.s.Server Started @2983976ms
[junit4] 2> 2983852 INFO (jetty-launcher-5321-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=57697}
[junit4] 2> 2983852 ERROR (jetty-launcher-5321-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2983852 INFO (jetty-launcher-5321-thread-1) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 2983853 INFO (jetty-launcher-5321-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 2983853 INFO (jetty-launcher-5321-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2983853 INFO (jetty-launcher-5321-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2983853 INFO (jetty-launcher-5321-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-06-10T06:38:10.633Z
[junit4] 2> 2983854 INFO (jetty-launcher-5321-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 2983859 INFO (zkConnectionManagerCallback-5326-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2983859 INFO (jetty-launcher-5321-thread-3) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 2983860 INFO (zkConnectionManagerCallback-5332-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2983866 INFO (zkConnectionManagerCallback-5330-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2983866 INFO (jetty-launcher-5321-thread-4) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 2983872 INFO (jetty-launcher-5321-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 2984024 INFO (jetty-launcher-5321-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35017/solr
[junit4] 2> 2984044 INFO (zkConnectionManagerCallback-5336-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2984045 DEBUG (jetty-launcher-5321-thread-1) [n:127.0.0.1:57697_solr ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$45/1540906135@e819ee8
[junit4] 2> 2984066 INFO (zkConnectionManagerCallback-5338-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2984189 INFO (jetty-launcher-5321-thread-4) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35017/solr
[junit4] 2> 2984204 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
[junit4] 2> 2984204 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984204 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
[junit4] 2> 2984205 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
[junit4] 2> 2984205 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
[junit4] 2> 2984205 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/security.json
[junit4] 2> 2984208 INFO (jetty-launcher-5321-thread-1) [n:127.0.0.1:57697_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2984208 DEBUG (jetty-launcher-5321-thread-1) [n:127.0.0.1:57697_solr ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/73949263645507590-127.0.0.1:57697_solr-n_0000000000
[junit4] 2> 2984208 INFO (jetty-launcher-5321-thread-1) [n:127.0.0.1:57697_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:57697_solr
[junit4] 2> 2984209 INFO (jetty-launcher-5321-thread-1) [n:127.0.0.1:57697_solr ] o.a.s.c.Overseer Overseer (id=73949263645507590-127.0.0.1:57697_solr-n_0000000000) starting
[junit4] 2> 2984219 INFO (zkConnectionManagerCallback-5344-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2984221 DEBUG (jetty-launcher-5321-thread-4) [n:127.0.0.1:33429_solr ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$45/1540906135@27796c82
[junit4] 2> 2984260 INFO (zkConnectionManagerCallback-5346-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2984266 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
[junit4] 2> 2984266 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984267 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
[junit4] 2> 2984267 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
[junit4] 2> 2984267 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
[junit4] 2> 2984268 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/security.json
[junit4] 2> 2984269 INFO (jetty-launcher-5321-thread-4) [n:127.0.0.1:33429_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2984269 DEBUG (jetty-launcher-5321-thread-4) [n:127.0.0.1:33429_solr ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/73949263645507592-127.0.0.1:33429_solr-n_0000000001
[junit4] 2> 2984269 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/overseer_elect/election/73949263645507590-127.0.0.1:57697_solr-n_0000000000
[junit4] 2> 2984270 DEBUG (jetty-launcher-5321-thread-4) [n:127.0.0.1:33429_solr ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/73949263645507590-127.0.0.1:57697_solr-n_0000000000 to know if I could be the leader
[junit4] 2> 2984270 INFO (jetty-launcher-5321-thread-4) [n:127.0.0.1:33429_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33429_solr
[junit4] 2> 2984270 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984270 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984287 INFO (zkConnectionManagerCallback-5351-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2984288 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
[junit4] 2> 2984288 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984288 INFO (jetty-launcher-5321-thread-1) [n:127.0.0.1:57697_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2984288 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
[junit4] 2> 2984289 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
[junit4] 2> 2984289 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
[junit4] 2> 2984289 INFO (jetty-launcher-5321-thread-1) [n:127.0.0.1:57697_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35017/solr ready
[junit4] 2> 2984294 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984294 INFO (zkCallback-5345-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2984303 DEBUG (OverseerStateUpdate-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.Overseer Starting to work on the main queue
[junit4] 2> 2984305 INFO (OverseerStateUpdate-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 2984306 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue-work
[junit4] 2> 2984306 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
[junit4] 2> 2984310 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
[junit4] 2> 2984311 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Cleaning up work-queue. #Running tasks: 0
[junit4] 2> 2984311 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor RunningTasks: []
[junit4] 2> 2984311 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
[junit4] 2> 2984311 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor CompletedTasks: []
[junit4] 2> 2984311 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: []
[junit4] 2> 2984311 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskQueue Peeking for top 100 elements. ExcludeSet: []
[junit4] 2> 2984312 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
[junit4] 2> 2984315 DEBUG (jetty-launcher-5321-thread-1) [n:127.0.0.1:57697_solr ] o.a.s.c.ZkController Publish node=127.0.0.1:57697_solr as DOWN
[junit4] 2> 2984316 DEBUG (OverseerAutoScalingTriggerThread-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and .scheduled_maintenance triggers
[junit4] 2> 2984316 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
[junit4] 2> 2984317 INFO (jetty-launcher-5321-thread-1) [n:127.0.0.1:57697_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 2984317 INFO (jetty-launcher-5321-thread-1) [n:127.0.0.1:57697_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57697_solr
[junit4] 2> 2984317 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984317 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984329 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984329 INFO (zkCallback-5337-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2984332 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
[junit4] 2> 2984332 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984332 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/autoscaling.json
[junit4] 2> 2984332 INFO (zkCallback-5345-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2984332 DEBUG (OverseerAutoScalingTriggerThread-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 1
[junit4] 2> 2984332 DEBUG (OverseerStateUpdate-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"downnode",
[junit4] 2> "node_name":"127.0.0.1:57697_solr"} current state version: 0
[junit4] 2> 2984333 DEBUG (OverseerAutoScalingTriggerThread-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion -1
[junit4] 2> 2984333 DEBUG (OverseerAutoScalingTriggerThread-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 1
[junit4] 2> 2984333 INFO (jetty-launcher-5321-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35017/solr
[junit4] 2> 2984340 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984340 INFO (zkCallback-5350-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 2984341 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/autoscaling/events/.scheduled_maintenance
[junit4] 2> 2984341 DEBUG (OverseerAutoScalingTriggerThread-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:57697_solr, 127.0.0.1:33429_solr]
[junit4] 2> 2984344 DEBUG (OverseerAutoScalingTriggerThread-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread -- clean old nodeAdded markers
[junit4] 2> 2984345 DEBUG (OverseerAutoScalingTriggerThread-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 1, lastZnodeVersion 1
[junit4] 2> 2984345 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/autoscaling/events/.auto_add_replicas
[junit4] 2> 2984345 DEBUG (ScheduledTrigger-7730-thread-2) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 2
[junit4] 2> 2984350 WARN (jetty-launcher-5321-thread-1) [n:127.0.0.1:57697_solr ] o.a.s.h.a.MetricsHistoryHandler Missing .system, keeping metrics history in memory
[junit4] 2> 2984392 INFO (jetty-launcher-5321-thread-1) [n:127.0.0.1:57697_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57697.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@43069555
[junit4] 2> 2984403 INFO (jetty-launcher-5321-thread-1) [n:127.0.0.1:57697_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57697.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@43069555
[junit4] 2> 2984403 INFO (zkConnectionManagerCallback-5359-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2984403 INFO (jetty-launcher-5321-thread-1) [n:127.0.0.1:57697_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_57697.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@43069555
[junit4] 2> 2984403 INFO (zkConnectionManagerCallback-5362-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2984405 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
[junit4] 2> 2984405 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984405 INFO (jetty-launcher-5321-thread-4) [n:127.0.0.1:33429_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2984405 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
[junit4] 2> 2984406 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
[junit4] 2> 2984406 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
[junit4] 2> 2984406 INFO (jetty-launcher-5321-thread-4) [n:127.0.0.1:33429_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35017/solr ready
[junit4] 2> 2984408 INFO (jetty-launcher-5321-thread-1) [n:127.0.0.1:57697_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_FB959E4A6DBE7BE5-001/tempDir-001/node1/.
[junit4] 2> 2984408 WARN (jetty-launcher-5321-thread-4) [n:127.0.0.1:33429_solr ] o.a.s.h.a.MetricsHistoryHandler Missing .system, keeping metrics history in memory
[junit4] 2> 2984423 DEBUG (jetty-launcher-5321-thread-2) [n:127.0.0.1:38009_solr ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$45/1540906135@40d77299
[junit4] 2> 2984433 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
[junit4] 2> 2984440 INFO (jetty-launcher-5321-thread-4) [n:127.0.0.1:33429_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33429.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@43069555
[junit4] 2> 2984454 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
[junit4] 2> 2984463 INFO (zkConnectionManagerCallback-5366-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2984467 INFO (jetty-launcher-5321-thread-4) [n:127.0.0.1:33429_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33429.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@43069555
[junit4] 2> 2984467 INFO (jetty-launcher-5321-thread-4) [n:127.0.0.1:33429_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33429.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@43069555
[junit4] 2> 2984470 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
[junit4] 2> 2984470 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984471 INFO (jetty-launcher-5321-thread-2) [n:127.0.0.1:38009_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 2984471 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
[junit4] 2> 2984471 INFO (jetty-launcher-5321-thread-4) [n:127.0.0.1:33429_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_FB959E4A6DBE7BE5-001/tempDir-001/node4/.
[junit4] 2> 2984471 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
[junit4] 2> 2984471 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
[junit4] 2> 2984472 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/security.json
[junit4] 2> 2984473 INFO (jetty-launcher-5321-thread-2) [n:127.0.0.1:38009_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2984474 DEBUG (jetty-launcher-5321-thread-2) [n:127.0.0.1:38009_solr ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/73949263645507596-127.0.0.1:38009_solr-n_0000000002
[junit4] 2> 2984474 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/overseer_elect/election/73949263645507592-127.0.0.1:33429_solr-n_0000000001
[junit4] 2> 2984474 DEBUG (jetty-launcher-5321-thread-2) [n:127.0.0.1:38009_solr ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/73949263645507592-127.0.0.1:33429_solr-n_0000000001 to know if I could be the leader
[junit4] 2> 2984474 DEBUG (jetty-launcher-5321-thread-2) [n:127.0.0.1:38009_solr ] o.a.s.c.ZkController Publish node=127.0.0.1:38009_solr as DOWN
[junit4] 2> 2984475 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
[junit4] 2> 2984476 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
[junit4] 2> 2984476 DEBUG (OverseerStateUpdate-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"downnode",
[junit4] 2> "node_name":"127.0.0.1:38009_solr"} current state version: 0
[junit4] 2> 2984476 INFO (jetty-launcher-5321-thread-2) [n:127.0.0.1:38009_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 2984476 INFO (jetty-launcher-5321-thread-2) [n:127.0.0.1:38009_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38009_solr
[junit4] 2> 2984477 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984477 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984477 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984477 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984477 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984478 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984478 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984478 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984478 INFO (zkCallback-5350-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 2984478 INFO (zkCallback-5345-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 2984479 INFO (zkCallback-5337-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 2984488 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984489 INFO (zkCallback-5358-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 2984493 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984493 INFO (zkCallback-5365-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 2984541 INFO (zkConnectionManagerCallback-5373-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2984541 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
[junit4] 2> 2984542 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984542 INFO (jetty-launcher-5321-thread-2) [n:127.0.0.1:38009_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 2984542 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
[junit4] 2> 2984542 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
[junit4] 2> 2984542 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
[junit4] 2> 2984543 INFO (jetty-launcher-5321-thread-2) [n:127.0.0.1:38009_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35017/solr ready
[junit4] 2> 2984543 WARN (jetty-launcher-5321-thread-2) [n:127.0.0.1:38009_solr ] o.a.s.h.a.MetricsHistoryHandler Missing .system, keeping metrics history in memory
[junit4] 2> 2984564 INFO (jetty-launcher-5321-thread-2) [n:127.0.0.1:38009_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38009.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@43069555
[junit4] 2> 2984575 INFO (jetty-launcher-5321-thread-2) [n:127.0.0.1:38009_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38009.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@43069555
[junit4] 2> 2984576 INFO (jetty-launcher-5321-thread-2) [n:127.0.0.1:38009_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38009.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@43069555
[junit4] 2> 2984576 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
[junit4] 2> 2984577 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
[junit4] 2> 2984578 INFO (jetty-launcher-5321-thread-2) [n:127.0.0.1:38009_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_FB959E4A6DBE7BE5-001/tempDir-001/node2/.
[junit4] 2> 2984654 INFO (jetty-launcher-5321-thread-3) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35017/solr
[junit4] 2> 2984655 INFO (zkConnectionManagerCallback-5378-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2984657 DEBUG (jetty-launcher-5321-thread-3) [n:127.0.0.1:53219_solr ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$45/1540906135@1c8b5c1a
[junit4] 2> 2984658 INFO (zkConnectionManagerCallback-5380-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2984666 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
[junit4] 2> 2984667 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984667 INFO (jetty-launcher-5321-thread-3) [n:127.0.0.1:53219_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 2984667 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
[junit4] 2> 2984668 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
[junit4] 2> 2984668 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
[junit4] 2> 2984669 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/security.json
[junit4] 2> 2984670 INFO (jetty-launcher-5321-thread-3) [n:127.0.0.1:53219_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2984671 DEBUG (jetty-launcher-5321-thread-3) [n:127.0.0.1:53219_solr ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/73949263645507599-127.0.0.1:53219_solr-n_0000000003
[junit4] 2> 2984671 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/overseer_elect/election/73949263645507596-127.0.0.1:38009_solr-n_0000000002
[junit4] 2> 2984671 DEBUG (jetty-launcher-5321-thread-3) [n:127.0.0.1:53219_solr ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/73949263645507596-127.0.0.1:38009_solr-n_0000000002 to know if I could be the leader
[junit4] 2> 2984672 DEBUG (jetty-launcher-5321-thread-3) [n:127.0.0.1:53219_solr ] o.a.s.c.ZkController Publish node=127.0.0.1:53219_solr as DOWN
[junit4] 2> 2984673 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
[junit4] 2> 2984673 INFO (jetty-launcher-5321-thread-3) [n:127.0.0.1:53219_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 2984673 INFO (jetty-launcher-5321-thread-3) [n:127.0.0.1:53219_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53219_solr
[junit4] 2> 2984673 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
[junit4] 2> 2984673 DEBUG (OverseerStateUpdate-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"downnode",
[junit4] 2> "node_name":"127.0.0.1:53219_solr"} current state version: 0
[junit4] 2> 2984674 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984674 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984674 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984675 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984675 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984675 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984675 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2984675 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984675 INFO (zkCallback-5365-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2984676 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984676 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984676 INFO (zkCallback-5337-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2984676 INFO (zkCallback-5358-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2984678 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984678 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984678 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984678 INFO (zkCallback-5350-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2984678 INFO (zkCallback-5345-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2984678 INFO (zkCallback-5372-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2984683 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984683 INFO (zkCallback-5379-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 2984705 INFO (zkConnectionManagerCallback-5387-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2984706 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
[junit4] 2> 2984706 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984706 INFO (jetty-launcher-5321-thread-3) [n:127.0.0.1:53219_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 2984706 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
[junit4] 2> 2984707 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
[junit4] 2> 2984707 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
[junit4] 2> 2984708 INFO (jetty-launcher-5321-thread-3) [n:127.0.0.1:53219_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35017/solr ready
[junit4] 2> 2984708 WARN (jetty-launcher-5321-thread-3) [n:127.0.0.1:53219_solr ] o.a.s.h.a.MetricsHistoryHandler Missing .system, keeping metrics history in memory
[junit4] 2> 2984733 INFO (jetty-launcher-5321-thread-3) [n:127.0.0.1:53219_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53219.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@43069555
[junit4] 2> 2984747 INFO (jetty-launcher-5321-thread-3) [n:127.0.0.1:53219_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53219.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@43069555
[junit4] 2> 2984748 INFO (jetty-launcher-5321-thread-3) [n:127.0.0.1:53219_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53219.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@43069555
[junit4] 2> 2984749 INFO (jetty-launcher-5321-thread-3) [n:127.0.0.1:53219_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_FB959E4A6DBE7BE5-001/tempDir-001/node3/.
[junit4] 2> 2984774 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
[junit4] 2> 2984775 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
[junit4] 2> 2984815 INFO (zkConnectionManagerCallback-5390-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2984819 INFO (zkConnectionManagerCallback-5395-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2984820 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
[junit4] 2> 2984820 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2984820 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 2984820 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
[junit4] 2> 2984821 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
[junit4] 2> 2984821 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
[junit4] 2> 2984821 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35017/solr ready
[junit4] 2> 2984897 INFO (qtp1781849107-20874) [n:127.0.0.1:57697_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :overseerstatus with params action=OVERSEERSTATUS&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2984899 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
[junit4] 2> 2984899 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
[junit4] 2> 2984900 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.ZkDistributedQueue Found child node with improper name: qnr-0000000000
[junit4] 2> 2984900 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/overseer/collection-queue-work/qnr-0000000000
[junit4] 2> 2984900 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
[junit4] 2> 2984900 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
[junit4] 2> 2984900 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.ZkDistributedQueue Found child node with improper name: qnr-0000000000
[junit4] 2> 2984901 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskQueue Returning topN elements: [/overseer/collection-queue-work/qn-0000000000, ]
[junit4] 2> 2984901 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Got 1 tasks from work-queue : [[org.apache.solr.cloud.OverseerTaskQueue$QueueEvent@2e684ecf]]
[junit4] 2> 2984901 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Marked task [/overseer/collection-queue-work/qn-0000000000] as running
[junit4] 2> 2984901 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{"operation":"overseerstatus"}
[junit4] 2> 2984901 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Cleaning up work-queue. #Running tasks: 1
[junit4] 2> 2984901 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor RunningTasks: [/overseer/collection-queue-work/qn-0000000000]
[junit4] 2> 2984901 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
[junit4] 2> 2984901 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor CompletedTasks: []
[junit4] 2> 2984902 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: [/overseer/collection-queue-work/qn-0000000000]
[junit4] 2> 2984902 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskQueue Peeking for top 99 elements. ExcludeSet: [/overseer/collection-queue-work/qn-0000000000],[]
[junit4] 2> 2984914 DEBUG (OverseerThreadFactory-7732-thread-1) [ ] o.a.s.c.OverseerTaskProcessor Runner processing /overseer/collection-queue-work/qn-0000000000
[junit4] 2> 2984914 DEBUG (OverseerThreadFactory-7732-thread-1) [ ] o.a.s.c.a.c.OverseerCollectionMessageHandler OverseerCollectionMessageHandler.processMessage : overseerstatus , {"operation":"overseerstatus"}
[junit4] 2> 2984918 DEBUG (OverseerThreadFactory-7732-thread-1) [ ] o.a.s.c.OverseerTaskProcessor Completed task:[/overseer/collection-queue-work/qn-0000000000]
[junit4] 2> 2984919 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: data: /solr/overseer/collection-queue-work/qnr-0000000000
[junit4] 2> 2984919 DEBUG (zkCallback-5337-thread-2) [ ] o.a.s.c.OverseerTaskQueue NodeDataChanged fired on path /overseer/collection-queue-work/qnr-0000000000 state SyncConnected latchEventType null
[junit4] 2> 2984919 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
[junit4] 2> 2984919 DEBUG (OverseerThreadFactory-7732-thread-1) [ ] o.a.s.c.OverseerTaskProcessor Marked task [/overseer/collection-queue-work/qn-0000000000] as completed.
[junit4] 2> 2984919 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
[junit4] 2> 2984919 DEBUG (OverseerThreadFactory-7732-thread-1) [ ] o.a.s.c.OverseerTaskProcessor RunningTasks: []
[junit4] 2> 2984920 DEBUG (OverseerThreadFactory-7732-thread-1) [ ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
[junit4] 2> 2984920 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
[junit4] 2> 2984920 DEBUG (OverseerThreadFactory-7732-thread-1) [ ] o.a.s.c.OverseerTaskProcessor CompletedTasks: [/overseer/collection-queue-work/qn-0000000000]
[junit4] 2> 2984920 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.ZkDistributedQueue Found child node with improper name: qnr-0000000000
[junit4] 2> 2984920 DEBUG (OverseerThreadFactory-7732-thread-1) [ ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: [/overseer/collection-queue-work/qn-0000000000]
[junit4] 2> 2984920 DEBUG (OverseerThreadFactory-7732-thread-1) [ ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Message id:/overseer/collection-queue-work/qn-0000000000 complete, response:{leader=127.0.0.1:57697_solr,overseer_queue_size=0,overseer_work_queue_size=0,overseer_collection_queue_size=2,overseer_operations={am_i_leader={requests=5,errors=0,avgRequestsPerSecond=8.144921493455607,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=0.39515420000000007,medianRequestTime=0.302098,75thPcRequestTime=0.514162,95thPcRequestTime=0.601633,99thPcRequestTime=0.601633,999thPcRequestTime=0.601633},downnode={requests=3,errors=0,avgRequestsPerSecond=5.136289973482756,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=0.012300666666666666,medianRequestTime=0.012747,75thPcRequestTime=0.013263,95thPcRequestTime=0.013263,99thPcRequestTime=0.013263,999thPcRequestTime=0.013263}},collection_operations={am_i_leader={requests=3,errors=0,avgRequestsPerSecond=4.9435309033695,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=0.31234166666666663,medianRequestTime=0.315955,75thPcRequestTime=0.395488,95thPcRequestTime=0.395488,99thPcRequestTime=0.395488,999thPcRequestTime=0.395488},overseerstatus={requests=0,errors=0,avgRequestsPerSecond=0.0,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=0.0,medianRequestTime=0.0,75thPcRequestTime=0.0,95thPcRequestTime=0.0,99thPcRequestTime=0.0,999thPcRequestTime=0.0}},overseer_queue={},overseer_internal_queue={peek={avgRequestsPerSecond=1.6371783719293824,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=0.439052,medianRequestTime=0.439052,75thPcRequestTime=0.439052,95thPcRequestTime=0.439052,99thPcRequestTime=0.439052,999thPcRequestTime=0.439052}},collection_queue={peektopn_wait2000={avgRequestsPerSecond=1.652359876664488,5minRateRequestsPerSecond=0.0,15minRateRequestsPerSecond=0.0,avgTimePerRequest=589.46209,medianRequestTime=589.46209,75thPcRequestTime=589.46209,95thPcRequestTime=589.46209,99thPcRequestTime=589.46209,999thPcRequestTime=589.46209}}}
[junit4] 2> 2984921 INFO (qtp1781849107-20874) [n:127.0.0.1:57697_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={action=OVERSEERSTATUS&wt=javabin&version=2} status=0 QTime=23
[junit4] 2> 2984923 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
[junit4] 1> Formatting using clusterid: testClusterID
[junit4] 2> 2985109 WARN (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
[junit4] 2> 2985122 WARN (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 2985124 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 2985141 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_45911_hdfs____.waa2ce/webapp
[junit4] 2> 2985349 DEBUG (ScheduledTrigger-7730-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 2985595 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45911
[junit4] 2> 2985802 WARN (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 2985804 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 2985836 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_51530_datanode____.femuar/webapp
[junit4] 2> 2986267 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:51530
[junit4] 2> 2986357 DEBUG (ScheduledTrigger-7730-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 2986533 WARN (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
[junit4] 2> 2986539 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.m.log jetty-6.1.26
[junit4] 2> 2986579 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.7.4-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_36392_datanode____yzsts/webapp
[junit4] 2> 2986684 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_FB959E4A6DBE7BE5-001/tempDir-002/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_FB959E4A6DBE7BE5-001/tempDir-002/hdfsBaseDir/data/data2/]] heartbeating to localhost/127.0.0.1:53803) [ ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
[junit4] 2> 2986712 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x668c3131315134: from storage DS-1a4b4b40-43b8-42ca-876d-68b2e001c6f3 node DatanodeRegistration(127.0.0.1:37981, datanodeUuid=a742c4f3-efbc-4cab-bcc3-994eeb26347e, infoPort=40601, infoSecurePort=0, ipcPort=49254, storageInfo=lv=-56;cid=testClusterID;nsid=1194327988;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
[junit4] 2> 2986712 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x668c3131315134: from storage DS-5ad84bb1-c2c0-43b8-a63b-948eabd91f7a node DatanodeRegistration(127.0.0.1:37981, datanodeUuid=a742c4f3-efbc-4cab-bcc3-994eeb26347e, infoPort=40601, infoSecurePort=0, ipcPort=49254, storageInfo=lv=-56;cid=testClusterID;nsid=1194327988;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
[junit4] 2> 2986902 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Got 0 tasks from work-queue : [[]]
[junit4] 2> 2986936 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Cleaning up work-queue. #Running tasks: 0
[junit4] 2> 2986936 INFO (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 2986936 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor RunningTasks: []
[junit4] 2> 2986936 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
[junit4] 2> 2986936 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor CompletedTasks: []
[junit4] 2> 2986936 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: []
[junit4] 2> 2986936 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskQueue Peeking for top 100 elements. ExcludeSet: []
[junit4] 2> 2987136 INFO (SUITE-MoveReplicaHDFSTest-seed#[FB959E4A6DBE7BE5]-worker) [ ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36392
[junit4] 2> 2987358 DEBUG (ScheduledTrigger-7730-thread-4) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 4
[junit4] 2> 2987376 ERROR (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_FB959E4A6DBE7BE5-001/tempDir-002/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_FB959E4A6DBE7BE5-001/tempDir-002/hdfsBaseDir/data/data4/]] heartbeating to localhost/127.0.0.1:53803) [ ] o.a.h.h.s.d.DirectoryScanner dfs.datanode.directoryscan.throttle.limit.ms.per.sec set to value below 1 ms/sec. Assuming default value of 1000
[junit4] 2> 2987404 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x668c315a48f949: from storage DS-dd817b4f-37fb-4e54-962d-8dd8999f327a node DatanodeRegistration(127.0.0.1:60662, datanodeUuid=54a53e55-8572-44ab-b67e-d8b5655c896b, infoPort=39319, infoSecurePort=0, ipcPort=44229, storageInfo=lv=-56;cid=testClusterID;nsid=1194327988;c=0), blocks: 0, hasStaleStorage: true, processing time: 0 msecs
[junit4] 2> 2987404 INFO (Block report processor) [ ] BlockStateChange BLOCK* processReport 0x668c315a48f949: from storage DS-b775105f-8631-47e3-b8d3-dd95ddecc82b node DatanodeRegistration(127.0.0.1:60662, datanodeUuid=54a53e55-8572-44ab-b67e-d8b5655c896b, infoPort=39319, infoSecurePort=0, ipcPort=44229, storageInfo=lv=-56;cid=testClusterID;nsid=1194327988;c=0), blocks: 0, hasStaleStorage: false, processing time: 0 msecs
[junit4] 2> 2987927 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.a.s.SolrTestCaseJ4 ###Starting testNormalMove
[junit4] 2> 2987928 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
[junit4] 2> 2987928 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2987928 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 2987929 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
[junit4] 2> 2987929 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
[junit4] 2> 2987929 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
[junit4] 2> 2987931 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.e.j.s.Server jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_172-b11
[junit4] 2> 2987936 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 2987936 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 2987936 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 2987936 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@30bc38f7{/solr,null,AVAILABLE}
[junit4] 2> 2987937 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.e.j.s.AbstractConnector Started ServerConnector@37934c5d{SSL,[ssl, http/1.1]}{127.0.0.1:55165}
[junit4] 2> 2987937 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.e.j.s.Server Started @2988061ms
[junit4] 2> 2987937 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=55165}
[junit4] 2> 2987937 ERROR (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 2987937 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 2987937 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 2987937 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 2987937 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 2987937 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-06-10T06:38:14.717Z
[junit4] 2> 2987949 INFO (zkConnectionManagerCallback-5399-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2987949 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 2988073 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35017/solr
[junit4] 2> 2988074 INFO (zkConnectionManagerCallback-5403-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2988076 DEBUG (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [n:127.0.0.1:55165_solr ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$45/1540906135@3bfccfb5
[junit4] 2> 2988077 INFO (zkConnectionManagerCallback-5405-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2988083 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
[junit4] 2> 2988083 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2988085 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [n:127.0.0.1:55165_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 2988085 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
[junit4] 2> 2988086 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
[junit4] 2> 2988086 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
[junit4] 2> 2988087 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/security.json
[junit4] 2> 2988088 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [n:127.0.0.1:55165_solr ] o.a.s.c.Overseer Overseer (id=null) closing
[junit4] 2> 2988088 DEBUG (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [n:127.0.0.1:55165_solr ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/73949263645507605-127.0.0.1:55165_solr-n_0000000004
[junit4] 2> 2988089 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/overseer_elect/election/73949263645507599-127.0.0.1:53219_solr-n_0000000003
[junit4] 2> 2988089 DEBUG (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [n:127.0.0.1:55165_solr ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/73949263645507599-127.0.0.1:53219_solr-n_0000000003 to know if I could be the leader
[junit4] 2> 2988089 DEBUG (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [n:127.0.0.1:55165_solr ] o.a.s.c.ZkController Publish node=127.0.0.1:55165_solr as DOWN
[junit4] 2> 2988090 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
[junit4] 2> 2988090 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [n:127.0.0.1:55165_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 2988090 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [n:127.0.0.1:55165_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55165_solr
[junit4] 2> 2988090 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
[junit4] 2> 2988091 DEBUG (OverseerStateUpdate-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
[junit4] 2> "operation":"downnode",
[junit4] 2> "node_name":"127.0.0.1:55165_solr"} current state version: 0
[junit4] 2> 2988091 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2988091 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2988091 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2988091 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2988091 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2988091 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2988091 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2988092 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2988092 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2988092 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/live_nodes
[junit4] 2> 2988096 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2988096 INFO (zkCallback-5358-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 2988096 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2988096 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2988096 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2988096 INFO (zkCallback-5379-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 2988097 INFO (zkCallback-5365-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 2988097 INFO (zkCallback-5345-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 2988097 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2988097 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2988097 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2988097 INFO (zkCallback-5337-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 2988097 INFO (zkCallback-5394-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 2988097 INFO (zkCallback-5350-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 2988097 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2988097 INFO (zkCallback-5394-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 2988097 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2988098 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2988098 INFO (zkCallback-5372-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 2988098 INFO (zkCallback-5404-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 2988098 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2988098 INFO (zkCallback-5386-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (5)
[junit4] 2> 2988118 INFO (zkConnectionManagerCallback-5412-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2988119 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterprops.json
[junit4] 2> 2988120 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/live_nodes
[junit4] 2> 2988120 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [n:127.0.0.1:55165_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (5)
[junit4] 2> 2988120 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: data: /solr/clusterstate.json
[junit4] 2> 2988120 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/collections
[junit4] 2> 2988120 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/aliases.json
[junit4] 2> 2988121 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [n:127.0.0.1:55165_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:35017/solr ready
[junit4] 2> 2988121 WARN (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [n:127.0.0.1:55165_solr ] o.a.s.h.a.MetricsHistoryHandler Missing .system, keeping metrics history in memory
[junit4] 2> 2988146 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [n:127.0.0.1:55165_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55165.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@43069555
[junit4] 2> 2988159 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [n:127.0.0.1:55165_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55165.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@43069555
[junit4] 2> 2988159 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [n:127.0.0.1:55165_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_55165.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@43069555
[junit4] 2> 2988161 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [n:127.0.0.1:55165_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.MoveReplicaHDFSTest_FB959E4A6DBE7BE5-001/tempDir-001/node5/.
[junit4] 2> 2988191 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/queue
[junit4] 2> 2988192 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/queue
[junit4] 2> 2988236 INFO (zkConnectionManagerCallback-5415-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 2988238 INFO (TEST-MoveReplicaHDFSTest.testNormalMove-seed#[FB959E4A6DBE7BE5]) [ ] o.a.s.c.MoveReplicaTest total_jettys: 5
[junit4] 2> 2988268 INFO (qtp1585025282-20885) [n:127.0.0.1:33429_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf1&maxShardsPerNode=2&autoAddReplicas=false&name=MoveReplicaHDFSTest_coll_false&nrtReplicas=2&action=CREATE&numShards=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 2988275 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
[junit4] 2> 2988275 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
[junit4] 2> 2988275 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: create/delete: /solr/overseer/collection-queue-work/qnr-0000000002
[junit4] 2> 2988275 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.ZkDistributedQueue Found child node with improper name: qnr-0000000002
[junit4] 2> 2988275 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch fired: children: /solr/overseer/collection-queue-work
[junit4] 2> 2988276 DEBUG (SyncThread:0) [ ] o.a.s.c.ZkTestServer Watch added: children: /solr/overseer/collection-queue-work
[junit4] 2> 2988276 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.ZkDistributedQueue Found child node with improper name: qnr-0000000002
[junit4] 2> 2988276 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskQueue Returning topN elements: [/overseer/collection-queue-work/qn-0000000002, ]
[junit4] 2> 2988276 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Got 1 tasks from work-queue : [[org.apache.solr.cloud.OverseerTaskQueue$QueueEvent@2e684ed1]]
[junit4] 2> 2988276 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Marked task [/overseer/collection-queue-work/qn-0000000002] as running
[junit4] 2> 2988276 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Overseer Collection Message Handler: Get the message id:/overseer/collection-queue-work/qn-0000000002 message:{
[junit4] 2> "name":"MoveReplicaHDFSTest_coll_false",
[junit4] 2> "fromApi":"true",
[junit4] 2> "replicationFactor":"2",
[junit4] 2> "collection.configName":"conf1",
[junit4] 2> "numShards":"2",
[junit4] 2> "maxShardsPerNode":"2",
[junit4] 2> "autoAddReplicas":"false",
[junit4] 2> "nrtReplicas":"2",
[junit4] 2> "stateFormat":"2",
[junit4] 2> "operation":"create"}
[junit4] 2> 2988277 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor Cleaning up work-queue. #Running tasks: 1
[junit4] 2> 2988277 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor RunningTasks: [/overseer/collection-queue-work/qn-0000000002]
[junit4] 2> 2988277 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor BlockedTasks: []
[junit4] 2> 2988277 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor CompletedTasks: []
[junit4] 2> 2988277 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskProcessor RunningZKTasks: [/overseer/collection-queue-work/qn-0000000002]
[junit4] 2> 2988277 DEBUG (OverseerCollectionConfigSetProcessor-73949263645507590-127.0.0.1:57697_solr-n_0000000000) [ ] o.a.s.c.OverseerTaskQueue Peeking for top 99 elements. ExcludeSet: [/overseer/collection-queue-work/qn-0000000002],[]
[junit4] 2> 2988277 DEBU
[...truncated too long message...]
e = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
jar-checksums:
[mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/null1340167127
[copy] Copying 39 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/null1340167127
[delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/null1340167127
resolve-example:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
resolve-server:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
ivy-fail:
ivy-configure:
[ivy:configure] :: loading settings :: file = /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/lucene/top-level-ivy-settings.xml
resolve:
jar-checksums:
[mkdir] Created dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/null1810442434
[copy] Copying 247 files to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/null1810442434
[delete] Deleting directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-BadApples-Tests-master/solr/null1810442434
check-working-copy:
[ivy:cachepath] :: resolving dependencies :: org.eclipse.jgit#org.eclipse.jgit-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.eclipse.jgit#org.eclipse.jgit;4.6.0.201612231935-r in public
[ivy:cachepath] found com.jcraft#jsch;0.1.53 in public
[ivy:cachepath] found com.googlecode.javaewah#JavaEWAH;1.1.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpclient;4.3.6 in public
[ivy:cachepath] found org.apache.httpcomponents#httpcore;4.3.3 in public
[ivy:cachepath] found commons-logging#commons-logging;1.1.3 in public
[ivy:cachepath] found commons-codec#commons-codec;1.6 in public
[ivy:cachepath] found org.slf4j#slf4j-api;1.7.2 in public
[ivy:cachepath] :: resolution report :: resolve 40ms :: artifacts dl 2ms
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 8 | 0 | 0 | 0 || 8 | 0 |
---------------------------------------------------------------------
[wc-checker] Initializing working copy...
[wc-checker] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[wc-checker] SLF4J: Defaulting to no-operation (NOP) logger implementation
[wc-checker] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[wc-checker] Checking working copy status...
-jenkins-base:
BUILD SUCCESSFUL
Total time: 286 minutes 14 seconds
Archiving artifacts
WARN: No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
WARN: java.lang.InterruptedException: no matches found within 10000
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)