[mesos] branch master updated: Fixed flaky SchedulerTest.MasterFailover.

[jo...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

josephwu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/mesos.git


The following commit(s) were added to refs/heads/master by this push:
     new 3126631  Fixed flaky SchedulerTest.MasterFailover.
3126631 is described below

commit 3126631542a25cd7d485e2e44e279dfbe2af7597
Author: Joseph Wu <jo...@apache.org>
AuthorDate: Tue Nov 6 17:11:35 2018 -0800

    Fixed flaky SchedulerTest.MasterFailover.
    
    This test was flaky because there is a double-master-detection race
    after the master fails over.  This test uses the Standalone master
    detector, which keeps a single Master PID in memory and always returns
    that one PID as the leader.  This means there is almost no delay
    between failing over the master and detecting a new leader.
    
    The scheduler in this test tries to send a SUBSCRIBE call to the master
    as soon as the master is detected.  Normally, there will only be two
    total SUBSCRIBE calls during the test, before and after the master
    failover.  However, the test also manually appoints the leader after
    failing over the master.  This step races against the scheduler's own
    retry logic, and can potentially cause a third SUBSCRIBE if the second
    SUBSCRIBE has already started.
    
    Because the scheduler in this test does not enable checkpointing, the
    third SUBSCRIBE will actively disconnect the framework, causing the
    master to remove the framework.  This removal also prevents the
    framework from ever registering again, and thereby times out the test.
    
    This fixes the test to prevent excess master detection events.
    
    We could also change the HTTP scheduler driver to ignore these extra
    master detection events when the master in question has not changed.
    
    Review: https://reviews.apache.org/r/69267
---
 src/tests/scheduler_tests.cpp | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/src/tests/scheduler_tests.cpp b/src/tests/scheduler_tests.cpp
index 0ee5b77..b571bb1 100644
--- a/src/tests/scheduler_tests.cpp
+++ b/src/tests/scheduler_tests.cpp
@@ -292,15 +292,18 @@ TEST_P_TEMP_DISABLED_ON_WINDOWS(SchedulerTest, MasterFailover)
     .WillRepeatedly(Return()); // Ignore future invocations.
 
   // Failover the master.
+  // Also wipe the leading master from the detector, so the scheduler
+  // does not try to reconnect while the master actor is not routable.
   master->reset();
+  detector->appoint(None());
+
   master = StartMaster();
   ASSERT_SOME(master);
 
   AWAIT_READY(disconnected);
 
-  // Scheduler library can fire new master detection more than once.
   EXPECT_CALL(*scheduler, connected(_))
-    .WillRepeatedly(
+    .WillOnce(
         v1::scheduler::SendSubscribe(v1::DEFAULT_FRAMEWORK_INFO, frameworkId));
 
   Future<Event::Subscribed> subscribed2;
@@ -308,6 +311,7 @@ TEST_P_TEMP_DISABLED_ON_WINDOWS(SchedulerTest, MasterFailover)
     .WillOnce(FutureArg<1>(&subscribed2))
     .WillRepeatedly(Return());
 
+  // Give the master leadership now that our scheduler expectations are set up.
   detector->appoint(master.get()->pid);
 
   AWAIT_READY(subscribed2);