[2/2] kudu git commit: De-flake ASAN TestSimultaneousLeaderTransferAndAbruptStepdown

[ad...@apache.org]

De-flake ASAN TestSimultaneousLeaderTransferAndAbruptStepdown

This test is flaky in ASAN when run with the entire kudu-admin-test,
though not when run by itself. I'm not sure why that is, but to try to
improve things, I'm reducing the frequency of the simultaneous abrupt
stepdown and graceful transfer in ASAN mode so there's more time for
the tablet to make progress.

I ran the new test as part of the entire kudu-admin-test 1000 times in
ASAN and saw no failures. I ran kudu-admin-test 1000 times in ASAN with
8 stress threads and saw 4 failures, none of which were
TestSimultaneousLeaderTransferAndAbruptStepdown.

Change-Id: Ic3237f3fde48daeb2f307dfb78d457217df2beed
Reviewed-on: http://gerrit.cloudera.org:8080/11737
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Will Berkeley <wd...@gmail.com>


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/7d65f495
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/7d65f495
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/7d65f495

Branch: refs/heads/master
Commit: 7d65f495db9f663f9626caba9b6880288d6cd83b
Parents: 4abc211
Author: Will Berkeley <wd...@gmail.org>
Authored: Fri Oct 19 11:06:15 2018 -0700
Committer: Will Berkeley <wd...@gmail.com>
Committed: Mon Oct 22 21:58:51 2018 +0000

----------------------------------------------------------------------
 src/kudu/tools/kudu-admin-test.cc | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/7d65f495/src/kudu/tools/kudu-admin-test.cc
----------------------------------------------------------------------
diff --git a/src/kudu/tools/kudu-admin-test.cc b/src/kudu/tools/kudu-admin-test.cc
index 78adeeb..3293fca 100644
--- a/src/kudu/tools/kudu-admin-test.cc
+++ b/src/kudu/tools/kudu-admin-test.cc
@@ -1481,11 +1481,20 @@ TEST_F(AdminCliTest, TestSimultaneousLeaderTransferAndAbruptStepdown) {
   workload.Setup();
   workload.Start();
 
+  // Sometimes this test is flaky under ASAN and the writes are never able to
+  // complete, so we'll back off on how frequently we disrupt leadership to give
+  // time for progress to be made.
+  #if defined(ADDRESS_SANITIZER)
+    const auto leader_change_period_sec = MonoDelta::FromMilliseconds(5000);
+  #else
+    const auto leader_change_period_sec = MonoDelta::FromMilliseconds(1000);
+  #endif
+
   const string& master_addr = cluster_->master()->bound_rpc_addr().ToString();
   while (workload.rows_inserted() < 1000) {
-    // Issue a graceful stepdown and then an abrupt stepdown, every second.
-    // The results are ignored because the tools might fail due to the
-    // constant leadership changes.
+    // Issue a graceful stepdown and then an abrupt stepdown, every
+    // 'leader_change_period_sec' seconds. The results are ignored because the
+    // tools might fail due to the constant leadership changes.
     ignore_result(RunKuduTool({
       "tablet",
       "leader_step_down",
@@ -1499,7 +1508,7 @@ TEST_F(AdminCliTest, TestSimultaneousLeaderTransferAndAbruptStepdown) {
       master_addr,
       tablet_id_
     }));
-    SleepFor(MonoDelta::FromMilliseconds(1000));
+    SleepFor(leader_change_period_sec);
   }
 }