(impala) branch master updated: IMPALA-12556: Fix flaky test test_two_statestored_with_force_active

[wz...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

wzhou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git


The following commit(s) were added to refs/heads/master by this push:
     new 97eae40d1 IMPALA-12556: Fix flaky test test_two_statestored_with_force_active
97eae40d1 is described below

commit 97eae40d10193b5cfb10ca4d2b4034dce44274d0
Author: wzhou-code <wz...@cloudera.com>
AuthorDate: Sun Nov 12 17:47:56 2023 -0800

    IMPALA-12556: Fix flaky test test_two_statestored_with_force_active
    
    Test test_two_statestored_with_force_active failed occasionally by
    cause of both statestore instances assigned with active roles.
    
    This patch fixes the issue to handle the case that both statestore
    instances are restarted with flag "statestore_force_active" in the
    same way as both statestore instances are restarted without flag
    "statestore_force_active".
    
    Testing:
     - Repeatedly ran test_two_statestored_with_force_active on Jenkins for
       hundreds of times without failure.
     - Repeatedly ran test_two_statestored_with_force_active on local
       machine for thousand times without failure.
     - Repeatedly ran all tests in test_statestored_ha.py for over 12 hours
       on Jenkins without failure.
     - Passed core tests.
    
    Change-Id: I3e6f85233ff6fa747a6aa5ef8d093627885d20b2
    Reviewed-on: http://gerrit.cloudera.org:8080/20699
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Wenzhe Zhou <wz...@cloudera.com>
---
 be/src/statestore/statestore-subscriber.cc | 2 ++
 be/src/statestore/statestore.cc            | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/be/src/statestore/statestore-subscriber.cc b/be/src/statestore/statestore-subscriber.cc
index 24b9e7e59..ed48f86e4 100644
--- a/be/src/statestore/statestore-subscriber.cc
+++ b/be/src/statestore/statestore-subscriber.cc
@@ -337,9 +337,11 @@ Status StatestoreSubscriber::Start() {
     if (statestore_is_active) {
       active_statestore_ = statestore_;
       standby_statestore_ = statestore2_;
+      LOG(INFO) << "Set active statestored as " << statestore_->GetAddress();
     } else if (statestore2_is_active) {
       active_statestore_ = statestore2_;
       standby_statestore_ = statestore_;
+      LOG(INFO) << "Set active statestored as " << statestore2_->GetAddress();
     } else {
       // Both statestoreds are not ready. If FLAGS_tolerate_statestore_startup_delay is
       // as true, subscriber enter recovery mode.
diff --git a/be/src/statestore/statestore.cc b/be/src/statestore/statestore.cc
index df2573b2a..ef1f5f539 100644
--- a/be/src/statestore/statestore.cc
+++ b/be/src/statestore/statestore.cc
@@ -1910,14 +1910,14 @@ Status Statestore::ReceiveHaHandshakeRequest(const TUniqueId& peer_statestore_id
   LOG(INFO) << "Receive Statestore HA handshake request";
   lock_guard<mutex> l(ha_lock_);
   peer_statestore_id_ = peer_statestore_id;
-  if (peer_force_active) {
+  if (peer_force_active && !FLAGS_statestore_force_active) {
     is_active_ = false;
     active_status_metric_->SetValue(is_active_);
     ha_active_ss_failure_detector_->UpdateHeartbeat(STATESTORE_ID, true);
     LOG(INFO) << "Set the statestored as standby since the peer is started with force "
               << "active flag";
   } else if (!is_active_) {
-    if (FLAGS_statestore_force_active) {
+    if (FLAGS_statestore_force_active && !peer_force_active) {
       is_active_ = true;
       active_status_metric_->SetValue(is_active_);
       active_version_ = UnixMicros();