[impala] branch master updated: IMPALA-9262: Bump statestore_heartbeat_frequency_ms in test_kill_impalad_with_running_queries

[st...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

stakiar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git


The following commit(s) were added to refs/heads/master by this push:
     new 55efe5c  IMPALA-9262: Bump statestore_heartbeat_frequency_ms in test_kill_impalad_with_running_queries
55efe5c is described below

commit 55efe5cacab4b1e10cebd4519aadf406120f5949
Author: Sahil Takiar <ta...@gmail.com>
AuthorDate: Wed Jan 29 11:15:25 2020 -0800

    IMPALA-9262: Bump statestore_heartbeat_frequency_ms in test_kill_impalad_with_running_queries
    
    CustomClusterTestSuite sets statestore_heartbeat_frequency_ms to 50,
    overriding the default value of 1000. This means that if a node does not
    respond to heartbeats for 500 milliseconds, it will time out and be
    removed from the cluster (vs the default of 10 seconds).
    
    A low value for statestore_heartbeat_frequency_ms is problematic for
    test_kill_impalad_with_running_queries because if a node is removed from
    the cluster membership, it is removed from the blacklist as well. The
    test asserts that "Blacklisted Executors" shows up in the runtime
    profile of a query immediately after running a query that causes a node
    to be blacklisted. Thus, there is a race condition between running the
    test query vs. the node being removed from the cluster membership.
    Increasing the value of statestore_heartbeat_frequency_ms should
    significantly reduce the chances of such a race.
    
    Testing:
    * Ran test_kill_impalad_with_running_queries locally
    * Not actually able to re-produce the flakiness locally
    
    Change-Id: I84e884efab35649b63db1a7a3b8c49b95b0b4648
    Reviewed-on: http://gerrit.cloudera.org:8080/15131
    Reviewed-by: Thomas Tauber-Marshall <tm...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 tests/custom_cluster/test_blacklist.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/custom_cluster/test_blacklist.py b/tests/custom_cluster/test_blacklist.py
index d2ae254..f9d13cc 100644
--- a/tests/custom_cluster/test_blacklist.py
+++ b/tests/custom_cluster/test_blacklist.py
@@ -116,7 +116,8 @@ class TestBlacklist(CustomClusterTestSuite):
     assert re.search("NumBackends: 3", result.runtime_profile), result.runtime_profile
 
   @pytest.mark.execute_serially
-  @CustomClusterTestSuite.with_args(num_exclusive_coordinators=1)
+  @CustomClusterTestSuite.with_args(num_exclusive_coordinators=1,
+      statestored_args="-statestore_heartbeat_frequency_ms=1000")
   def test_kill_impalad_with_running_queries(self, cursor):
     """Verifies that when an Impala executor is killed while running a query, that the
     Coordinator blacklists the killed executor."""