You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sahil Takiar (Code Review)" <ge...@cloudera.org> on 2020/01/29 19:22:20 UTC
[Impala-ASF-CR] IMPALA-9262: Bump statestore heartbeat frequency ms in test kill impalad with running queries
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15131
Change subject: IMPALA-9262: Bump statestore_heartbeat_frequency_ms in test_kill_impalad_with_running_queries
......................................................................
IMPALA-9262: Bump statestore_heartbeat_frequency_ms in test_kill_impalad_with_running_queries
CustomClusterTestSuite sets statestore_heartbeat_frequency_ms to 50,
overriding the default value of 1000. This means that if a node does not
respond to heartbeats for 500 milliseconds, it will time out and be
removed from the cluster (vs the default of 10 seconds).
A low value for statestore_heartbeat_frequency_ms is problematic for
test_kill_impalad_with_running_queries because if a node is removed from
the cluster membership, it is removed from the blacklist as well. The
test asserts that "Blacklisted Executors" shows up in the runtime
profile of a query immediately after running a query that causes a node
to be blacklisted. Thus, there is a race condition between running the
test query vs. the node being removed from the cluster membership.
Increasing the value of statestore_heartbeat_frequency_ms should
significantly reduce the chances of such a race.
Testing:
* Ran test_kill_impalad_with_running_queries locally
* Not actually able to re-produce the flakiness locally
Change-Id: I84e884efab35649b63db1a7a3b8c49b95b0b4648
---
M tests/custom_cluster/test_blacklist.py
1 file changed, 2 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/15131/1
--
To view, visit http://gerrit.cloudera.org:8080/15131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I84e884efab35649b63db1a7a3b8c49b95b0b4648
Gerrit-Change-Number: 15131
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
[Impala-ASF-CR] IMPALA-9262: Bump statestore heartbeat frequency ms in test kill impalad with running queries
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15131 )
Change subject: IMPALA-9262: Bump statestore_heartbeat_frequency_ms in test_kill_impalad_with_running_queries
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/5541/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/15131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84e884efab35649b63db1a7a3b8c49b95b0b4648
Gerrit-Change-Number: 15131
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Wed, 29 Jan 2020 20:08:12 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9262: Bump statestore heartbeat frequency ms in test kill impalad with running queries
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15131 )
Change subject: IMPALA-9262: Bump statestore_heartbeat_frequency_ms in test_kill_impalad_with_running_queries
......................................................................
IMPALA-9262: Bump statestore_heartbeat_frequency_ms in test_kill_impalad_with_running_queries
CustomClusterTestSuite sets statestore_heartbeat_frequency_ms to 50,
overriding the default value of 1000. This means that if a node does not
respond to heartbeats for 500 milliseconds, it will time out and be
removed from the cluster (vs the default of 10 seconds).
A low value for statestore_heartbeat_frequency_ms is problematic for
test_kill_impalad_with_running_queries because if a node is removed from
the cluster membership, it is removed from the blacklist as well. The
test asserts that "Blacklisted Executors" shows up in the runtime
profile of a query immediately after running a query that causes a node
to be blacklisted. Thus, there is a race condition between running the
test query vs. the node being removed from the cluster membership.
Increasing the value of statestore_heartbeat_frequency_ms should
significantly reduce the chances of such a race.
Testing:
* Ran test_kill_impalad_with_running_queries locally
* Not actually able to re-produce the flakiness locally
Change-Id: I84e884efab35649b63db1a7a3b8c49b95b0b4648
Reviewed-on: http://gerrit.cloudera.org:8080/15131
Reviewed-by: Thomas Tauber-Marshall <tm...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M tests/custom_cluster/test_blacklist.py
1 file changed, 2 insertions(+), 1 deletion(-)
Approvals:
Thomas Tauber-Marshall: Looks good to me, approved
Impala Public Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/15131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I84e884efab35649b63db1a7a3b8c49b95b0b4648
Gerrit-Change-Number: 15131
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
[Impala-ASF-CR] IMPALA-9262: Bump statestore heartbeat frequency ms in test kill impalad with running queries
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15131 )
Change subject: IMPALA-9262: Bump statestore_heartbeat_frequency_ms in test_kill_impalad_with_running_queries
......................................................................
Patch Set 1:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5475/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/15131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84e884efab35649b63db1a7a3b8c49b95b0b4648
Gerrit-Change-Number: 15131
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Wed, 29 Jan 2020 20:00:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9262: Bump statestore heartbeat frequency ms in test kill impalad with running queries
Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/15131 )
Change subject: IMPALA-9262: Bump statestore_heartbeat_frequency_ms in test_kill_impalad_with_running_queries
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/15131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84e884efab35649b63db1a7a3b8c49b95b0b4648
Gerrit-Change-Number: 15131
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Wed, 29 Jan 2020 19:56:28 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9262: Bump statestore heartbeat frequency ms in test kill impalad with running queries
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15131 )
Change subject: IMPALA-9262: Bump statestore_heartbeat_frequency_ms in test_kill_impalad_with_running_queries
......................................................................
Patch Set 1: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/15131
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I84e884efab35649b63db1a7a3b8c49b95b0b4648
Gerrit-Change-Number: 15131
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Thu, 30 Jan 2020 00:45:21 +0000
Gerrit-HasComments: No