You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@impala.apache.org by "Sahil Takiar (Code Review)" <ge...@cloudera.org> on 2020/01/07 22:31:32 UTC

[Impala-ASF-CR] IMPALA-9262: De-flake TestBlacklist::test kill impalad with running queries

Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14985


Change subject: IMPALA-9262: De-flake TestBlacklist::test_kill_impalad_with_running_queries
......................................................................

IMPALA-9262: De-flake TestBlacklist::test_kill_impalad_with_running_queries

test_blacklist.py::TestBlacklist::test_kill_impalad_with_running_queries
runs a query asynchronously, waits for it to reach the RUNNING or FINISHED
state, kills an impalad, and then expects a fetch results request for
the query to fail. The test is flaky because it is possible the query can
finish successfully before an impalad is successfully killed.

The fix is to make the query slower using a debug action.

Testing:
* Looped the test for several hours

Change-Id: I8129323a7eb62cef61f1c6c34da06f08cf6d4b06
---
M tests/custom_cluster/test_blacklist.py
1 file changed, 3 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/14985/1
-- 
To view, visit http://gerrit.cloudera.org:8080/14985
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8129323a7eb62cef61f1c6c34da06f08cf6d4b06
Gerrit-Change-Number: 14985
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>

[Impala-ASF-CR] IMPALA-9262: De-flake TestBlacklist::test kill impalad with running queries

Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.

Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14985 )

Change subject: IMPALA-9262: De-flake TestBlacklist::test_kill_impalad_with_running_queries
......................................................................


Patch Set 1:

(2 comments)

Test ran successfully across three builds, and I looped the test on my dev machine for a few hours without issue. So hopefully the flakiness is gone. The last Jenkins failure was due to a flaky test (IMPALA-9292).

http://gerrit.cloudera.org:8080/#/c/14985/1/tests/custom_cluster/test_blacklist.py
File tests/custom_cluster/test_blacklist.py:

http://gerrit.cloudera.org:8080/#/c/14985/1/tests/custom_cluster/test_blacklist.py@129
PS1, Line 129:         'debug_action': '0:GETNEXT:DELAY|1:GETNEXT:DELAY'})
> If I understand correctly, this will just add 200ms or so to execution time
The debug action adds 100ms delay to each call to GetNext for scan nodes 0 and 1. Since many row batches need to be scanned for this query, this causes the runtime to go from a few seconds to a few minutes.

The issue with the WAIT debug action is that it pauses the entire row processing pipeline, which means killing a node doesn't actually cause the the query to fail (since no RPCs between nodes are occurring). The query eventually gets cancelled by the statestore timeout.


http://gerrit.cloudera.org:8080/#/c/14985/1/tests/custom_cluster/test_blacklist.py@149
PS1, Line 149:  
> flake8: E261 at least two spaces before inline comment
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/14985
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8129323a7eb62cef61f1c6c34da06f08cf6d4b06
Gerrit-Change-Number: 14985
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Mon, 13 Jan 2020 16:55:44 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-9262: De-flake TestBlacklist::test kill impalad with running queries

Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.

Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14985 )

Change subject: IMPALA-9262: De-flake TestBlacklist::test_kill_impalad_with_running_queries
......................................................................


Patch Set 2: Verified+1

Carrying Jenkins +1.


-- 
To view, visit http://gerrit.cloudera.org:8080/14985
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8129323a7eb62cef61f1c6c34da06f08cf6d4b06
Gerrit-Change-Number: 14985
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Mon, 13 Jan 2020 16:56:26 +0000
Gerrit-HasComments: No