You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Tim Armstrong (Code Review)" <ge...@cloudera.org> on 2018/10/04 23:07:39 UTC
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11587
Change subject: IMPALA-7643: report # queries actually executing in stress test
......................................................................
IMPALA-7643: report # queries actually executing in stress test
With admission control it's interesting to separate out two categories
of queries:
1. Queries that have started up and are executing
2. Queries that have not made it that far yet, e.g. are waiting to
establish a client connection (hitting --fe_service_threads limit),
are in the planner, are queued in admission control or are starting
up.
We now call 1+2 "Active" and 2 "Executing".
Example output:
Done | Active | Executing | Mem Lmt Ex | AC Reject | AC Timeout | Time Out | Cancel | Err | Incorrect | Next Qry Mem Lmt | Tot Qry Mem Lmt | Tracked Mem | RSS Mem
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
0 | 10 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 510 | 3922 | 158 | 4541
0 | 20 | 10 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 390 | 8534 | 570 | 4517
Refactored QueryRunner.run_query() to reduce nesting and make it more
readable.
Testing:
Ran local stress tests with and without --test_admission_control set.
Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
---
M tests/stress/concurrent_select.py
1 file changed, 105 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/11587/1
--
To view, visit http://gerrit.cloudera.org:8080/11587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Gerrit-Change-Number: 11587
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11587 )
Change subject: IMPALA-7643: report # queries actually executing in stress test
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3288/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/11587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Gerrit-Change-Number: 11587
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Oct 2018 23:21:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11587 )
Change subject: IMPALA-7643: report # queries actually executing in stress test
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/11587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Gerrit-Change-Number: 11587
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 06 Oct 2018 03:13:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11587 )
Change subject: IMPALA-7643: report # queries actually executing in stress test
......................................................................
Patch Set 1:
Fixed bot comments
--
To view, visit http://gerrit.cloudera.org:8080/11587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Gerrit-Change-Number: 11587
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Oct 2018 23:13:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11587 )
Change subject: IMPALA-7643: report # queries actually executing in stress test
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/949/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Gerrit-Change-Number: 11587
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Oct 2018 23:40:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11587 )
Change subject: IMPALA-7643: report # queries actually executing in stress test
......................................................................
Patch Set 2:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/11587/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11587/2//COMMIT_MSG@17
PS2, Line 17: 2
> 1?
Oops, yep had it reversed.
http://gerrit.cloudera.org:8080/#/c/11587/2/tests/stress/concurrent_select.py
File tests/stress/concurrent_select.py:
http://gerrit.cloudera.org:8080/#/c/11587/2/tests/stress/concurrent_select.py@1037
PS2, Line 1037: self.parent = None
The binary search caught a pretty bad bug - we don't set runner.parent when doing the binary search. I reran it successfully.
http://gerrit.cloudera.org:8080/#/c/11587/2/tests/stress/concurrent_select.py@1144
PS2, Line 1144: canel
> cancel
Done
http://gerrit.cloudera.org:8080/#/c/11587/2/tests/stress/concurrent_select.py@1153
PS2, Line 1153: if (not started_running and query_state not in ('PENDING_STATE',
: 'INITIALIZED_STATE')):
: started_running = True
: self.parent.increment_num_queries_started_running()
:
: if query_state not in ('PENDING_STATE', 'INITIALIZED_STATE', 'RUNNING_STATE'):
: return True
> Please leave a comment on the logic here. While it makes sense if you look
Good point. I tried to clarify it with some comments, lmk if it could be further improved.
http://gerrit.cloudera.org:8080/#/c/11587/2/tests/stress/concurrent_select.py@1164
PS2, Line 1164: self._cancel(cursor, report)
: if not started_running:
: self.parent.increment_num_queries_started_running()
> I'm trying to understand the reasoning for the increment here. I think that
I think the naming of the variable is a little misleading since it includes queries that failed or were cancelled. I made the name longer and more verbose, LMK if it makes it clearer (I think it probably does).
--
To view, visit http://gerrit.cloudera.org:8080/11587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Gerrit-Change-Number: 11587
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Oct 2018 23:10:08 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11587 )
Change subject: IMPALA-7643: report # queries actually executing in stress test
......................................................................
Patch Set 1:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py
File tests/stress/concurrent_select.py:
http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py@431
PS1, Line 431: d
flake8: E501 line too long (91 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py@465
PS1, Line 465: ,
flake8: E501 line too long (101 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py@688
PS1, Line 688: a
flake8: E501 line too long (94 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py@1078
PS1, Line 1078: e
flake8: E501 line too long (94 > 90 characters)
http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py@1113
PS1, Line 1113: d
flake8: E303 too many blank lines (2)
http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py@1141
PS1, Line 1141: d
flake8: E303 too many blank lines (2)
http://gerrit.cloudera.org:8080/#/c/11587/1/tests/stress/concurrent_select.py@1153
PS1, Line 1153: :
flake8: E501 line too long (91 > 90 characters)
--
To view, visit http://gerrit.cloudera.org:8080/11587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Gerrit-Change-Number: 11587
Gerrit-PatchSet: 1
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Oct 2018 23:08:23 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11587 )
Change subject: IMPALA-7643: report # queries actually executing in stress test
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/11587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Gerrit-Change-Number: 11587
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Oct 2018 23:21:30 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Michael Brown, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11587
to look at the new patch set (#2).
Change subject: IMPALA-7643: report # queries actually executing in stress test
......................................................................
IMPALA-7643: report # queries actually executing in stress test
With admission control it's interesting to separate out two categories
of queries:
1. Queries that have started up and are executing
2. Queries that have not made it that far yet, e.g. are waiting to
establish a client connection (hitting --fe_service_threads limit),
are in the planner, are queued in admission control or are starting
up.
We now call 1+2 "Active" and 2 "Executing".
Example output:
Done | Active | Executing | Mem Lmt Ex | AC Reject | AC Timeout | Time Out | Cancel | Err | Incorrect | Next Qry Mem Lmt | Tot Qry Mem Lmt | Tracked Mem | RSS Mem
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
0 | 10 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 510 | 3922 | 158 | 4541
0 | 20 | 10 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 390 | 8534 | 570 | 4517
Refactored QueryRunner.run_query() to reduce nesting and make it more
readable.
Testing:
Ran local stress tests with and without --test_admission_control set.
Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
---
M tests/stress/concurrent_select.py
1 file changed, 109 insertions(+), 56 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/11587/2
--
To view, visit http://gerrit.cloudera.org:8080/11587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Gerrit-Change-Number: 11587
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has posted comments on this change. ( http://gerrit.cloudera.org:8080/11587 )
Change subject: IMPALA-7643: report # queries actually executing in stress test
......................................................................
Patch Set 2:
(4 comments)
Can you also do a quick binary search? It's fine if you do that just with TPCH or something. Sometimes that can regress.
http://gerrit.cloudera.org:8080/#/c/11587/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11587/2//COMMIT_MSG@17
PS2, Line 17: 2
1?
http://gerrit.cloudera.org:8080/#/c/11587/2/tests/stress/concurrent_select.py
File tests/stress/concurrent_select.py:
http://gerrit.cloudera.org:8080/#/c/11587/2/tests/stress/concurrent_select.py@1144
PS2, Line 1144: canel
cancel
http://gerrit.cloudera.org:8080/#/c/11587/2/tests/stress/concurrent_select.py@1153
PS2, Line 1153: if (not started_running and query_state not in ('PENDING_STATE',
: 'INITIALIZED_STATE')):
: started_running = True
: self.parent.increment_num_queries_started_running()
:
: if query_state not in ('PENDING_STATE', 'INITIALIZED_STATE', 'RUNNING_STATE'):
: return True
Please leave a comment on the logic here. While it makes sense if you look at common/thrift/hive-1-api/TCLIService.thrift, it makes less sense if you're just trying to read the code in this file. In particular, the confusing part is that something called "increment_num_queries_started_running()" (with the word "running" in it) gets called if query_state = 'RUNNING_STATE', but we're still not ready to return True L1159 due to L1158. A bit of commenting makes the logic more clear for readers.
http://gerrit.cloudera.org:8080/#/c/11587/2/tests/stress/concurrent_select.py@1164
PS2, Line 1164: self._cancel(cursor, report)
: if not started_running:
: self.parent.increment_num_queries_started_running()
I'm trying to understand the reasoning for the increment here. I think that means a comment is necessary. :)
--
To view, visit http://gerrit.cloudera.org:8080/11587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Gerrit-Change-Number: 11587
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Oct 2018 17:57:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11587 )
Change subject: IMPALA-7643: report # queries actually executing in stress test
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/971/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Gerrit-Change-Number: 11587
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Oct 2018 23:43:41 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Hello Michael Brown, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11587
to look at the new patch set (#3).
Change subject: IMPALA-7643: report # queries actually executing in stress test
......................................................................
IMPALA-7643: report # queries actually executing in stress test
With admission control it's interesting to separate out two categories
of queries:
1. Queries that have started up and are executing
2. Queries that have not made it that far yet, e.g. are waiting to
establish a client connection (hitting --fe_service_threads limit),
are in the planner, are queued in admission control or are starting
up.
We now call 1+2 "Active" and 1 "Executing".
Example output:
Done | Active | Executing | Mem Lmt Ex | AC Reject | AC Timeout | Time Out | Cancel | Err | Incorrect | Next Qry Mem Lmt | Tot Qry Mem Lmt | Tracked Mem | RSS Mem
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
0 | 10 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 510 | 3922 | 158 | 4541
0 | 20 | 10 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 390 | 8534 | 570 | 4517
Refactored QueryRunner.run_query() to reduce nesting and make it more
readable.
Testing:
Ran local stress tests with and without --test_admission_control set.
Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
---
M tests/stress/concurrent_select.py
1 file changed, 117 insertions(+), 58 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/87/11587/3
--
To view, visit http://gerrit.cloudera.org:8080/11587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Gerrit-Change-Number: 11587
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11587 )
Change subject: IMPALA-7643: report # queries actually executing in stress test
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/950/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Gerrit-Change-Number: 11587
Gerrit-PatchSet: 2
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 Oct 2018 23:45:36 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Posted by "Michael Brown (Code Review)" <ge...@cloudera.org>.
Michael Brown has posted comments on this change. ( http://gerrit.cloudera.org:8080/11587 )
Change subject: IMPALA-7643: report # queries actually executing in stress test
......................................................................
Patch Set 3: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11587/2/tests/stress/concurrent_select.py
File tests/stress/concurrent_select.py:
http://gerrit.cloudera.org:8080/#/c/11587/2/tests/stress/concurrent_select.py@1153
PS2, Line 1153: while True:
: query_state = cursor.status()
: # Check if the query got past the PENDING/INITIALIZED states, either because
: # it's executing or hit an error.
: if (not started_running_or_cancelled and query_state not in ('PENDING_STATE',
: 'INITIALIZED_STATE')):
: started_run
> Good point. I tried to clarify it with some comments, lmk if it could be fu
Perfect, thanks.
--
To view, visit http://gerrit.cloudera.org:8080/11587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Gerrit-Change-Number: 11587
Gerrit-PatchSet: 3
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 Oct 2018 23:13:40 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7643: report # queries actually executing in stress test
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11587 )
Change subject: IMPALA-7643: report # queries actually executing in stress test
......................................................................
IMPALA-7643: report # queries actually executing in stress test
With admission control it's interesting to separate out two categories
of queries:
1. Queries that have started up and are executing
2. Queries that have not made it that far yet, e.g. are waiting to
establish a client connection (hitting --fe_service_threads limit),
are in the planner, are queued in admission control or are starting
up.
We now call 1+2 "Active" and 1 "Executing".
Example output:
Done | Active | Executing | Mem Lmt Ex | AC Reject | AC Timeout | Time Out | Cancel | Err | Incorrect | Next Qry Mem Lmt | Tot Qry Mem Lmt | Tracked Mem | RSS Mem
0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | |
0 | 10 | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 510 | 3922 | 158 | 4541
0 | 20 | 10 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 390 | 8534 | 570 | 4517
Refactored QueryRunner.run_query() to reduce nesting and make it more
readable.
Testing:
Ran local stress tests with and without --test_admission_control set.
Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Reviewed-on: http://gerrit.cloudera.org:8080/11587
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M tests/stress/concurrent_select.py
1 file changed, 117 insertions(+), 58 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/11587
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I5692e8e5ba3224becefc24437197bf5a5b450335
Gerrit-Change-Number: 11587
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>