You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org> on 2018/10/29 18:32:49 UTC
[Impala-ASF-CR] IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
Bikramjeet Vig has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11816
Change subject: IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
......................................................................
IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
This patch adds the queuing status, that is, whether the query was
queued and what was the latest queuing reason, to the ExecSummary.
Also added changes to allow impala-shell to expose this status by
pulling it out from the ExecSummary when either live_summary or
live_progress is set to true.
Testing:
Added custom cluster tests.
Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
---
M be/src/service/impala-server.cc
M common/thrift/ExecStats.thrift
M shell/impala_shell.py
M tests/custom_cluster/test_admission_controller.py
A tests/custom_cluster/test_shell_interactive.py
5 files changed, 100 insertions(+), 24 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/11816/1
--
To view, visit http://gerrit.cloudera.org:8080/11816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
Gerrit-Change-Number: 11816
Gerrit-PatchSet: 1
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
[Impala-ASF-CR] IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11816 )
Change subject: IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
......................................................................
Patch Set 1:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/11816/1/tests/custom_cluster/test_shell_interactive.py
File tests/custom_cluster/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/11816/1/tests/custom_cluster/test_shell_interactive.py@26
PS1, Line 26: class TestShellInteractive(CustomClusterTestSuite):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/11816/1/tests/custom_cluster/test_shell_interactive.py@29
PS1, Line 29:
flake8: E251 unexpected spaces around keyword / parameter equals
--
To view, visit http://gerrit.cloudera.org:8080/11816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
Gerrit-Change-Number: 11816
Gerrit-PatchSet: 1
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pooja Nilangekar <po...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Oct 2018 18:33:36 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11816 )
Change subject: IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/11816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
Gerrit-Change-Number: 11816
Gerrit-PatchSet: 3
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pooja Nilangekar <po...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Oct 2018 21:59:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11816 )
Change subject: IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3376/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/11816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
Gerrit-Change-Number: 11816
Gerrit-PatchSet: 3
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pooja Nilangekar <po...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Oct 2018 21:59:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11816 )
Change subject: IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
......................................................................
Patch Set 2: Code-Review+2
carrying over +2
--
To view, visit http://gerrit.cloudera.org:8080/11816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
Gerrit-Change-Number: 11816
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pooja Nilangekar <po...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Oct 2018 21:58:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Hello Pooja Nilangekar, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/11816
to look at the new patch set (#2).
Change subject: IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
......................................................................
IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
This patch adds the queuing status, that is, whether the query was
queued and what was the latest queuing reason, to the ExecSummary.
Also added changes to allow impala-shell to expose this status by
pulling it out from the ExecSummary when either live_summary or
live_progress is set to true.
Testing:
Added custom cluster tests.
Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
---
M be/src/service/impala-server.cc
M common/thrift/ExecStats.thrift
M shell/impala_shell.py
M tests/custom_cluster/test_admission_controller.py
A tests/custom_cluster/test_shell_interactive.py
5 files changed, 102 insertions(+), 24 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/11816/2
--
To view, visit http://gerrit.cloudera.org:8080/11816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
Gerrit-Change-Number: 11816
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pooja Nilangekar <po...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11816 )
Change subject: IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/1203/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
Gerrit-Change-Number: 11816
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pooja Nilangekar <po...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Oct 2018 22:29:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11816 )
Change subject: IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
......................................................................
IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
This patch adds the queuing status, that is, whether the query was
queued and what was the latest queuing reason, to the ExecSummary.
Also added changes to allow impala-shell to expose this status by
pulling it out from the ExecSummary when either live_summary or
live_progress is set to true.
Testing:
Added custom cluster tests.
Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
Reviewed-on: http://gerrit.cloudera.org:8080/11816
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/service/impala-server.cc
M common/thrift/ExecStats.thrift
M shell/impala_shell.py
M tests/custom_cluster/test_admission_controller.py
A tests/custom_cluster/test_shell_interactive.py
5 files changed, 102 insertions(+), 24 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/11816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
Gerrit-Change-Number: 11816
Gerrit-PatchSet: 4
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pooja Nilangekar <po...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11816 )
Change subject: IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
......................................................................
Patch Set 1: Code-Review+2
(4 comments)
http://gerrit.cloudera.org:8080/#/c/11816/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11816/1//COMMIT_MSG@9
PS1, Line 9: This patch adds the queuing status, that is, whether the query was
Can you mention that this is showed when either live_summary or live_progress is true.
http://gerrit.cloudera.org:8080/#/c/11816/1/common/thrift/ExecStats.thrift
File common/thrift/ExecStats.thrift:
http://gerrit.cloudera.org:8080/#/c/11816/1/common/thrift/ExecStats.thrift@104
PS1, Line 104: queued
"is currently queued by admission control" to be totally clear
http://gerrit.cloudera.org:8080/#/c/11816/1/common/thrift/ExecStats.thrift@107
PS1, Line 107: // Contains the latest queuing reason if the query is queued.
Same as above.
http://gerrit.cloudera.org:8080/#/c/11816/1/shell/impala_shell.py
File shell/impala_shell.py:
http://gerrit.cloudera.org:8080/#/c/11816/1/shell/impala_shell.py@969
PS1, Line 969: if summary.is_queued:
It's a little subtle that this handles both None and False. I'll leave it up to you whether this is worth documenting.
--
To view, visit http://gerrit.cloudera.org:8080/11816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
Gerrit-Change-Number: 11816
Gerrit-PatchSet: 1
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pooja Nilangekar <po...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Oct 2018 20:40:01 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
Posted by "Pooja Nilangekar (Code Review)" <ge...@cloudera.org>.
Pooja Nilangekar has posted comments on this change. ( http://gerrit.cloudera.org:8080/11816 )
Change subject: IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
......................................................................
Patch Set 1: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11816/1/tests/custom_cluster/test_shell_interactive.py
File tests/custom_cluster/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/11816/1/tests/custom_cluster/test_shell_interactive.py@38
PS1, Line 38: proc = pexpect.spawn(' '.join([SHELL_CMD, "-i localhost:21000"]))
I think it would help connecting to another port (i.e. another impalad) to ensure that the admission controller status is propagated across impalads. (Not sure if it'd make things flaky though).
--
To view, visit http://gerrit.cloudera.org:8080/11816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
Gerrit-Change-Number: 11816
Gerrit-PatchSet: 1
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pooja Nilangekar <po...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Oct 2018 19:24:48 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11816 )
Change subject: IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/1196/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/11816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
Gerrit-Change-Number: 11816
Gerrit-PatchSet: 1
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pooja Nilangekar <po...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Oct 2018 19:07:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11816 )
Change subject: IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/11816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
Gerrit-Change-Number: 11816
Gerrit-PatchSet: 3
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pooja Nilangekar <po...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Oct 2018 02:02:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/11816 )
Change subject: IMPALA-7735: Expose queuing status in ExecSummary and impala-shell
......................................................................
Patch Set 1: Code-Review+2
(6 comments)
http://gerrit.cloudera.org:8080/#/c/11816/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11816/1//COMMIT_MSG@9
PS1, Line 9: This patch adds the queuing status, that is, whether the query was
> Can you mention that this is showed when either live_summary or live_progre
yup, thats already mentioned at the end of this paragraph
http://gerrit.cloudera.org:8080/#/c/11816/1/common/thrift/ExecStats.thrift
File common/thrift/ExecStats.thrift:
http://gerrit.cloudera.org:8080/#/c/11816/1/common/thrift/ExecStats.thrift@104
PS1, Line 104: queued
> "is currently queued by admission control" to be totally clear
Done
http://gerrit.cloudera.org:8080/#/c/11816/1/common/thrift/ExecStats.thrift@107
PS1, Line 107: // Contains the latest queuing reason if the query is queued.
> Same as above.
Done
http://gerrit.cloudera.org:8080/#/c/11816/1/tests/custom_cluster/test_shell_interactive.py
File tests/custom_cluster/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/11816/1/tests/custom_cluster/test_shell_interactive.py@26
PS1, Line 26: class TestShellInteractive(CustomClusterTestSuite):
> flake8: E302 expected 2 blank lines, found 1
Done
http://gerrit.cloudera.org:8080/#/c/11816/1/tests/custom_cluster/test_shell_interactive.py@29
PS1, Line 29:
> flake8: E251 unexpected spaces around keyword / parameter equals
Done
http://gerrit.cloudera.org:8080/#/c/11816/1/tests/custom_cluster/test_shell_interactive.py@38
PS1, Line 38: proc = pexpect.spawn(' '.join([SHELL_CMD, "-i localhost:21000"]))
> I think it would help connecting to another port (i.e. another impalad) to
Good point! but we already have tests that rely on pool stats being propagated across the cluster
--
To view, visit http://gerrit.cloudera.org:8080/11816
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibde447b01559b9f0f3e970d4fa10f6ee4064bd49
Gerrit-Change-Number: 11816
Gerrit-PatchSet: 1
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Pooja Nilangekar <po...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Oct 2018 21:58:21 +0000
Gerrit-HasComments: Yes