You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2020/06/18 14:10:52 UTC
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16096
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
IMPALA-9569: Show progress bar and live_summary of the retried query
Impala-shell periodically calls GetExecSummary() when the query is
queuing or running. If the query is being retried, GetExecSummary()
should return the TExecSummary of the retried query. So the progress bar
and live_summary can reflect the most recent state.
This patch also modifies get_summary() to return retry information in
error_logs of TExecSummary. Impala-shell and other clients can print the
info right after the query starts being retried.
Example output when the retried query is running:
Query: select count(*) from functional.alltypes where bool_col = sleep(60)
Query submitted at: 2020-06-18 22:08:49 (Coordinator: http://quanlong-OptiPlex-BJ:25000)
Query progress can be monitored at: http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=9444fe7f0df0da28:29134b0800000000
Failed due to unreachable impalad(s): quanlong-OptiPlex-BJ:22001
Retrying query using query id: 5748d9a3ccc28ba8:a75e2fab00000000
Retried query link: http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=5748d9a3ccc28ba8:a75e2fab00000000
[################################################## ] 50%
Tests:
- Manually verify the progress bar and live_summary work when the query
is being retried.
Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
---
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M shell/impala_shell.py
3 files changed, 27 insertions(+), 7 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/16096/1
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/16096/1/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/16096/1/be/src/service/impala-server.cc@746
PS1, Line 746: if (query_handle->IsRetriedQuery()) {
> what happens if the above if condition is true? e.g. if 'query_handle->exec
Yes, you are right. The retried query will be re-scheduled and should wait for admission control again. Will move this block outside.
http://gerrit.cloudera.org:8080/#/c/16096/1/be/src/service/impala-server.cc@749
PS1, Line 749: DCHECK(status.ok()) << "Original query handle not found!";
> This might actually cause issues. Looking through this method, if GetQueryH
Nice catch! The only error is query_driver not found. I thought it's ok since GetQueryHandle uses the same query_id as GetActiveQueryHandle at line 717. After digged deeper, I realize there is a race if the query is unregistered concurrently between this two lines, then the query driver will be not found in the query_driver_map.
I think a better way is getting the original query handle along with the latest query handle at line 717 in one call. Will update the codes.
http://gerrit.cloudera.org:8080/#/c/16096/1/be/src/service/impala-server.cc@763
PS1, Line 763: // Look for the query in completed query log.
> somewhat related to this patch, what happens if you run a query in the impa
Yeah, not just "summary", but "profile" also has the same problem. Created IMPALA-9870 for this. I didn't modify this part since it's not used in the live_summary code path.
For "profile" command, I think we need to decide whether to return two profiles since one profile is already big enough...
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jun 2020 02:52:15 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
Patch Set 4:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6041/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Jun 2020 00:24:02 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6031/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jun 2020 03:27:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Fix progress bar and live summary to show info of the retried query
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Fix progress bar and live_summary to show info of the retried query
......................................................................
Patch Set 7: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 7
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Jun 2020 07:08:44 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6369/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jun 2020 14:38:43 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
Patch Set 4:
> Patch Set 3:
>
> When I apply this patch and IMPALA-9213 locally and do a clean build, all the tests in test_query_retries.py start to fail.
It's still ok in my local dev box. Let me rebase the patch and run a GVO dryrun again.
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Jun 2020 00:23:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Fix progress bar and live summary to show info of the retried query
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Fix progress bar and live_summary to show info of the retried query
......................................................................
IMPALA-9569: Fix progress bar and live_summary to show info of the retried query
Impala-shell periodically calls GetExecSummary() when the query is
queuing or running. If the query is being retried, GetExecSummary()
should return the TExecSummary of the retried query. So the progress bar
and live_summary can reflect the most recent state.
This patch also modifies get_summary() to return retry information in
error_logs of TExecSummary. Impala-shell and other clients can print the
info right after the query starts being retried. Modified impala-shell
to print the retried query link when the retried query is running.
Example output when the retried query is running:
Query: select count(*) from functional.alltypes where bool_col = sleep(60)
Query submitted at: 2020-06-18 22:08:49 (Coordinator: http://quanlong-OptiPlex-BJ:25000)
Query progress can be monitored at: http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=9444fe7f0df0da28:29134b0800000000
Failed due to unreachable impalad(s): quanlong-OptiPlex-BJ:22001
Retrying query using query id: 5748d9a3ccc28ba8:a75e2fab00000000
Retried query link: http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=5748d9a3ccc28ba8:a75e2fab00000000
[############################### ] 50%
Tests:
- Manually verify the progress bar and live_summary work when the query
is being retried.
- Add tests in test_query_retries.py to validate the get_summary()
results.
Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Reviewed-on: http://gerrit.cloudera.org:8080/16096
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M shell/impala_shell.py
M tests/common/impala_connection.py
M tests/custom_cluster/test_query_retries.py
5 files changed, 120 insertions(+), 28 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 8
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/16096/1/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/16096/1/be/src/service/impala-server.cc@746
PS1, Line 746: if (query_handle->IsRetriedQuery()) {
what happens if the above if condition is true? e.g. if 'query_handle->exec_state() == ClientRequestState::ExecState::PENDING' is true? should this code block be moved outside this if statement?
http://gerrit.cloudera.org:8080/#/c/16096/1/be/src/service/impala-server.cc@749
PS1, Line 749: DCHECK(status.ok()) << "Original query handle not found!";
This might actually cause issues. Looking through this method, if GetQueryHandle returns a failure status, it looks through the 'completed query log' to search for the query instead rather than throwing an error.
http://gerrit.cloudera.org:8080/#/c/16096/1/be/src/service/impala-server.cc@763
PS1, Line 763: // Look for the query in completed query log.
somewhat related to this patch, what happens if you run a query in the impala-shell, it gets retried and completes successfully, and then the user runs the 'summary;' command? does it display the summary of the original query or the retried query? does it have any indication that the query is retried? since summaries are pretty small, we might be able to get do something like:
localhost:21000] default> select 1;
<for arguments sake lets sake this query got retried>
[localhost:21000] default> summary;
Query was retried. Original query id: <original-id>, Retried query id <retried-id>
Original Summary:
+----------+--------+-------+----------+----------+-------+------------+----------+---------------+--------+
| Operator | #Hosts | #Inst | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail |
+----------+--------+-------+----------+----------+-------+------------+----------+---------------+--------+
| F00:ROOT | 1 | 1 | 14.57us | 14.57us | | | 0 B | 0 B | |
| 00:UNION | 1 | 1 | 31.12us | 31.12us | 1 | 1 | 0 B | 0 B | |
+----------+--------+-------+----------+----------+-------+------------+----------+---------------+--------+
Retried Summary:
+----------+--------+-------+----------+----------+-------+------------+----------+---------------+--------+
| Operator | #Hosts | #Inst | Avg Time | Max Time | #Rows | Est. #Rows | Peak Mem | Est. Peak Mem | Detail |
+----------+--------+-------+----------+----------+-------+------------+----------+---------------+--------+
| F00:ROOT | 1 | 1 | 14.57us | 14.57us | | | 0 B | 0 B | |
| 00:UNION | 1 | 1 | 31.12us | 31.12us | 1 | 1 | 0 B | 0 B | |
+----------+--------+-------+----------+----------+-------+------------+----------+---------------+--------+
This is slightly orthogonal to the goal of this patch so feel free to file a follow up JIRA on this and we can pursue it separately.
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Thu, 18 Jun 2020 17:10:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
Patch Set 5:
Fixed an issue that the retry info is not printed above the progress bar due to the last refactoring. Added test coverage for this.
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Jun 2020 07:11:23 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Fix progress bar and live summary to show info of the retried query
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Fix progress bar and live_summary to show info of the retried query
......................................................................
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16096/5/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/16096/5/be/src/service/impala-server.cc@759
PS5, Line 759: if (query_handle->IsRetriedQuery()) result->__set_error_logs(error_logs);
> is there a specific reason to set the error logs at the end of the method r
Sorry, I should merge the if-statement here... We can't set the error logs inside the above if-statement because line 748 will overwrite the TExecSummary result in calling GetTExecSummary().
http://gerrit.cloudera.org:8080/#/c/16096/5/shell/impala_shell.py
File shell/impala_shell.py:
http://gerrit.cloudera.org:8080/#/c/16096/5/shell/impala_shell.py@1100
PS5, Line 1100: if query_id_search and len(query_id_search.groups()) >= 1:
> the length of the group should only be 1, right?
Yeah, done.
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Sun, 28 Jun 2020 06:36:16 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
Patch Set 5:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/16096/5/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/16096/5/be/src/service/impala-server.cc@759
PS5, Line 759: if (query_handle->IsRetriedQuery()) result->__set_error_logs(error_logs);
is there a specific reason to set the error logs at the end of the method rather than inside the above if statement that adds the entries to error_logs?
i guess one benefit is that if the Coordinator is null, then the error logs can still be exposed?
might be good to add a code comment mentioning this because its not immediately obvious give that this if statement condition is the same as the above if statement condition
http://gerrit.cloudera.org:8080/#/c/16096/5/shell/impala_shell.py
File shell/impala_shell.py:
http://gerrit.cloudera.org:8080/#/c/16096/5/shell/impala_shell.py@1100
PS5, Line 1100: if query_id_search and len(query_id_search.groups()) >= 1:
the length of the group should only be 1, right?
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Sat, 27 Jun 2020 02:21:33 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9569: Fix progress bar and live summary to show info of the retried query
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Fix progress bar and live_summary to show info of the retried query
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Jun 2020 00:38:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
Patch Set 3:
When I apply this patch and IMPALA-9213 locally and do a clean build, all the tests in test_query_retries.py start to fail.
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Tue, 23 Jun 2020 17:13:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Fix progress bar and live summary to show info of the retried query
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Fix progress bar and live_summary to show info of the retried query
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6075/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 7
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Jun 2020 07:08:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
Patch Set 4: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 4
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Jun 2020 05:30:31 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 3
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jun 2020 08:29:03 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6405/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Wed, 24 Jun 2020 07:38:16 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Fix progress bar and live summary to show info of the retried query
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Sahil Takiar, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16096
to look at the new patch set (#6).
Change subject: IMPALA-9569: Fix progress bar and live_summary to show info of the retried query
......................................................................
IMPALA-9569: Fix progress bar and live_summary to show info of the retried query
Impala-shell periodically calls GetExecSummary() when the query is
queuing or running. If the query is being retried, GetExecSummary()
should return the TExecSummary of the retried query. So the progress bar
and live_summary can reflect the most recent state.
This patch also modifies get_summary() to return retry information in
error_logs of TExecSummary. Impala-shell and other clients can print the
info right after the query starts being retried. Modified impala-shell
to print the retried query link when the retried query is running.
Example output when the retried query is running:
Query: select count(*) from functional.alltypes where bool_col = sleep(60)
Query submitted at: 2020-06-18 22:08:49 (Coordinator: http://quanlong-OptiPlex-BJ:25000)
Query progress can be monitored at: http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=9444fe7f0df0da28:29134b0800000000
Failed due to unreachable impalad(s): quanlong-OptiPlex-BJ:22001
Retrying query using query id: 5748d9a3ccc28ba8:a75e2fab00000000
Retried query link: http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=5748d9a3ccc28ba8:a75e2fab00000000
[############################### ] 50%
Tests:
- Manually verify the progress bar and live_summary work when the query
is being retried.
- Add tests in test_query_retries.py to validate the get_summary()
results.
Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
---
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M shell/impala_shell.py
M tests/common/impala_connection.py
M tests/custom_cluster/test_query_retries.py
5 files changed, 120 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/16096/6
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
[Impala-ASF-CR] IMPALA-9569: Fix progress bar and live summary to show info of the retried query
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Fix progress bar and live_summary to show info of the retried query
......................................................................
Patch Set 6:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6442/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 6
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Sun, 28 Jun 2020 07:03:59 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Fix progress bar and live summary to show info of the retried query
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Fix progress bar and live_summary to show info of the retried query
......................................................................
Patch Set 7: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 7
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Jun 2020 12:11:22 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Sahil Takiar, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16096
to look at the new patch set (#2).
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
IMPALA-9569: Show progress bar and live_summary of the retried query
Impala-shell periodically calls GetExecSummary() when the query is
queuing or running. If the query is being retried, GetExecSummary()
should return the TExecSummary of the retried query. So the progress bar
and live_summary can reflect the most recent state.
This patch also modifies get_summary() to return retry information in
error_logs of TExecSummary. Impala-shell and other clients can print the
info right after the query starts being retried.
Example output when the retried query is running:
Query: select count(*) from functional.alltypes where bool_col = sleep(60)
Query submitted at: 2020-06-18 22:08:49 (Coordinator: http://quanlong-OptiPlex-BJ:25000)
Query progress can be monitored at: http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=9444fe7f0df0da28:29134b0800000000
Failed due to unreachable impalad(s): quanlong-OptiPlex-BJ:22001
Retrying query using query id: 5748d9a3ccc28ba8:a75e2fab00000000
Retried query link: http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=5748d9a3ccc28ba8:a75e2fab00000000
[############################### ] 50%
Tests:
- Manually verify the progress bar and live_summary work when the query
is being retried.
Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
---
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M shell/impala_shell.py
3 files changed, 50 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/16096/2
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16096 )
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/6377/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Jun 2020 03:54:35 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9569: Show progress bar and live summary of the retried query
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Hello Sahil Takiar, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/16096
to look at the new patch set (#5).
Change subject: IMPALA-9569: Show progress bar and live_summary of the retried query
......................................................................
IMPALA-9569: Show progress bar and live_summary of the retried query
Impala-shell periodically calls GetExecSummary() when the query is
queuing or running. If the query is being retried, GetExecSummary()
should return the TExecSummary of the retried query. So the progress bar
and live_summary can reflect the most recent state.
This patch also modifies get_summary() to return retry information in
error_logs of TExecSummary. Impala-shell and other clients can print the
info right after the query starts being retried.
Example output when the retried query is running:
Query: select count(*) from functional.alltypes where bool_col = sleep(60)
Query submitted at: 2020-06-18 22:08:49 (Coordinator: http://quanlong-OptiPlex-BJ:25000)
Query progress can be monitored at: http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=9444fe7f0df0da28:29134b0800000000
Failed due to unreachable impalad(s): quanlong-OptiPlex-BJ:22001
Retrying query using query id: 5748d9a3ccc28ba8:a75e2fab00000000
Retried query link: http://quanlong-OptiPlex-BJ:25000/query_plan?query_id=5748d9a3ccc28ba8:a75e2fab00000000
[############################### ] 50%
Tests:
- Manually verify the progress bar and live_summary work when the query
is being retried.
Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
---
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M shell/impala_shell.py
M tests/common/impala_connection.py
M tests/custom_cluster/test_query_retries.py
5 files changed, 121 insertions(+), 28 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/16096/5
--
To view, visit http://gerrit.cloudera.org:8080/16096
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8f96919f00e0b64d589efd15b6b5ec82fb725d56
Gerrit-Change-Number: 16096
Gerrit-PatchSet: 5
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>