You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Zihao Ye (Code Review)" <ge...@cloudera.org> on 2023/11/07 03:43:59 UTC

[Impala-ASF-CR] IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell

Zihao Ye has uploaded this change for review. ( http://gerrit.cloudera.org:8080/20672


Change subject: IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell
......................................................................

IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell

This patch modifies the query progress bar in impala-shell by replacing
the scan progress with the query progress. The query progress is
calculated using the number of completed fragment instances divided by
the total number of fragment instances. This approach provides a more
accurate reflection of the actual completion progress of the query
compared to the scan progress, which is calculated based on the
completed ScanRanges divided by the total ScanRanges. Especially for
computation-intensive queries that involve complex aggregations or
sorting, such as tpcds query78, there is often additional time required
for computation even after the scanning is complete. In such cases,
displaying 100% scan progress would be inaccurate.

Change-Id: I11a704885505442b7499a026fcee3b86696cd064
---
M shell/impala_shell.py
1 file changed, 5 insertions(+), 4 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/20672/1
-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 1
Gerrit-Owner: Zihao Ye <ey...@163.com>

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................


Patch Set 5: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9913/


-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 5
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zihao Ye <ey...@163.com>
Gerrit-Comment-Date: Fri, 10 Nov 2023 04:29:01 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Zihao Ye (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20672

to look at the new patch set (#4).

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................

IMPALA-12544: Add additional query progress reporting for the shell

This patch modifies the dynamic query progress reporting in impala-shell
by adding an extra query progress bar below the scan progress bar.
The query progress is calculated using the number of completed fragment
instances divided by the total number of fragment instances. Compared to
the scan progress, which is calculated based on completed scan ranges
divided by the total scan ranges, the query progress provides a more
accurate reflection of the actual completion progress of the query.
Particularly for computationally intensive queries involving complex
aggregations or sorting, such as tpcds query78, there is often
additional computation time required after the scanning is complete. In
such cases, displaying only 100% scan progress would be inaccurate.

Change-Id: I11a704885505442b7499a026fcee3b86696cd064
---
M shell/impala_shell.py
1 file changed, 15 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/20672/4
-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 4
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................


Patch Set 4: Code-Review+1

(1 comment)

This looks nice, thanks!

http://gerrit.cloudera.org:8080/#/c/20672/4/shell/impala_shell.py
File shell/impala_shell.py:

http://gerrit.cloudera.org:8080/#/c/20672/4/shell/impala_shell.py@1307
PS4, Line 1307:         if not progress.total_fragment_instances and \
Is there a scenario where we'll have scan range info but not fragments? We could still show something in that case.



-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 4
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Nov 2023 17:13:46 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has removed a vote on this change.

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................


Removed Code-Review+2 by Michael Smith <mi...@cloudera.com>
-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 5
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zihao Ye <ey...@163.com>

[Impala-ASF-CR] IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14355/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 3
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Nov 2023 08:37:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9913/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 5
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zihao Ye <ey...@163.com>
Gerrit-Comment-Date: Thu, 09 Nov 2023 18:28:45 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................


Patch Set 5:

> Patch Set 5: Verified-1
> 
> Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/9913/

Some of the shell tests, notably shell/test_shell_commandline.py::TestImpalaShell::test_cancellation, seem to be failing and/or timing out.


-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 5
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zihao Ye <ey...@163.com>
Gerrit-Comment-Date: Fri, 10 Nov 2023 06:07:58 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20672/2/shell/impala_shell.py
File shell/impala_shell.py:

http://gerrit.cloudera.org:8080/#/c/20672/2/shell/impala_shell.py@1313
PS2, Line 1313: /
flake8: W504 line break after binary operator



-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 2
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Nov 2023 06:03:12 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Zihao Ye (Code Review)" <ge...@cloudera.org>.
Zihao Ye has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................


Patch Set 5:

(1 comment)

Thanks for the review!

http://gerrit.cloudera.org:8080/#/c/20672/5/shell/impala_shell.py
File shell/impala_shell.py:

http://gerrit.cloudera.org:8080/#/c/20672/5/shell/impala_shell.py@1317
PS5, Line 1317:  
> nit: Remove this blank?

This blank is for aligning the progress bar behind it, see the comments of IMPALA-12544.



-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 5
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zihao Ye <ey...@163.com>
Gerrit-Comment-Date: Thu, 09 Nov 2023 07:53:58 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................


Patch Set 5: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 5
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zihao Ye <ey...@163.com>
Gerrit-Comment-Date: Thu, 09 Nov 2023 18:28:29 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14388/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 5
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Thu, 09 Nov 2023 03:38:41 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/9915/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 6
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zihao Ye <ey...@163.com>
Gerrit-Comment-Date: Fri, 10 Nov 2023 09:14:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................

IMPALA-12544: Add additional query progress reporting for the shell

This patch modifies the dynamic query progress reporting in impala-shell
by adding an extra query progress bar below the scan progress bar.
The query progress is calculated using the number of completed fragment
instances divided by the total number of fragment instances. Compared to
the scan progress, which is calculated based on completed scan ranges
divided by the total scan ranges, the query progress provides a more
accurate reflection of the actual completion progress of the query.
Particularly for computationally intensive queries involving complex
aggregations or sorting, such as tpcds query78, there is often
additional computation time required after the scanning is complete. In
such cases, displaying only 100% scan progress would be inaccurate.

Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Reviewed-on: http://gerrit.cloudera.org:8080/20672
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
---
M shell/impala_shell.py
M tests/custom_cluster/test_web_pages.py
M tests/shell/test_shell_commandline.py
M tests/shell/util.py
4 files changed, 20 insertions(+), 10 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Michael Smith: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 7
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zihao Ye <ey...@163.com>

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................


Patch Set 6: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 6
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zihao Ye <ey...@163.com>
Gerrit-Comment-Date: Fri, 10 Nov 2023 13:39:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Zihao Ye (Code Review)" <ge...@cloudera.org>.
Hello Yifan Zhang, Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20672

to look at the new patch set (#6).

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................

IMPALA-12544: Add additional query progress reporting for the shell

This patch modifies the dynamic query progress reporting in impala-shell
by adding an extra query progress bar below the scan progress bar.
The query progress is calculated using the number of completed fragment
instances divided by the total number of fragment instances. Compared to
the scan progress, which is calculated based on completed scan ranges
divided by the total scan ranges, the query progress provides a more
accurate reflection of the actual completion progress of the query.
Particularly for computationally intensive queries involving complex
aggregations or sorting, such as tpcds query78, there is often
additional computation time required after the scanning is complete. In
such cases, displaying only 100% scan progress would be inaccurate.

Change-Id: I11a704885505442b7499a026fcee3b86696cd064
---
M shell/impala_shell.py
M tests/custom_cluster/test_web_pages.py
M tests/shell/test_shell_commandline.py
M tests/shell/util.py
4 files changed, 20 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/20672/6
-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 6
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zihao Ye <ey...@163.com>

[Impala-ASF-CR] IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell

Posted by "Zihao Ye (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20672

to look at the new patch set (#3).

Change subject: IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell
......................................................................

IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell

This patch modifies the query progress bar in impala-shell by replacing
the scan progress with the query progress. The query progress is
calculated using the number of completed fragment instances divided by
the total number of fragment instances. This approach provides a more
accurate reflection of the actual completion progress of the query
compared to the scan progress, which is calculated based on the
completed ScanRanges divided by the total ScanRanges. Especially for
computation-intensive queries that involve complex aggregations or
sorting, such as tpcds query78, there is often additional time required
for computation even after the scanning is complete. In such cases,
displaying 100% scan progress would be inaccurate.

Change-Id: I11a704885505442b7499a026fcee3b86696cd064
---
M shell/impala_shell.py
1 file changed, 5 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/20672/3
-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 3
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14369/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 4
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 08 Nov 2023 04:15:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14405/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 6
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zihao Ye <ey...@163.com>
Gerrit-Comment-Date: Fri, 10 Nov 2023 08:16:12 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................


Patch Set 6: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 6
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zihao Ye <ey...@163.com>
Gerrit-Comment-Date: Fri, 10 Nov 2023 16:18:42 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell
......................................................................


Patch Set 2:

Build Failed 

https://jenkins.impala.io/job/gerrit-code-review-checks/14353/ : Initial code review checks failed. See linked job for details on the failure.


-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 2
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Nov 2023 06:26:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell

Posted by "Zihao Ye (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20672

to look at the new patch set (#2).

Change subject: IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell
......................................................................

IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell

This patch modifies the query progress bar in impala-shell by replacing
the scan progress with the query progress. The query progress is
calculated using the number of completed fragment instances divided by
the total number of fragment instances. This approach provides a more
accurate reflection of the actual completion progress of the query
compared to the scan progress, which is calculated based on the
completed ScanRanges divided by the total ScanRanges. Especially for
computation-intensive queries that involve complex aggregations or
sorting, such as tpcds query78, there is often additional time required
for computation even after the scanning is complete. In such cases,
displaying 100% scan progress would be inaccurate.

Change-Id: I11a704885505442b7499a026fcee3b86696cd064
---
M shell/impala_shell.py
1 file changed, 5 insertions(+), 4 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/20672/2
-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 2
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Zihao Ye (Code Review)" <ge...@cloudera.org>.
Hello Michael Smith, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/20672

to look at the new patch set (#5).

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................

IMPALA-12544: Add additional query progress reporting for the shell

This patch modifies the dynamic query progress reporting in impala-shell
by adding an extra query progress bar below the scan progress bar.
The query progress is calculated using the number of completed fragment
instances divided by the total number of fragment instances. Compared to
the scan progress, which is calculated based on completed scan ranges
divided by the total scan ranges, the query progress provides a more
accurate reflection of the actual completion progress of the query.
Particularly for computationally intensive queries involving complex
aggregations or sorting, such as tpcds query78, there is often
additional computation time required after the scanning is complete. In
such cases, displaying only 100% scan progress would be inaccurate.

Change-Id: I11a704885505442b7499a026fcee3b86696cd064
---
M shell/impala_shell.py
1 file changed, 16 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/72/20672/5
-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 5
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Zihao Ye (Code Review)" <ge...@cloudera.org>.
Zihao Ye has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20672/4/shell/impala_shell.py
File shell/impala_shell.py:

http://gerrit.cloudera.org:8080/#/c/20672/4/shell/impala_shell.py@1307
PS4, Line 1307:         if not progress.total_fragment_instances and \
> Is there a scenario where we'll have scan range info but not
 > fragments? We could still show something in that case.

Thank you for the reminder! Assuming version consistency, it's difficult for me to imagine a scenario where we have scan range information but not fragment information. However, if we were to connect to an older version of Impala, it's possible that such a situation could arise. In that case, we should continue to display the existing content.



-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 4
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Zihao Ye <ey...@163.com>
Gerrit-Comment-Date: Thu, 09 Nov 2023 03:42:52 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12544: Add additional query progress reporting for the shell

Posted by "Yifan Zhang (Code Review)" <ge...@cloudera.org>.
Yifan Zhang has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Add additional query progress reporting for the shell
......................................................................


Patch Set 5: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20672/5/shell/impala_shell.py
File shell/impala_shell.py:

http://gerrit.cloudera.org:8080/#/c/20672/5/shell/impala_shell.py@1317
PS5, Line 1317:  
nit: Remove this blank?



-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 5
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Yifan Zhang <ch...@163.com>
Gerrit-Reviewer: Zihao Ye <ey...@163.com>
Gerrit-Comment-Date: Thu, 09 Nov 2023 07:28:12 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/20672/1/shell/impala_shell.py
File shell/impala_shell.py:

http://gerrit.cloudera.org:8080/#/c/20672/1/shell/impala_shell.py@1313
PS1, Line 1313: /
flake8: W504 line break after binary operator



-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 1
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Nov 2023 03:44:57 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/20672 )

Change subject: IMPALA-12544: Replace scan progress with query progress as progress reporting for the shell
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/14350/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/20672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a704885505442b7499a026fcee3b86696cd064
Gerrit-Change-Number: 20672
Gerrit-PatchSet: 1
Gerrit-Owner: Zihao Ye <ey...@163.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 07 Nov 2023 04:07:10 +0000
Gerrit-HasComments: No