You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Joe McDonnell (Code Review)" <ge...@cloudera.org> on 2025/05/22 20:35:10 UTC
[Impala-ASF-CR] IMPALA-14087: Fix shell live progress output display issue on Python 3
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22941
Change subject: IMPALA-14087: Fix shell live_progress output display issue on Python 3
......................................................................
IMPALA-14087: Fix shell live_progress output display issue on Python 3
When running the shell in a terminal with live_progress=true, live
progress overwrites its output by using the ANSI up character to
rewrite lines with updated on the query progress. On Python 3,
we found that the updates to clear the live progress were overwriting
the actual output in the terminal. e.g.
+----------+
| count(*) |
+----------+
Fetched 1 row(s) in 5.20s
To avoid this, the live progress lines need to be fully flushed to stderr
before starting to output the result to stdout. This adds a flush call
in OverwritingStdErrOutputStream::clear() to force this.
Testing:
- Hand tested queries with live progress
- Added test that redirects stdout and stderr to the same file and
verifies that no ANSI up character comes after the query output
Change-Id: Id2e21224253f76b2a04767a57b3ade49ce2c914f
---
M shell/impala_shell/shell_output.py
M tests/shell/test_shell_interactive.py
2 files changed, 25 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/22941/1
--
To view, visit http://gerrit.cloudera.org:8080/22941
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id2e21224253f76b2a04767a57b3ade49ce2c914f
Gerrit-Change-Number: 22941
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
[Impala-ASF-CR] IMPALA-14087: Fix shell live progress output display issue on Python 3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22941 )
Change subject: IMPALA-14087: Fix shell live_progress output display issue on Python 3
......................................................................
Patch Set 2: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/22941
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id2e21224253f76b2a04767a57b3ade49ce2c914f
Gerrit-Change-Number: 22941
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 May 2025 21:16:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14087: Fix shell live progress output display issue on Python 3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22941 )
Change subject: IMPALA-14087: Fix shell live_progress output display issue on Python 3
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19292/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22941
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id2e21224253f76b2a04767a57b3ade49ce2c914f
Gerrit-Change-Number: 22941
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 May 2025 20:56:54 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14087: Fix shell live progress output display issue on Python 3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22941 )
Change subject: IMPALA-14087: Fix shell live_progress output display issue on Python 3
......................................................................
Patch Set 1:
(1 comment)
gerrit-auto-critic failed. You can reproduce it locally using command:
python3 bin/jenkins/critique-gerrit-review.py --dryrun
To run it, you might need a virtual env with Python3's venv installed.
http://gerrit.cloudera.org:8080/#/c/22941/1/tests/shell/test_shell_interactive.py
File tests/shell/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/22941/1/tests/shell/test_shell_interactive.py@324
PS1, Line 324: +
flake8: E226 missing whitespace around arithmetic operator
--
To view, visit http://gerrit.cloudera.org:8080/22941
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id2e21224253f76b2a04767a57b3ade49ce2c914f
Gerrit-Change-Number: 22941
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 May 2025 20:38:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14087: Fix shell live progress output display issue on Python 3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22941 )
Change subject: IMPALA-14087: Fix shell live_progress output display issue on Python 3
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/11927/ DRY_RUN=true
--
To view, visit http://gerrit.cloudera.org:8080/22941
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id2e21224253f76b2a04767a57b3ade49ce2c914f
Gerrit-Change-Number: 22941
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 May 2025 21:55:53 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14087: Fix shell live progress output display issue on Python 3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/22941 )
Change subject: IMPALA-14087: Fix shell live_progress output display issue on Python 3
......................................................................
Patch Set 2: Code-Review+2
Tested myself. This is good. Thanks!
--
To view, visit http://gerrit.cloudera.org:8080/22941
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id2e21224253f76b2a04767a57b3ade49ce2c914f
Gerrit-Change-Number: 22941
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sat, 24 May 2025 04:29:09 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14087: Fix shell live progress output display issue on Python 3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22941 )
Change subject: IMPALA-14087: Fix shell live_progress output display issue on Python 3
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/22941
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id2e21224253f76b2a04767a57b3ade49ce2c914f
Gerrit-Change-Number: 22941
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 23 May 2025 04:14:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14087: Fix shell live progress output display issue on Python 3
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/22941 )
Change subject: IMPALA-14087: Fix shell live_progress output display issue on Python 3
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/19294/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/22941
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id2e21224253f76b2a04767a57b3ade49ce2c914f
Gerrit-Change-Number: 22941
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 May 2025 21:01:25 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-14087: Fix shell live progress output display issue on Python 3
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22941
to look at the new patch set (#2).
Change subject: IMPALA-14087: Fix shell live_progress output display issue on Python 3
......................................................................
IMPALA-14087: Fix shell live_progress output display issue on Python 3
When running the shell in a terminal with live_progress=true, live
progress overwrites its output by using the ANSI up character to
rewrite lines with updated on the query progress. On Python 3,
we found that the updates to clear the live progress were overwriting
the actual output in the terminal. e.g.
+----------+
| count(*) |
+----------+
Fetched 1 row(s) in 5.20s
To avoid this, the live progress lines need to be fully flushed to stderr
before starting to output the result to stdout. This adds a flush call
in OverwritingStdErrOutputStream::clear() to force this.
Testing:
- Hand tested queries with live progress
- Added test that redirects stdout and stderr to the same file and
verifies that no ANSI up character comes after the query output
Change-Id: Id2e21224253f76b2a04767a57b3ade49ce2c914f
---
M shell/impala_shell/shell_output.py
M tests/shell/test_shell_interactive.py
2 files changed, 25 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/22941/2
--
To view, visit http://gerrit.cloudera.org:8080/22941
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Id2e21224253f76b2a04767a57b3ade49ce2c914f
Gerrit-Change-Number: 22941
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-14087: Fix shell live progress output display issue on Python 3
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/22941 )
Change subject: IMPALA-14087: Fix shell live_progress output display issue on Python 3
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/22941/1/tests/shell/test_shell_interactive.py
File tests/shell/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/22941/1/tests/shell/test_shell_interactive.py@324
PS1, Line 324: +
> flake8: E226 missing whitespace around arithmetic operator
Done
--
To view, visit http://gerrit.cloudera.org:8080/22941
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id2e21224253f76b2a04767a57b3ade49ce2c914f
Gerrit-Change-Number: 22941
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Thu, 22 May 2025 20:41:04 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-14087: Fix shell live progress output display issue on Python 3
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/22941 )
Change subject: IMPALA-14087: Fix shell live_progress output display issue on Python 3
......................................................................
IMPALA-14087: Fix shell live_progress output display issue on Python 3
When running the shell in a terminal with live_progress=true, live
progress overwrites its output by using the ANSI up character to
rewrite lines with updated on the query progress. On Python 3,
we found that the updates to clear the live progress were overwriting
the actual output in the terminal. e.g.
+----------+
| count(*) |
+----------+
Fetched 1 row(s) in 5.20s
To avoid this, the live progress lines need to be fully flushed to stderr
before starting to output the result to stdout. This adds a flush call
in OverwritingStdErrOutputStream::clear() to force this.
Testing:
- Hand tested queries with live progress
- Added test that redirects stdout and stderr to the same file and
verifies that no ANSI up character comes after the query output
Change-Id: Id2e21224253f76b2a04767a57b3ade49ce2c914f
Reviewed-on: http://gerrit.cloudera.org:8080/22941
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Riza Suminto <ri...@cloudera.com>
---
M shell/impala_shell/shell_output.py
M tests/shell/test_shell_interactive.py
2 files changed, 25 insertions(+), 0 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Riza Suminto: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/22941
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Id2e21224253f76b2a04767a57b3ade49ce2c914f
Gerrit-Change-Number: 22941
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>