You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Quanlong Huang (Code Review)" <ge...@cloudera.org> on 2019/06/13 08:45:52 UTC
[Impala-ASF-CR](2.x) IMPALA-7259: Improve Impala shell performance
Hello Impala Public Jenkins,
I'd like you to do a code review. Please visit
http://gerrit.cloudera.org:8080/13614
to review the following change.
Change subject: IMPALA-7259: Improve Impala shell performance
......................................................................
IMPALA-7259: Improve Impala shell performance
This patch fixes the slow performance in Impala shell, especially for
large queries by replacing all calls to sqlparse.format(sql_string,
strip_comments=True) with the custom implementation of strip comments
that does not use grouping. The code to strip leading comments was also
refactored to not use grouping.
* Benchmark running a query with 12K columns *
Before the patch:
$ time impala-shell.sh -f large.sql --quiet
real 2m4.154s
user 2m0.536s
sys 0m0.088s
After the patch:
$ time impala-shell.sh -f large.sql --quiet
real 0m3.885s
user 0m1.516s
sys 0m0.048s
Testing:
- Added a new test to test the Impala shell performance
- Ran all shell tests on Python 2.6 and Python 2.7
Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81
Reviewed-on: http://gerrit.cloudera.org:8080/10939
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M shell/impala_shell.py
M tests/shell/test_shell_commandline.py
2 files changed, 70 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/13614/1
--
To view, visit http://gerrit.cloudera.org:8080/13614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81
Gerrit-Change-Number: 13614
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR](2.x) IMPALA-7259: Improve Impala shell performance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13614 )
Change subject: IMPALA-7259: Improve Impala shell performance
......................................................................
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13614/1/shell/impala_shell.py
File shell/impala_shell.py:
http://gerrit.cloudera.org:8080/#/c/13614/1/shell/impala_shell.py@62
PS1, Line 62: def strip_comments(sql):
flake8: E302 expected 2 blank lines, found 1
http://gerrit.cloudera.org:8080/#/c/13614/1/shell/impala_shell.py@1373
PS1, Line 1373: t
flake8: E126 continuation line over-indented for hanging indent
http://gerrit.cloudera.org:8080/#/c/13614/1/shell/impala_shell.py@1378
PS1, Line 1378: t
flake8: E126 continuation line over-indented for hanging indent
http://gerrit.cloudera.org:8080/#/c/13614/1/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:
http://gerrit.cloudera.org:8080/#/c/13614/1/tests/shell/test_shell_commandline.py@689
PS1, Line 689: '
flake8: E128 continuation line under-indented for visual indent
--
To view, visit http://gerrit.cloudera.org:8080/13614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81
Gerrit-Change-Number: 13614
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Jun 2019 08:48:54 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR](2.x) IMPALA-7259: Improve Impala shell performance
Posted by "Quanlong Huang (Code Review)" <ge...@cloudera.org>.
Quanlong Huang has abandoned this change. ( http://gerrit.cloudera.org:8080/13614 )
Change subject: IMPALA-7259: Improve Impala shell performance
......................................................................
Abandoned
Sorry that I accidentally submit these patches. There're clean for cherry-pick so don't need for review.
--
To view, visit http://gerrit.cloudera.org:8080/13614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: abandon
Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81
Gerrit-Change-Number: 13614
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR](2.x) IMPALA-7259: Improve Impala shell performance
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13614 )
Change subject: IMPALA-7259: Improve Impala shell performance
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/3599/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/13614
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81
Gerrit-Change-Number: 13614
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang <hu...@gmail.com>
Gerrit-Reviewer: Fredy Wijaya <fw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Thu, 13 Jun 2019 09:33:47 +0000
Gerrit-HasComments: No