You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/09/18 23:26:00 UTC
[jira] [Commented] (IMPALA-10051) impala-shell exits with
ValueError with WITH clauses
[ https://issues.apache.org/jira/browse/IMPALA-10051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17198583#comment-17198583 ]
ASF subversion and git services commented on IMPALA-10051:
----------------------------------------------------------
Commit 3ef77566286c0077b89c0b8ce529ea9985018dd6 in impala's branch refs/heads/master from Tamas Mate
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=3ef7756 ]
IMPALA-10051: impala-shell exits with ValueError with WITH clauses
When a query contains WITH clause impala-shell tries to identify whether
it is a DML query or not, so that later it can provide appropriate
result messages. Earlier shlex was used to create tokens and assess the
query type based on that. However shlex can misinterpret some query
strings where whitespace charachters are mixed with quotes, because it
splits the string based on whitespace charachters. In some scenarios
'ValueError: No closing quotation' error can occur.
This change moves the tokenization from shlex to sqlparse.
Testing:
- Added unit test to cover queries that contain mixed whitespaces
and strings
Change-Id: I442d3bc65b90a55c73c847948d5179a8586d71ad
Reviewed-on: http://gerrit.cloudera.org:8080/16389
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> impala-shell exits with ValueError with WITH clauses
> ----------------------------------------------------
>
> Key: IMPALA-10051
> URL: https://issues.apache.org/jira/browse/IMPALA-10051
> Project: IMPALA
> Issue Type: Bug
> Components: Clients
> Affects Versions: Impala 4.0
> Reporter: Tamas Mate
> Assignee: Tamas Mate
> Priority: Major
>
> Some strings can cause shlex to throw an exception in WITH clauses, for example in a regexp_replace. This should be handled more gracefully and correctly.
> Working query (impala-shell forwards the query for analysis):
> {code:java}
> impala-shell.sh -q 'with select regexp_replace(column_name, "[a-zA-Z]", "+ ");'
> {code}
> While same query fails with ValueError when empty spaces are removed from the arguments of the regexp_replace:
> {code:java}
> tmate@tmate-box:~/Projects/Impala$ impala-shell.sh -q 'with select regexp_replace(column_name,"[a-zA-Z]","+ ");'
> Starting Impala Shell with no authentication using Python 2.7.16
> Warning: live_progress only applies to interactive shell sessions, and is being skipped for now.
> Opened TCP connection to localhost:21000
> Connected to localhost:21000
> Server version: impalad version 4.0.0-SNAPSHOT DEBUG (build b29cb4ca82a4f05ea7dc0eadc330a64fbe685ef0)
> Traceback (most recent call last):
> File "/home/tmate/Projects/Impala/shell/impala_shell.py", line 1973, in <module>
> impala_shell_main()
> File "/home/tmate/Projects/Impala/shell/impala_shell.py", line 1927, in impala_shell_main
> if execute_queries_non_interactive_mode(options, query_options):
> File "/home/tmate/Projects/Impala/shell/impala_shell.py", line 1731, in execute_queries_non_interactive_mode
> shell.execute_query_list(queries))
> File "/home/tmate/Projects/Impala/shell/impala_shell.py", line 1564, in execute_query_list
> if self.onecmd(q) is CmdStatus.ERROR:
> File "/home/tmate/Projects/Impala/shell/impala_shell.py", line 675, in onecmd
> return func(arg)
> File "/home/tmate/Projects/Impala/shell/impala_shell.py", line 1276, in do_with
> tokens = shlex.split(strip_comments(query.lstrip()), posix=False)
> File "/home/tmate/Projects/Impala/toolchain/toolchain-packages-gcc7.5.0/python-2.7.16/lib/python2.7/shlex.py", line 279, in split
> return list(lex)
> File "/home/tmate/Projects/Impala/toolchain/toolchain-packages-gcc7.5.0/python-2.7.16/lib/python2.7/shlex.py", line 269, in next
> token = self.get_token()
> File "/home/tmate/Projects/Impala/toolchain/toolchain-packages-gcc7.5.0/python-2.7.16/lib/python2.7/shlex.py", line 96, in get_token
> raw = self.read_token()
> File "/home/tmate/Projects/Impala/toolchain/toolchain-packages-gcc7.5.0/python-2.7.16/lib/python2.7/shlex.py", line 172, in read_token
> raise ValueError, "No closing quotation"
> ValueError: No closing quotation
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org