[jira] [Commented] (IMPALA-7259) impala-shell is weirdly slow with some large queries

["Nghia Le (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-7259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541299#comment-16541299 ] 

Nghia Le commented on IMPALA-7259:
----------------------------------

I have done some experiments using cProfile with your file and found out that there are 2 main functions which made the shell slow with large queries: parse_query_text() and strip_leading_comment() in impala_shell.py

For parse_query_text():

    The problem is in this statement

 

    ``` if query_list and not sqlparse.format(query_list[-1], strip_comments=True).strip("\n"):

         query_list.pop()

 

    Since your last query is a really long string, the sqlparse.format needs a lot of time to check whether that string is a comment or not.

 

For strip_leading_comment():

    The problem also lies at string processing part.

> impala-shell is weirdly slow with some large queries
> ----------------------------------------------------
>
>                 Key: IMPALA-7259
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7259
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Clients
>    Affects Versions: Impala 3.1.0
>            Reporter: Tim Armstrong
>            Assignee: Nghia Le
>            Priority: Major
>         Attachments: wide-parquet-agg.sql
>
>
> impala-shell is very slow at processing some large queries - it takes over a minute to actually submit the query. I've attached an example.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org