[jira] [Commented] (IMPALA-13012) Completed queries write fails regularly under heavy load

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-13012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17841385#comment-17841385 ] 

ASF subversion and git services commented on IMPALA-13012:
----------------------------------------------------------

Commit ba32d70891fd68c5c1234ed543b74c51661bf272 in impala's branch refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ba32d7089 ]

IMPALA-13012: Lower default query_log_max_queued

Sets the query_log_max_queued default such that

  query_log_max_queued * num_columns(49) < statement_expression_limit

to avoid triggering e.g.

  AnalysisException: Exceeded the statement expression limit (250000)
  Statement has 370039 expressions.

Also increases statement_expression_limit for insertion to avoid an
error if query_log_max_queued is changed.

Logs time taken to write to the queries table for help with debugging
and adds histogram "impala-server.completed-queries.write-durations".

Fixes InternalServer so it uses 'default_query_options'.

Change-Id: I6535675307d88cb65ba7d908f3c692e0cf3259d7
Reviewed-on: http://gerrit.cloudera.org:8080/21351
Reviewed-by: Michael Smith <mi...@cloudera.com>
Tested-by: Michael Smith <mi...@cloudera.com>
Reviewed-by: Riza Suminto <ri...@cloudera.com>


> Completed queries write fails regularly under heavy load
> --------------------------------------------------------
>
>                 Key: IMPALA-13012
>                 URL: https://issues.apache.org/jira/browse/IMPALA-13012
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 4.4.0
>            Reporter: Michael Smith
>            Assignee: Michael Smith
>            Priority: Critical
>
> Under heavy test load (running EE tests), Impala regularly fails to write completed queries with errors like
> {code}
> W0411 19:11:07.764967 32713 workload-management.cc:435] failed to write completed queries table="sys.impala_query_log" record_count="10001"
> W0411 19:11:07.764983 32713 workload-management.cc:437] AnalysisException: Exceeded the statement expression limit (250000)
> Statement has 370039 expressions.
> {code}
> After a few attempts, it floods logs with an error for each query that could not be written
> {code}
> E0411 19:11:24.646953 32713 workload-management.cc:376] could not write completed query table="sys.impala_query_log" query_id="3142ceb1380b58e6:715b83d900000000"
> {code}
> This seems like poor default behavior. Options for addressing it:
> # Decrease the default for {{query_log_max_queued}}. Inserts are pretty constant at 37 expressions per entry. I'm not sure why that isn't 49, since that's the number of columns we have; maybe some fields are frequently omitted. I would cap {{query_log_max_queued}} to {{statement_expression_limit / number_of_columns ~ 5100}}.
> # Allow workload management to {{set statement_expression_limit}} higher using a similar formula. This may be relatively safe as the expressions are simple.
> # Ideally we would skip expression parsing and construct TExecRequest directly, but that's a much larger effort.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org