[jira] [Commented] (IMPALA-14445) Improve error message for RPC backpressure and other KRPC error messages

["Joe McDonnell (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18021904#comment-18021904 ] 

Joe McDonnell commented on IMPALA-14445:
----------------------------------------

The executor can be rejecting the message due to hitting a memory limit, so we should make clear which memory limit it is hitting. This may be completely unrelated to the sizes of the control service queue / data stream service queue, as it may be hitting the process memory limit.

> Improve error message for RPC backpressure and other KRPC error messages
> ------------------------------------------------------------------------
>
>                 Key: IMPALA-14445
>                 URL: https://issues.apache.org/jira/browse/IMPALA-14445
>             Project: IMPALA
>          Issue Type: Task
>          Components: Distributed Exec
>    Affects Versions: Impala 5.0.0
>            Reporter: Joe McDonnell
>            Priority: Critical
>
> On a cluster, we saw queries failing with errors like this:
> {noformat}
> Exec() rpc failed: Remote error: Service unavailable: ExecQueryFInstances request on impala.ControlService from 1.2.3.4:12345 dropped due to backpressure. The service queue contains 0 items out of a maximum of 2147483647; memory consumption is 0.{noformat}
> In this case, the executor is rejecting a message from the coordinator (1.2.3.4:12345 is the IP:port of a coordinator) and the query is failing due to that. The error message doesn't make it clear that the executor is the one rejecting the message, so this can lead to focusing on the coordinator rather than tracking down the executor. Also, the message should include the executor hostname to make it easy to know where to look.
> The executor black-listing / query retry logic constructs a better message for the retryable status, but we don't use that for backpressure.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org