[jira] [Commented] (IMPALA-4317) Single Overloaded Impalad Causes Entire Cluster to Hang

["Tim Armstrong (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140802#comment-17140802 ] 

Tim Armstrong commented on IMPALA-4317:
---------------------------------------

This is quite old and it sounds like the problem was likely connected to the old RPC stack and thread counts. There is some additional work to blacklist unhealthy nodes that is relevant.

We can't detect *in general* slowness of a single impalad and blacklist based on that, but I think we would have fixed this particular scenario. So I'll close out this JIRA.

> Single Overloaded Impalad Causes Entire Cluster to Hang
> -------------------------------------------------------
>
>                 Key: IMPALA-4317
>                 URL: https://issues.apache.org/jira/browse/IMPALA-4317
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 2.5.0
>         Environment: Enterprise CDH 5.7.0, Parcels
> impalad version 2.5.0-cdh5.7.0 RELEASE (build ad3f5adabedf56fe6bd9eea39147c067cc552703)
>            Reporter: Scott Wallace
>            Priority: Major
>         Attachments: cached_clients.png, health.png, load.png, queries.png, threads.png, worker23.png
>
>
> Occasionally we experience heavy load on a single impalad host. This leads to the entire cluster to hang and prevents any impala queries from being able to execute.
> Here's what we observe:
> -load increases on a single impalad
> -query throughput across the entire impala cluster drops and we cannot get any queries to execute
> -running threads continues to increase until we restart the impala service
> -in the impalad logs we see errors connecting to the unhealthy host. Example: Couldn't open transport for ux-reporting-engine-worker-23-prod-us-east-1a:22000 (connect() failed: Connection timed out)
> Questions:
> Why does the entire Impala service become unstable due to the health of a single impalad?
> Theoretically, shouldn't the impala statestore prevent the single impalad host from being used and allow queries to be processed by healthy nodes?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org