[jira] [Commented] (IMPALA-8403) Possible thread leak in impalad

["Michael Ho (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-8403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898339#comment-16898339 ] 

Michael Ho commented on IMPALA-8403:
------------------------------------

This is most likely due to Thrift connections accumulated in the client cache of various other Impalads over time. These connections aren't being closed. Each connection on the client side will correspond to a thread on the server side and there is no limit enforced on the number of these Thrift connection threads for "backend" service. Overtime, the number of backend threads grow.

We have converted quite a number of backend service to KRPC, including the biggest offenders (e.g. {{TransmitData()}}, {{ReportExecStatus()}}). Once IMPALA-7984 is fixed, we can remove the Thrift server for backend services.

> Possible thread leak in impalad
> -------------------------------
>
>                 Key: IMPALA-8403
>                 URL: https://issues.apache.org/jira/browse/IMPALA-8403
>             Project: IMPALA
>          Issue Type: Bug
>    Affects Versions: Impala 2.12.0
>            Reporter: Quanlong Huang
>            Priority: Major
>         Attachments: image-2019-04-10-11-15-11-321.png, reproIMPALA-8403.tgz
>
>
> The metric of thread-manager.running-threads got from http://${impalad_host}:25000/metrics?json shows that the number of running threads keeps increasing. (See the snapshot) This phenomenon is most noticeable in coordinators.
> Maybe a counter bug or threads leak.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org