[jira] [Resolved] (IMPALA-13164) Add metrics for RPC reactor and deserialization threads

["Michael Smith (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-13164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Smith resolved IMPALA-13164.
------------------------------------
    Resolution: Fixed

> Add metrics for RPC reactor and deserialization threads
> -------------------------------------------------------
>
>                 Key: IMPALA-13164
>                 URL: https://issues.apache.org/jira/browse/IMPALA-13164
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend, Distributed Exec
>    Affects Versions: Impala 4.4.0
>            Reporter: Michael Smith
>            Assignee: Michael Smith
>            Priority: Critical
>             Fix For: Impala 5.0.0
>
>
> When Impala starts to build up a queue in an RPC service - usually DataStreamService - they become a bottleneck that slows down all exchanges going through that node. To identify what's happening we generally need to collect pstacks to identify why the queue is backed up.
> If all KrpcDataStreamMgr deserialization threads are in use, it can help to increase the number of threads available via {{{}datastream_service_num_deserialization_threads{}}}. This is the most common bottleneck on larger machines; DataStream packets can be large and take time to deserialize.
> Less frequently the number of reactor threads in RpcMgr sending requests and receiving responses (so handling the actual network I/O) might be full; this is much less common because {{num_reactor_threads}} defaults to the number of CPU cores.
> ControlService uses a single thread pool for network I/O and deserialization because the packets tend to be much smaller; this is controlled by {{{}control_service_num_svc_threads{}}}, which defaults to the number of CPU cores as well.
> It would help to have metrics for RPC reactor and deserialization threads to tell when they're fully loaded and building up a queue; this could identify when we need to increase {{datastream_service_num_deserialization_threads}} or {{{}num_reactor_threads{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)