You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Quanlong Huang (Jira)" <ji...@apache.org> on 2022/11/10 05:24:00 UTC
[jira] [Assigned] (IMPALA-11659) Server network util of Statestored is much higher than impala-3.4
[ https://issues.apache.org/jira/browse/IMPALA-11659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Quanlong Huang reassigned IMPALA-11659:
---------------------------------------
Assignee: zhi tang
> Server network util of Statestored is much higher than impala-3.4
> -----------------------------------------------------------------
>
> Key: IMPALA-11659
> URL: https://issues.apache.org/jira/browse/IMPALA-11659
> Project: IMPALA
> Issue Type: Bug
> Components: Perf Investigation
> Affects Versions: Impala 4.1.0
> Reporter: Yuchen Fan
> Assignee: zhi tang
> Priority: Major
> Attachments: image-2022-10-13-19-49-28-456.png, image-2022-10-13-19-57-31-248.png, image-2022-10-13-20-08-24-931.png
>
>
> We found that server network traffic of Statestored is rised immediatrly after upgraded to Impala-4.1. Using 'iftop' shows Statestored has aboud 6MB/s outbound network communication with every Impalad. With more than 250 nodes of Impalad, Statestored server has 10~15Gb/s(1~2GB/s) outbound(60x higher than it before upgrade) and about 90Mb/s(>10MB/s) inbound(10x higher than it before upgrade) network communication, which will occupy about 75% server network util. TCP packet snapshot shows detail of packet is 'Pool' and 'Stat' information in impala-request-queue topic. We found that value of key 'STAT:' contains per host stat of all Impalads. Related function is from https://issues.apache.org/jira/browse/IMPALA-8762. So initial update size of impala-request-queue is more than 4MB when there are more than 250 Impalads. If cluster has more Impalad, Statestored server may hang because of network blocking.
> I think coordinators updating topic of per host statistics is just fine. All Impalads(including coordinators and executors) registering subscriber to topic of impala-request-queue makes Statestored need to broadcast all per node statistics to all Impalads.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org