You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2018/08/21 21:39:00 UTC
[jira] [Commented] (IMPALA-7449) TotalNetworkThroughput in
KrpcDataStreamSender is broken
[ https://issues.apache.org/jira/browse/IMPALA-7449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16588040#comment-16588040 ]
ASF subversion and git services commented on IMPALA-7449:
---------------------------------------------------------
Commit 2a60655b09afaa76d3bf2120c7043eb0b22eefcf in impala's branch refs/heads/master from Michael Ho
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=2a60655 ]
IMPALA-7449: Fix network throughput calculation of DataStreamSender
Currently, the network throughput presented in the query profile
for DataStreamSender is computed by dividing the total bytes sent
by the total network time which is the sum of observed network time
of all individual RPCs. This is wrong in general and may only make
sense if the network throughput is fixed. In addition, RPCs are
asynchronous and they overlap with each other. So, dividing the
total byte sent by network throughput may result in time which exceeds
the wall clock time, making it impossible to interpret.
This change fixes the problem by measuring the network throughput
of each individual RPC and uses a summary counter to track avg/min/max
of network throughputs instead.
Change-Id: I344ac76c0a1a49b4da3d37d2c547f3d5051ebe24
Reviewed-on: http://gerrit.cloudera.org:8080/11241
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> TotalNetworkThroughput in KrpcDataStreamSender is broken
> --------------------------------------------------------
>
> Key: IMPALA-7449
> URL: https://issues.apache.org/jira/browse/IMPALA-7449
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
> Affects Versions: Impala 3.0, Impala 2.12.0
> Reporter: Michael Ho
> Assignee: Michael Ho
> Priority: Critical
>
> The network throughput computation fails to take into account of the fact that multiple RPCs can be happening in parallel. Currently, the throughput is computed by (total bytes sent / total network time). The total network time is the aggregate of the network time observed of each RPC. This seems hard to understand (or wrong?) when there are drastically different throughput when sending to different hosts. It may be slightly easier to understand if we switch to measuring the observed network throughput of each individual RPC and use a summary counter or a histogram to record the throughput.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org