You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/06/12 04:09:00 UTC
[jira] [Commented] (IMPALA-9752) Move instance profile operations
to executors
[ https://issues.apache.org/jira/browse/IMPALA-9752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133903#comment-17133903 ]
ASF subversion and git services commented on IMPALA-9752:
---------------------------------------------------------
Commit 67b4764853ed08b1861bcc854adc047ae9420676 in impala's branch refs/heads/master from Tim Armstrong
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=67b4764 ]
IMPALA-9752: aggregate profile stats on executor
Before this change the coordinator depended on getting the full
fragment instance profiles from executors to pull out various
things. This change removes that dependency by pulling out the
information on the executor, and including it in the status
report protobuf. This should slightly reduce the amount of work
done on the coordinator, but more importantly, makes it easier
to switch to sending aggregated profiles from executor to
coordinator, because the coordinator no longer depends on
receiving individual instance profiles.
Per-host peak memory is included directly in the status report.
Per-backend stats - where the per-backend total is needed -
are aggregated on the executor with the result included in the
status report. These counters are: BytesRead, ScanRangesComplete,
TotalBytesSent, TotalThreads{User,Sys}Time.
One subtlety to keep in mind that status reports don't include
stats for instances where the final update was sent in a previous
status report. So the executor needs to ensure that stats for
finished fragment instances are included in updates. This is
achieved by caching those values in FragmentInstanceState.
The stats used in the exec summary were previously also plucked
out of the profile on the coordinator. This change moves the work
to the executor, and includes the per-node stats in the status
report.
I did a little cleanup of the profile counter declarations,
making sure they were consistently inside the impala namespace
in the files that I touched.
Testing:
Ran core tests.
Manually checked exec summary, query profiles and backends
page for a running query.
Change-Id: Ia2aca354d803ce78a798a1a64f9f98353b813e4a
Reviewed-on: http://gerrit.cloudera.org:8080/16050
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Move instance profile operations to executors
> ---------------------------------------------
>
> Key: IMPALA-9752
> URL: https://issues.apache.org/jira/browse/IMPALA-9752
> Project: IMPALA
> Issue Type: Sub-task
> Components: Distributed Exec
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
> Fix For: Impala 4.0
>
>
> IMPALA-9382 will be simplified if we can move the logic in CoordinatorBackendState::InstanceStats that adds or plucks out values from instance profiles to the executor side, so that we don't need separate code paths for aggregated and non-aggregated profiles sent to the coordinator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org