[Impala-ASF-CR] IMPALA-9752: aggregate profile stats on executor

["Tim Armstrong (Code Review)" <ge...@cloudera.org>]

Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/16050

to look at the new patch set (#4).

Change subject: IMPALA-9752: aggregate profile stats on executor
......................................................................

IMPALA-9752: aggregate profile stats on executor

Before this change the coordinator depended on getting the full
fragment instance profiles from executors to pull out various
things. This change removes that dependency by pulling out the
information on the executor, and including it in the status
report protobuf. This should slightly reduce the amount of work
done on the coordinator, but more importantly, makes it easier
to switch to sending aggregated profiles from executor to
coordinator, because the coordinator no longer depends on
receiving individual instance profiles.

Per-host peak memory is included directly in the status report.

Per-backend stats - where the per-backend total is needed -
are aggregated on the executor with the result included in the
status report. These counters are: BytesRead, ScanRangesComplete,
TotalBytesSent, TotalThreads{User,Sys}Time.

One subtlety to keep in mind that status reports don't include
stats for instances where the final update was sent in a previous
status report. So the executor needs to ensure that stats for
finished fragment instances are included in updates. This is
achieved by caching those values in FragmentInstanceState.

The stats used in the exec summary were previously also plucked
out of the profile on the coordinator. This change moves the work
to the executor, and includes the per-node stats in the status
report.

I did a little cleanup of the profile counter declarations,
making sure they were consistently inside the impala namespace
in the files that I touched.

Testing:
Ran core tests.

Manually checked exec summary, query profiles and backends
page for a running query.

Change-Id: Ia2aca354d803ce78a798a1a64f9f98353b813e4a
---
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/kudu-scan-node-base.cc
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/fragment-instance-state.cc
M be/src/runtime/fragment-instance-state.h
M be/src/runtime/query-state.cc
M common/protobuf/control_service.proto
9 files changed, 213 insertions(+), 134 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/16050/4
-- 
To view, visit http://gerrit.cloudera.org:8080/16050
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia2aca354d803ce78a798a1a64f9f98353b813e4a
Gerrit-Change-Number: 16050
Gerrit-PatchSet: 4
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>