[Impala-ASF-CR] WIP: IMPALA-8998: admission control accounting for mt dop

["Tim Armstrong (Code Review)" <ge...@cloudera.org>]

Hello Andrew Sherman, Lars Volker, Abhishek Rawat, Bikramjeet Vig, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/14357

to look at the new patch set (#5).

Change subject: WIP: IMPALA-8998: admission control accounting for mt_dop
......................................................................

WIP: IMPALA-8998: admission control accounting for mt_dop

This integrates mt_dop with the "slots" mechanism that's used
for non-default executor groups.

The idea is simple - the degree of parallelism on a backend
determines the number of slots consumed. The effective
degree of parallelism is used, not the raw mt_dop setting.
E.g. if the query only has a single input split and executes
only a single fragment instance on a host, we don't want
to count the full mt_dop value for admission control.

The admission control logic is extended to take this into
account. We also add an immediate rejection code path
since it is now possible for queries to not be admittable
based on the # of available slots.

We only factor in the "width" of the plan - i.e. the number
of instances of fragments. We don't account for the number
of distinct fragments, since they may not actually execute
in parallel with each other because of dependencies.

This number is added to the per-host profile as the
"AdmissionSlots" counter.

Testing:
Added unit tests for rejection and queue/admit checks.

Added end-to-end tests:
* test_admission_slots in test_mt_dop.py that checks the
  admission slot calculation via the profile.
* TODO: End-to-end admission test

Added checks to test_verify_metrics (which runs after
end-to-end tests) to ensure that the per-backend
slots in use goes to 0 when the cluster is quiesced.

Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5
---
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/query-schedule.h
M be/src/scheduling/scheduler.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M common/thrift/StatestoreService.thrift
A testdata/workloads/functional-query/queries/QueryTest/mt-dop-parquet-admission-slots.test
M tests/custom_cluster/test_executor_groups.py
M tests/query_test/test_mt_dop.py
M tests/verifiers/test_verify_metrics.py
M www/backends.tmpl
16 files changed, 311 insertions(+), 57 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/14357/5
-- 
To view, visit http://gerrit.cloudera.org:8080/14357
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7b6b6262ef238df26b491352656a26e4163e46e5
Gerrit-Change-Number: 14357
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>
Gerrit-Reviewer: Abhishek Rawat <ar...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>