[Impala-ASF-CR] WIP IMPALA-8484: Run queries on disjoint executor groups

["Lars Volker (Code Review)" <ge...@cloudera.org>]

Hello Andrew Sherman, Tim Armstrong, Bikramjeet Vig, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/13550

to look at the new patch set (#7).

Change subject: WIP IMPALA-8484: Run queries on disjoint executor groups
......................................................................

WIP IMPALA-8484: Run queries on disjoint executor groups

This change adds support running queries of a single admission control
pool on one of several, disjoint sets of executors called 'executor
groups'.

This change also re-schedules queries before admitting them if the
admission controller detects that the cluster membership has changed.

TODO: Elaborate on change and design

Known limitations:
- When using executor groups, only a single coordinator and a single AC
pool are supported.

Open questions and issues:
- Executor groups can become unhealthy when one or more of their
  executors fail. We should build support to heal or decommission
  unhealthy groups in a subsequent change.

Change-Id: I8a1d0900f2a82bd2fc0a906cc094e442cffa189b
---
M be/src/runtime/exec-env.cc
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/cluster-membership-mgr-test.cc
M be/src/scheduling/cluster-membership-mgr.cc
M be/src/scheduling/cluster-membership-mgr.h
M be/src/scheduling/cluster-membership-test-util.cc
M be/src/scheduling/executor-group-test.cc
M be/src/scheduling/executor-group.cc
M be/src/scheduling/executor-group.h
M be/src/scheduling/query-schedule.h
M be/src/scheduling/scheduler-test-util.cc
M be/src/scheduling/scheduler.cc
M be/src/scheduling/scheduler.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/service/query-options.cc
M be/src/service/query-options.h
M be/src/util/runtime-profile.h
M bin/start-impala-cluster.py
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/StatestoreService.thrift
M tests/common/impala_cluster.py
M tests/common/impala_service.py
M tests/common/resource_pool_config.py
M tests/custom_cluster/test_admission_controller.py
A tests/custom_cluster/test_auto_scaling.py
A tests/util/auto_scaler.py
A tests/util/concurrent_workload.py
M www/backends.tmpl
35 files changed, 1,522 insertions(+), 432 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/13550/7
-- 
To view, visit http://gerrit.cloudera.org:8080/13550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8a1d0900f2a82bd2fc0a906cc094e442cffa189b
Gerrit-Change-Number: 13550
Gerrit-PatchSet: 7
Gerrit-Owner: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>