[Impala-ASF-CR] IMPALA-7486: Add specialized estimation scheme for dedicated coordinators

["Bikramjeet Vig (Code Review)" <ge...@cloudera.org>]

Hello Andrew Sherman, Lars Volker, Tim Armstrong, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/13740

to look at the new patch set (#12).

Change subject: IMPALA-7486: Add specialized estimation scheme for dedicated coordinators
......................................................................

IMPALA-7486: Add specialized estimation scheme for dedicated coordinators

This patch computes two memory estimates in the frontend:
an estimate for any host that is an executor (including
a combined coordinator and executor) and an estimate
for a dedicated coordinator. This is computed regardless
of whether it is a dedicated coordinator or not.

Admission control then, in the case when the coordinator
is dedicated, uses the coordinator memory estimate for
the coordinator node and the executor memory estimate
for all other nodes.

Other highlights:
 - if MEM_LIMIT query option is set, it is applied to all backends
 - the min_query_mem_limit pool config is not enforced on the
   dedicated coordinator estimates unless MEM_LIMIT query option is set
 - the lower cap on estimates and the admission checks based on the
   min mem limit required for reservation are applied separately on
   coordinator's and executors' mem requirements.
 - Added a hidden startup option 'use_dedicated_coordinator_estimates'
   which if set to false, reverts to previous estimation behavior.

Testing:
- Added unit test for admission/rejection in dedicated
  coordinator clusters.
- Added end to end tests.

Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21
---
M be/src/runtime/coordinator.cc
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/query-schedule.cc
M be/src/scheduling/query-schedule.h
M be/src/scheduling/scheduler.cc
M be/src/service/client-request-state.cc
M be/src/service/impala-http-handler.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/planner/PlanFragment.java
M fe/src/main/java/org/apache/impala/planner/Planner.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M tests/custom_cluster/test_admission_controller.py
M www/admission_controller.tmpl
18 files changed, 674 insertions(+), 108 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/13740/12
-- 
To view, visit http://gerrit.cloudera.org:8080/13740
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2b94e7293b91dec8a18491079c34923eadd94b21
Gerrit-Change-Number: 13740
Gerrit-PatchSet: 12
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>