[Impala-ASF-CR] IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

["Andrew Sherman (Code Review)" <ge...@cloudera.org>]

Andrew Sherman has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/13307 )

Change subject: IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.
......................................................................

IMPALA-8536: Add Scalable Pool Configuration to Admission Controller.

Add 3 configuration parameters to Admission Controller that scale with
the number of hosts in the resource pool. These parameters are specified
to the Impalad through the -llama_site_path flag which points to a Llama
XML configuration file.

The new configuration parameters are:
+ Max Running Queries Multiple - this floating point number is
  multiplied by the current total number of running Impalads at runtime
  to give the maximum number of concurrently running queries allowed in
  the pool. This calculation is rounded up to the nearest integer so the
  result will always be at least one as long as the parameter is
  non-zero.
+ Max Queued Queries Multiple - this floating point number is multiplied
  by the current total number of running Impalads at runtime to give the
  maximum number of queries that can be queued in the pool. This
  calculation is rounded up to the nearest integer so the result will
  always be at least one as long as the parameter is non-zero.
+ Max Memory Multiple - the Multiple of the current total number of
  running Impalads which gives the maximum memory available across the
  cluster for the pool.

The new parameters are exposed through the webui.

At various points in the code Admission Controller looks at the Pool
Config objects to find non-scalable parameters such as the max number of
queries that can run in the pool. These access have been encapsulated in
functions that return the scalable version of the configuration value if
the new scalable parameters are being used. Diagnostic messages are
enhanced to show the origin of the encapsulated parameters.

TESTING

All end-to-end tests are running clean with ASAN.

The unit test admission-controller-test.cc has been expanded to test the
newly added code.

Added an end-to-end test that adds and removes Impalads from a
minicluster.

Change-Id: If47508728124076f3b9200c27cffc989f7a4f188
---
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M common/thrift/ImpalaInternalService.thrift
M common/thrift/generate_metrics.py
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/util/RequestPoolService.java
M fe/src/test/java/org/apache/impala/util/TestRequestPoolService.java
M fe/src/test/resources/fair-scheduler-test.xml
M fe/src/test/resources/fair-scheduler-test2.xml
M fe/src/test/resources/llama-site-test.xml
M fe/src/test/resources/llama-site-test2.xml
M testdata/workloads/functional-query/queries/QueryTest/admission-max-min-mem-limits.test
M testdata/workloads/functional-query/queries/QueryTest/admission-reject-mem-estimate.test
M testdata/workloads/functional-query/queries/QueryTest/admission-reject-min-reservation.test
M tests/common/impala_cluster.py
M tests/custom_cluster/test_admission_controller.py
M tests/webserver/test_web_pages.py
M www/admission_controller.tmpl
19 files changed, 1,107 insertions(+), 243 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/13307/3
-- 
To view, visit http://gerrit.cloudera.org:8080/13307
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If47508728124076f3b9200c27cffc989f7a4f188
Gerrit-Change-Number: 13307
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Lars Volker <lv...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>