[Impala-ASF-CR] IMPALA-8803: Coordinator should release admitted memory per-backend

["Sahil Takiar (Code Review)" <ge...@cloudera.org>]

Hello Michael Ho, Tim Armstrong, Bikramjeet Vig, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/14104

to look at the new patch set (#7).

Change subject: IMPALA-8803: Coordinator should release admitted memory per-backend
......................................................................

IMPALA-8803: Coordinator should release admitted memory per-backend

Changes the Coordinator to release admitted memory when each Backend
completes, rather than waiting for the entire query to complete before
releasing admitted memory. When the Coordinator detects that a Backend
has completed (via ControlService::ReportExecStatus) it updates the
state of the Backend in Coordinator::BackendResourceState.
BackendResourceState tracks the state of the admitted resources for
each Backend and decides when the resources for a group of Backend
states should be released. BackendResourceState defines a state machine
to help coordinate the state of the admitted memory for each Backend.
It guarantees that by the time the query is shutdown, all Backends
release their admitted memory.

BackendResourceState implements three rules to control the rate at
which the Coordinator releases admitted memory from the
AdmissionController:
* Resources are released at most once every 1 second, this prevents
short lived queries from causing high load on the admission controller
* Resources are released at most O(log(num_backends)) times; the
BackendResourceStates can release multiple BackendStates from the
AdmissionController at a time
* All pending resources are released if the only remaining Backend is
the Coordinator Backend; this is useful for result spooling where all
Backends may complete, except for the Coordinator Backend

Exposes the following hidden startup flags to help tune the heuristics
above:

--batched_release_decay_factor
* Defaults to 2
* Controls the base value for the O(log(num_backends)) bound when
batching the release of Backends.

--release_backend_states_delay_ms
* Defaults to 1000 milliseconds
* Controls how often Backends can release their resources.

Testing:
* Ran core tests
* Added new tests to test_result_spooling.py and
test_admission_controller.py

Change-Id: I88bb11e0ede7574568020e0277dd8ac8d2586dc9
---
M be/src/runtime/CMakeLists.txt
A be/src/runtime/coordinator-backend-resource-state.cc
A be/src/runtime/coordinator-backend-state-test.cc
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator-backend-state.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/scheduling/admission-controller-test.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M tests/custom_cluster/test_admission_controller.py
M tests/custom_cluster/test_auto_scaling.py
M tests/custom_cluster/test_executor_groups.py
A tests/custom_cluster/test_result_spooling.py
M tests/util/auto_scaler.py
A tests/util/web_pages_util.py
16 files changed, 1,027 insertions(+), 94 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/14104/7
-- 
To view, visit http://gerrit.cloudera.org:8080/14104
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I88bb11e0ede7574568020e0277dd8ac8d2586dc9
Gerrit-Change-Number: 14104
Gerrit-PatchSet: 7
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>