[Impala-ASF-CR] [Preview]IMPALA-5216: Make admission control queuing async

["Bikramjeet Vig (Code Review)" <ge...@cloudera.org>]

Hello Tim Armstrong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/10060

to look at the new patch set (#2).

Change subject: [Preview]IMPALA-5216: Make admission control queuing async
......................................................................

[Preview]IMPALA-5216: Make admission control queuing async

Implement asynchronous admission control queuing. This is achieved by
running the admission control code-path in a separate thread. Major
changes include: propagating cancellation to the admission control
thread and dequeuing thread, and ensuring that the state expected by the
impala-server from the ClientRequest objects across its life cycle is
the same before and after this patch.

Testing:
- Added a deterministic end to end test
- Ran multiple stress tests successfully with a cancellation probability
of 60% and with different values for the following parameters:
max_requests, queue_wait_timeout_ms. Ensured that the impalad was in a
valid state afterwards (no orphan fragments or wrong metrics).

TODO: add stress tests.
TODO: add test to verify the codepath where dequeue thread
      comes across a cancelled query.
TODO: change terminology of "in_flight_query" to "submitted_queries"
      need to identify all refernces of this terminology, eg. in
      comments, tests, variable names, etc.
TODO: Currently the impala debug page shows the query profile
      but other tabs like 'Plan', 'Query' etc dont work. Fix that.

Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf
---
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/util/promise-test.cc
M be/src/util/promise.h
M common/thrift/ImpalaService.thrift
M tests/custom_cluster/test_admission_controller.py
13 files changed, 383 insertions(+), 134 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/60/10060/2
-- 
To view, visit http://gerrit.cloudera.org:8080/10060
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I989cf5b259afb8f5bc5c35590c94961c81ce88bf
Gerrit-Change-Number: 10060
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>