[Impala-ASF-CR] IMPALA-8571: harden QueryEventHook execution

["radford nguyen (Code Review)" <ge...@cloudera.org>]

Hello Bharath Vissapragada, Andrew Sherman, Fredy Wijaya, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/13748

to look at the new patch set (#24).

Change subject: IMPALA-8571: harden QueryEventHook execution
......................................................................

IMPALA-8571: harden QueryEventHook execution

This commit hardens guarantees around QueryEventHook execution by
adding the following features:

*hook execution*

Query event hooks are still executed in a thread-pool, but you can
now configure the type of (Java) threads used to execute them via
the newly-added backend flag `query_event_hook_use_daemon_threads`,
which takes a true/false value.

See Java Thread.setDaemon(boolean) for what a "daemon thread" entails.

*hook timeout/cancellation*

A timeout for hook execution can be configured through the backend
flag `query_event_hook_timeout_s`, which specified a timeout value
in seconds. If a hook has not completed execution within this timeout
(measured from hook submission, not execution) then the hook task
will be cancelled in order to free up resources.

*hook rejection*

The hook execution engine now has a fixed-capacity work queue whose
capacity can be configured through the backend flag
`query_event_hook_queue_capacity`. This queue is used to store
hook tasks that are submitted when there are no free threads
available for hook execution. All hook tasks submitted when the
queue is at capacity will be rejected and logged without affecting
the result of the query.

*hook performance metrics*

Hook performance metrics are captured by the frontend, but not yet
published to the backend and are therefore not user-visible. See
IMPALA-8914 for implementation of this feature.

Testing:

- added unit tests for new features
- re-ran existing E2E tests

Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
---
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java
M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
12 files changed, 968 insertions(+), 111 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/24
-- 
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 24
Gerrit-Owner: radford nguyen <ra...@gmail.com>
Gerrit-Reviewer: Andrew Sherman <as...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Fredy Wijaya <fr...@apache.org>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: radford nguyen <ra...@gmail.com>