[jira] [Updated] (HIVE-9423) HiveServer2: Implement some admission control mechanism for graceful degradation when resources are exhausted

["Peter Vary (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HIVE-9423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Peter Vary updated HIVE-9423:
-----------------------------
    Attachment: HIVE-9423.patch

THRIFT-2046 handles connection overflow with closing the connection without writing any data into it. See:
{code}
            if (t instanceof RejectedExecutionException) {
              retryCount++;
              try {
                if (remainTimeInMillis > 0) {
                  //do a truncated 20 binary exponential backoff sleep
[..]
                } else {
                  client.close();
                  wp = null;
                  LOGGER.warn("Task has been rejected by ExecutorService " + retryCount
                      + " times till timedout, reason: " + t);
                  break;
                }
              }
{code}

On the client side this generates a TTransportException in TIOStreamTransport.java with specific types, which helps us to differentiate between the cases.

So in the proposed solution we could print different error message, when the connection pool is exhausted, and when the connection is not available, etc.

What do you think about the proposed solution [~aihuaxu], [~ctang.ma], [~ngangam]?

> HiveServer2: Implement some admission control mechanism for graceful degradation when resources are exhausted
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-9423
>                 URL: https://issues.apache.org/jira/browse/HIVE-9423
>             Project: Hive
>          Issue Type: Bug
>          Components: HiveServer2
>    Affects Versions: 0.12.0, 0.13.0, 0.14.0, 0.15.0
>            Reporter: Vaibhav Gumashta
>            Assignee: Peter Vary
>         Attachments: HIVE-9423.patch
>
>
> An example of where it is needed: it has been reported that when # of client connections is greater than   {{hive.server2.thrift.max.worker.threads}}, HiveServer2 stops accepting new connections and ends up having to be restarted. This should be handled more gracefully by the server and the JDBC driver, so that the end user gets aware of the problem and can take appropriate steps (either close existing connections or bump of the config value or use multiple server instances with dynamic service discovery enabled). Similarly, we should also review the behaviour of background thread pool to have a well defined behavior on the the pool getting exhausted. 
> Ideally implementing some form of general admission control will be a better solution, so that we do not accept new work unless sufficient resources are available and display graceful degradation under overload.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)