[jira] [Commented] (IMPALA-7800) Reject or timeout new incoming connections once --fe_service_threads limit is reached

["Zoram Thanga (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-7800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739889#comment-16739889 ] 

Zoram Thanga commented on IMPALA-7800:
--------------------------------------

The current implementation of TAccepQueueServer inherits directly from TServer, and essentially implements a blocking server, in the sense that once the accept queue is full new connection requests are made to block. Also, accepted connections also block in SetupConnection if maxTasks_ threads (controlled by --fe_service_threads) are already in use. Since the accepted connection queue is very big (10,000 by default) it's unlikely that we would bump against this limit.

To enable us to reject OpenSession() requests on breaching the fe_service_threads limit, we have a couple of options:

# Use the existing TAcceptQueueServer with a new (boolean) blocking/non-blocking flag that we can check in TAcceptQueueServer::SetupConnection(), and close/reject the connection once we're at the threads limit.
# Implement a new, "non-blocking" TNonblockingServer which is basically a non-blocking variant of TAcceptQueueServer, which can be instantiated instead in ImpalaServer::Start() if a non-blocking thrift server is desired by the startup flag (--use_nonblocking_thrift_server=true, or something like that).

The second option might be an overkill though. I've tried the first option and it seems to work well enough in my limited testing.

Thoughts? cc: [~tarmstrong@cloudera.com], [~kwho]

> Reject or timeout new incoming connections once --fe_service_threads limit is reached
> -------------------------------------------------------------------------------------
>
>                 Key: IMPALA-7800
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7800
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Clients
>    Affects Versions: Impala 3.0, Impala 2.12.0
>            Reporter: Michael Ho
>            Assignee: Zoram Thanga
>            Priority: Major
>
> Currently, the number of frontend service threads is controlled by {{--fe_service_threads}}. Once this limit is reached, the worker threads of {{connection_setup_pool}} will block until some of the service threads exit. New incoming connections will be buffered in a queue in {{connection_setup_pool}} until it fills up. Currently, there is no time out for all these buffered connections in the queue. Consequently, if the number of frontend service threads max'ed out for an extended period of time, a client trying to connect to Impala will feel "stuck" until a front end service thread is available.
> We should consider either rejecting any new connections once {{--fe_service_threads}} limit is reached or imposing a tunable upper bound on the wait time for connections in the queue to improve the users' experience. cc'ing [~zoram]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org