[jira] [Commented] (CASSANDRA-7594) Disruptor Thrift server worker thread pool not adjustable

["Rick Branson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-7594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14071975#comment-14071975 ] 

Rick Branson commented on CASSANDRA-7594:
-----------------------------------------

The ideal situation would be if rpc_min_threads and rpc_max_threads still worked, at least roughly. The problem with implementing this is that the TDisruptorServer only creates fixed-size thread pools specified by numWorkersPerSelector. The variable-size thread pools are useful because they allow working around temporary congestion problems by using a large number of executor threads, but fall back to a smaller thread pool when unneeded (i.e. 128 -> 2048). An example of this is when a two replicas for the same range are simultaneously slow or unresponsive using QUORUM on RF=3. The RPC thread pool is *rapidly* consumed, which can cause fallout to other, unrelated requests.

> Disruptor Thrift server worker thread pool not adjustable
> ---------------------------------------------------------
>
>                 Key: CASSANDRA-7594
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7594
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Rick Branson
>            Assignee: Pavel Yaskevich
>
> For the THsHaDisruptorServer, there may not be enough threads to run blocking StorageProxy methods. The current number of worker threads is hardcoded at 2 per selector, so 2 * numAvailableProcessors(), or 64 threads on a 16-core hyperthreaded machine. StorageProxy methods block these threads, so this puts an upper bound on the throughput if hsha is enabled. If operations take 10ms on average, the node can only handle a maximum of 6,400 operations per second. This is a regression from hsha on 1.2.x, where the thread pool was tunable using rpc_min_threads and rpc_max_threads.



--
This message was sent by Atlassian JIRA
(v6.2#6252)