[jira] [Commented] (SOLR-7344) Allow Jetty thread pool limits while still avoiding distributed deadlock.

["Mark Robert Miller (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-7344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17356766#comment-17356766 ] 

Mark Robert Miller commented on SOLR-7344:
------------------------------------------

[~atris] i missed that comment. Taking advantage of the QosFilter is really the best strategy. These other issues around thread pools don’t solve the problem at all. Given that you only want as many threads as you have cores + what may be needed while blocking or waiting (that can be reduced huge as well), current thread usage is nowhere near optimal or good. It’s extremely costly, circuit breaker approaches are better than nothing but equivalent to a hack compared to what you actually want. 

But yeah, I doubt that gets solved in Solr. But the largest issue is the unlimited jetty pool. This can be improved on a few fronts and should be. Beyond the obvious issues with the unlimited pool, jetty looks at the max threads config to configure and control behavior. Sizing the reserve executor. Deciding how many threads to keep spinning up in ensureThreads, etc. The certainly don’t expect to be consulting Integer.MAX_VALUE. 

There are all kinds of pitfalls with choosing a reasonable limit with Solr though. With the QoSFilter you can work towards a nicer model than hard thread limits anyway. You can dynamically limit the number of external concurrent requests.

In the end, you don’t want to actually “dumbly” started limited resources. So you mark internal requests with a header and limit external requests. That keeps in flight requests moving and this is how you work towards achieving more optimal throughout under load rather than death spirals or everyone pays starvation and mass partial work done rejections. 

External requests can be momentarily paused and then if necessary suspended - holding onto 0 threads while they wait for availability. If that takes too long, then you can reject with a proper response that keeps clients from retry bombarding. 

The goal should be high throughput and a consistent thread count not far above your available cpu cores. It takes way more than the above to do that, but circuit breakers and hard limits would never come anywhere near the optimal situation.  Just a more brute force, better than nothing approach. Jetty allows for that with its own thread pool via its configurable low resource detection sledgehammer. 

> Allow Jetty thread pool limits while still avoiding distributed deadlock.
> -------------------------------------------------------------------------
>
>                 Key: SOLR-7344
>                 URL: https://issues.apache.org/jira/browse/SOLR-7344
>             Project: Solr
>          Issue Type: Improvement
>          Components: SolrCloud
>            Reporter: Mark Miller
>            Priority: Major
>         Attachments: SOLR-7344.patch
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org