Limiting Jetty threads?

[David Smiley <ds...@apache.org>]

Has anyone experimented with reducing the number of request/Jetty threads
in jetty.xml?  The default is 10K.  I'm concerned about the use of
ThreadLocals caching stuff, like maybe Lucene analysis chains (Analyzer
ReuseStrategy) or other things.  I think Michael Gibney reported to the
Jetty project that these could be closed more aggressively.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley

Re: Limiting Jetty threads?

[Michael Gibney <mi...@michaelgibney.net>]

Yes, this was observed to be a problem with
CompressedStoredFieldsReader, and there may indeed be other cases. The
new storedFields API should help as of Lucene 9.5, but I think we'd
still do well to also pursue a faster QueuedThreadPool shrinking
configuration.

The current Solr default jetty config of 10k maxThreads is less
problematic IMO than the default jetty idleTimeout config of 2
minutes, which amounts to "pool shrinks at the rate of 1 thread every
2 minutes". I suspect a default idleTimeout of 5s or something would
be preferable, but I'm hopeful that we'll have better options soon
from upstream jetty. I wouldn't be surprised if many cases could do
just fine by reducing the maxThreads below 10k, but again I think the
shrink rate is the more important factor, and memory issues are
possible (at least pre-Lucene-9.5) even at way fewer threads (e.g.,
4k).

There's a PR eclipse/jetty.project#9498 [1] that improves the
situation somewhat (especially for shrinking fast) and appears to be
close to being merged; but if configured to shrink very fast it will
shrink pool capacity that shouldn't be considered "idle" by any
reasonable definition. I have an open PR eclipse/jetty.project#9532
[2] that's been tracking the first PR, and fully separates the concept
of pool shrink rate from that of idleTimeout as a keepalive/minimumTTL
for pool capacity to be considered idle (I'd appreciate any feedback
on the PR btw). The initial enhancement proposal also provides some
context [3].

Tangential but relevant: definitely eagerly anticipating incorporating
Lucene-9.5/storedFields change.

[1] https://github.com/eclipse/jetty.project/pull/9498
[2] https://github.com/eclipse/jetty.project/pull/9532
[3] https://github.com/eclipse/jetty.project/issues/9237

On Wed, Mar 29, 2023 at 5:51 PM David Smiley <ds...@apache.org> wrote:
>
> Has anyone experimented with reducing the number of request/Jetty threads
> in jetty.xml?  The default is 10K.  I'm concerned about the use of
> ThreadLocals caching stuff, like maybe Lucene analysis chains (Analyzer
> ReuseStrategy) or other things.  I think Michael Gibney reported to the
> Jetty project that these could be closed more aggressively.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org