HttpShardHandlerFactory - should max total connections be configurable?

[Shawn Heisey <so...@elyograg.org>]

While working on SOLR-5741, I noticed that HttpShardHandlerFactory did
not have an option to configure the max total connections.  I *think*
that this value ultimately gets configured at 10000.  Do we need to make
it configurable?  Does this get used for SolrCloud as well as the old
distributed search?

http://wiki.apache.org/solr/SolrConfigXml#Configuration_of_Shard_Handlers_for_Distributed_searches

The default of 10000 is well beyond what mere mortals will require, but
if there are 1000 shards and the search application manages to crank up
the simultaneous requests to 11, that won't be enough.

There are other things that would need changes to handle that scenario,
like maxThreads and the HTTP header size on the container.

If it doesn't already exist, we probably need a reference guide section
that covers what needs to be changed from defaults when you're going for
extreme scalability.

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org

Re: HttpShardHandlerFactory - should max total connections be configurable?

["Ramkumar R. Aiyengar" <an...@gmail.com>]

+1. We are probably already breaching this with 256 shards, something we
are still investigating.

On the question of other limits, there's also the limit on how many threads
your OS can support, we have in the past hit 65k threads and errored out
because of 64 shards per machine, not sure as yet if this is configurable
(needless to say a scalability problem nevertheless.. I know there are some
nascent ideas floating about with doing away with the thread per connection
model, but none concrete afaik).

We are currently migrating to 4.6.1 from 4.4, planning to attack this once
that's done. The mess you've got to clear up with the 4.4 SolrCloud after
its screwed makes you think twice before attempting to test it again :-)
On 18 Feb 2014 06:12, "Shawn Heisey" <so...@elyograg.org> wrote:

> While working on SOLR-5741, I noticed that HttpShardHandlerFactory did
> not have an option to configure the max total connections.  I *think*
> that this value ultimately gets configured at 10000.  Do we need to make
> it configurable?  Does this get used for SolrCloud as well as the old
> distributed search?
>
>
> http://wiki.apache.org/solr/SolrConfigXml#Configuration_of_Shard_Handlers_for_Distributed_searches
>
> The default of 10000 is well beyond what mere mortals will require, but
> if there are 1000 shards and the search application manages to crank up
> the simultaneous requests to 11, that won't be enough.
>
> There are other things that would need changes to handle that scenario,
> like maxThreads and the HTTP header size on the container.
>
> If it doesn't already exist, we probably need a reference guide section
> that covers what needs to be changed from defaults when you're going for
> extreme scalability.
>
> Thanks,
> Shawn
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>