You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by J Mohamed Zahoor <za...@indix.com> on 2013/04/17 11:46:18 UTC
Max http connections in CloudSolrServer
Hi
I am pumping parallel select queries using CloudSolrServer.
It looks like it can handle only certain no of max connections...
my Question is,
How many concurrent queries does a CloudSolrServer can handle?
An old thread tries to answer this by asking to give our own instance of LBHttpSolrServer...
But it looks like there is no way from LBHttpSolrServer to up the maxConnection of the httpClient it has...
Can someone let me know how to bump up the maxConnections and maxConnectionsPerHost parameter for the httpCLient used by cloudSolrServer?
./zahoor
Re: Max http connections in CloudSolrServer
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/17/2013 3:21 PM, Chris Hostetter wrote:
> I think the name comes from the effect it has on the underlying HttpClient
> code ... it's possible to configure a HttpConnectionManager such that it
> has different number of max connections per host -- ie: host1 has max
> connections of 23, host2 has max connections of 45, etc.... i believe
> that method just changes the "default" when there isn't something
> specifically set for an individual host..
That puts it into complete perspective, so changing the name is a bad
idea. I do think it might be a good idea to include HttpSolrServer's
convenience methods in the other classes, even if there is a working
HttpClient -> LBHttpSolrServer -> CloudSolrServer way to change things.
Thanks,
Shawn
Re: Max http connections in CloudSolrServer
Posted by Chris Hostetter <ho...@fucit.org>.
: Side issue: shouldn't that be setMaxConnectionsPerHost instead of including
: the word Default? If there's no objection, I would plan on adding the renamed
: method and using a typical deprecation procedure for the old one.
I think the name comes from the effect it has on the underlying HttpClient
code ... it's possible to configure a HttpConnectionManager such that it
has different number of max connections per host -- ie: host1 has max
connections of 23, host2 has max connections of 45, etc.... i believe
that method just changes the "default" when there isn't something
specifically set for an individual host..
-Hoss
Re: Max http connections in CloudSolrServer
Posted by J Mohamed Zahoor <za...@indix.com>.
On 18-Apr-2013, at 9:43 PM, Shawn Heisey <so...@elyograg.org> wrote:
> Are you using the Jetty included with Solr, or a Jetty installed separately?
I am using the Jetty that comes with Solr.
> The Jetty included with Solr has a maxThreads value of 10000 in its config. The default would be closer to 200, and a single request from a Cloud client likely uses multiple Jetty threads.
The default maxThreads is 10000 and minThreads is 10.
./zahoor
Re: Max http connections in CloudSolrServer
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/18/2013 6:42 AM, J Mohamed Zahoor wrote:
>
> I dont yet know if this is the reason...
> I am looking if jetty has some limit on accepting connections..
Are you using the Jetty included with Solr, or a Jetty installed
separately? The Jetty included with Solr has a maxThreads value of
10000 in its config. The default would be closer to 200, and a single
request from a Cloud client likely uses multiple Jetty threads.
Thanks,
Shawn
Re: Max http connections in CloudSolrServer
Posted by J Mohamed Zahoor <za...@indix.com>.
I dont yet know if this is the reason...
I am looking if jetty has some limit on accepting connections..
./zahoor
On 18-Apr-2013, at 12:52 PM, J Mohamed Zahoor <za...@indix.com> wrote:
>
> Thanks for this.
> The reason i asked this was.. when i fire 30 queries simultaneously from 30 threads using the same CloudSolrServer instance,
> some queries gets fired after a delay.. sometime the delay is 30-50 seconds...
>
> In solr logs i can see.. 20+ queries get fired almost immediately... but some of them gets fired late..
>
> i increased the connections per host from 32 to 200.. still no respite...
>
> ./zahoor
>
> On 18-Apr-2013, at 12:20 AM, Shawn Heisey <so...@elyograg.org> wrote:
>
>> ModifiableSolrParams params = new ModifiableSolrParams();
>> params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 1000);
>> params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 200);
>> HttpClient client = HttpClientUtil.createClient(params);
>> LBHttpSolrServer lbServer = new LBHttpSolrServer
>> (client, "http://localhost/solr");
>> lbServer.removeSolrServer("http://localhost/solr");
>> SolrServer server = new CloudSolrServer(zkHost, lbServer);
>
Re: Max http connections in CloudSolrServer
Posted by J Mohamed Zahoor <za...@indix.com>.
Thanks for this.
The reason i asked this was.. when i fire 30 queries simultaneously from 30 threads using the same CloudSolrServer instance,
some queries gets fired after a delay.. sometime the delay is 30-50 seconds...
In solr logs i can see.. 20+ queries get fired almost immediately... but some of them gets fired late..
i increased the connections per host from 32 to 200.. still no respite...
./zahoor
On 18-Apr-2013, at 12:20 AM, Shawn Heisey <so...@elyograg.org> wrote:
> ModifiableSolrParams params = new ModifiableSolrParams();
> params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 1000);
> params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 200);
> HttpClient client = HttpClientUtil.createClient(params);
> LBHttpSolrServer lbServer = new LBHttpSolrServer
> (client, "http://localhost/solr");
> lbServer.removeSolrServer("http://localhost/solr");
> SolrServer server = new CloudSolrServer(zkHost, lbServer);
Re: Max http connections in CloudSolrServer
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/17/2013 3:46 AM, J Mohamed Zahoor wrote:
> Hi
>
> I am pumping parallel select queries using CloudSolrServer.
> It looks like it can handle only certain no of max connections...
>
> my Question is,
> How many concurrent queries does a CloudSolrServer can handle?
Looking into the code for 4.x versions, I found that the default max
number of connections is 128, and the default max number of connections
per host is 32. This is set in the HttpSolrServer constructor.
> An old thread tries to answer this by asking to give our own instance of LBHttpSolrServer...
> But it looks like there is no way from LBHttpSolrServer to up the maxConnection of the httpClient it has...
>
> Can someone let me know how to bump up the maxConnections and maxConnectionsPerHost parameter for the httpCLient used by cloudSolrServer?
You should be able to create an instance of LBHttpSolrServer, which in
turn lets you use a custom HttpClient, where you could set the
connection limits. Then you could use the LBHttpSolrServer object to
create an instance of CloudSolrServer.
The following code MIGHT work (probably in a try block), but I haven't
tried to actually use it, so it might be horribly broken.
ModifiableSolrParams params = new ModifiableSolrParams();
params.set(HttpClientUtil.PROP_MAX_CONNECTIONS, 1000);
params.set(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 200);
HttpClient client = HttpClientUtil.createClient(params);
LBHttpSolrServer lbServer = new LBHttpSolrServer
(client, "http://localhost/solr");
lbServer.removeSolrServer("http://localhost/solr");
SolrServer server = new CloudSolrServer(zkHost, lbServer);
I would argue that CloudSolrServer (and therefore LBHttpSolrServer)
should have many of the setters available on HttpSolrServer, including
setDefaultMaxConnectionsPerHost and setMaxTotalConnections. Perhaps not
all of them, some of them are things that most people would never really
need, and some of them might not make sense for a clustered server object.
Side issue: shouldn't that be setMaxConnectionsPerHost instead of
including the word Default? If there's no objection, I would plan on
adding the renamed method and using a typical deprecation procedure for
the old one.
Thanks,
Shawn