You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by climbingrose <cl...@gmail.com> on 2007/12/18 03:00:43 UTC
Resource contention problem in Solrj
There seems to be resource contention problem with Solrj under load. To
reproduce the problem: set up a sample webapp with solrj connect to a HTTP
Solr instance and hammer the webapp with Apache ab (say 10 concurrent
connection with 100 requests). You'll notice that the webapp's servlet
container quickly consumes 100% CPU and stays there unless you restart it. I
can confirm that this happens with both Tomcat and Jetty. Meanwhile, the
server that Solr is deployed on seems to be running fine.
>From this observation, I suspect that Solrj has connection contention
problem. And this seems to be the case if you look at CommonHttpSolrServer.
This class uses MultiThreadedHttpConnectionManager which has
maxConnectionsPerHost set to 2 by default. When the number of thread
increases, this is obviously not enough and leads to connection contention
problem. I quickly solve problem by adding another constructor to
CommonHttpSolrServer that allows setting maxConnectionsPerHost and
maxTotalConnections:
public CommonsHttpSolrServer(int maxConsPerHost, int maxTotalCons, String
solrServerUrl) throws MalformedURLException {
this(solrServerUrl);
this.maxConsPerHost = maxConsPerHost;
this.maxTotalCons = maxTotalCons;
HttpConnectionManagerParams params = new HttpConnectionManagerParams();
params.setDefaultMaxConnectionsPerHost(maxConsPerHost);
params.setMaxTotalConnections(maxTotalCons);
_connectionManager.setParams(params);
}
Hope this information would help others.
--
Regards,
Cuong Hoang
Re: Resource contention problem in Solrj
Posted by Ryan McKinley <ry...@gmail.com>.
Yonik Seeley wrote:
> On Dec 18, 2007 9:50 AM, Ryan McKinley <ry...@gmail.com> wrote:
>> perhaps we should increase the default maxConnectionsPerHost. 10? We
>> should also add some comment about setting it?
>
> Yes, we should definitely change the default (probably higher... 32 or
> 64?). The total max connections should be increased to (if too low).
>
check rev 605324
increased to 32/host and 128 max -- originally 2/host and 20 max.
Other number suggestions?
> I assume if you want to hit a solr server with multiple threads from
> SolrJ, that the best practice / indended use is to create a single
> SolrServer, and not one per-thread?
>
I always use one SolrServer instance shared across threads. It seems
like MultiThreadedHttpConnectionManager will do a better job managing
threads then we would.
ryan
Re: Resource contention problem in Solrj
Posted by Yonik Seeley <yo...@apache.org>.
On Dec 18, 2007 9:50 AM, Ryan McKinley <ry...@gmail.com> wrote:
> perhaps we should increase the default maxConnectionsPerHost. 10? We
> should also add some comment about setting it?
Yes, we should definitely change the default (probably higher... 32 or
64?). The total max connections should be increased to (if too low).
I assume if you want to hit a solr server with multiple threads from
SolrJ, that the best practice / indended use is to create a single
SolrServer, and not one per-thread?
-Yonik
Re: Resource contention problem in Solrj
Posted by Ryan McKinley <ry...@gmail.com>.
climbingrose wrote:
> There seems to be resource contention problem with Solrj under load. To
> reproduce the problem: set up a sample webapp with solrj connect to a HTTP
> Solr instance and hammer the webapp with Apache ab (say 10 concurrent
> connection with 100 requests). You'll notice that the webapp's servlet
> container quickly consumes 100% CPU and stays there unless you restart it. I
> can confirm that this happens with both Tomcat and Jetty. Meanwhile, the
> server that Solr is deployed on seems to be running fine.
>
>>>From this observation, I suspect that Solrj has connection contention
> problem. And this seems to be the case if you look at CommonHttpSolrServer.
> This class uses MultiThreadedHttpConnectionManager which has
> maxConnectionsPerHost set to 2 by default. When the number of thread
> increases, this is obviously not enough and leads to connection contention
I'm reluctant to add a constructor to the API since all the params can
be set via getConnectionManager()
perhaps we should increase the default maxConnectionsPerHost. 10? We
should also add some comment about setting it?
>
> Hope this information would help others.
>
yes, thanks!
ryan
Re: Resource contention problem in Solrj
Posted by climbingrose <cl...@gmail.com>.
Oops! I didn't notice these methods. However, for me, it's convenient to
have the constructor so that I can easily create SolrServer instance in
Spring.
On Dec 19, 2007 12:19 AM, Will Johnson <wi...@gmail.com> wrote:
> Fyi: the CommonsHttpSolrServer already has method to do all of those
> things:
>
> /** set connectionTimeout on the underlying
> MultiThreadedHttpConnectionManager */
> public void setConnectionTimeout(int timeout) {
> _connectionManager.getParams().setConnectionTimeout(timeout);
> }
>
> /** set maxConnectionsPerHost on the underlying
> MultiThreadedHttpConnectionManager */
> public void setDefaultMaxConnectionsPerHost(int connections) {
>
>
> _connectionManager.getParams().setDefaultMaxConnectionsPerHost(connections);
> }
>
> /** set maxTotalConnection on the underlying
> MultiThreadedHttpConnectionManager */
> public void setMaxTotalConnections(int connections) {
> _connectionManager.getParams().setMaxTotalConnections(connections);
> }
>
>
> You can also get the underlying connection factory if you want to do other
> crazier stuff.
>
> public MultiThreadedHttpConnectionManager getConnectionManager() {
> return _connectionManager;
> }
>
>
>
> - will
>
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
> Seeley
> Sent: Monday, December 17, 2007 9:18 PM
> To: solr-dev@lucene.apache.org
> Subject: Re: Resource contention problem in Solrj
>
> Excellent! Thanks for diagnosing this!
> -Yonik
>
> On Dec 17, 2007 9:00 PM, climbingrose <cl...@gmail.com> wrote:
> > There seems to be resource contention problem with Solrj under load. To
> > reproduce the problem: set up a sample webapp with solrj connect to a
> HTTP
> > Solr instance and hammer the webapp with Apache ab (say 10 concurrent
> > connection with 100 requests). You'll notice that the webapp's servlet
> > container quickly consumes 100% CPU and stays there unless you restart
> it.
> I
> > can confirm that this happens with both Tomcat and Jetty. Meanwhile, the
> > server that Solr is deployed on seems to be running fine.
> >
> > From this observation, I suspect that Solrj has connection contention
> > problem. And this seems to be the case if you look at
> CommonHttpSolrServer.
> > This class uses MultiThreadedHttpConnectionManager which has
> > maxConnectionsPerHost set to 2 by default. When the number of thread
> > increases, this is obviously not enough and leads to connection
> contention
> > problem. I quickly solve problem by adding another constructor to
> > CommonHttpSolrServer that allows setting maxConnectionsPerHost and
> > maxTotalConnections:
> >
> > public CommonsHttpSolrServer(int maxConsPerHost, int maxTotalCons,
> String
> > solrServerUrl) throws MalformedURLException {
> > this(solrServerUrl);
> > this.maxConsPerHost = maxConsPerHost;
> > this.maxTotalCons = maxTotalCons;
> > HttpConnectionManagerParams params = new
> HttpConnectionManagerParams();
> > params.setDefaultMaxConnectionsPerHost(maxConsPerHost);
> > params.setMaxTotalConnections(maxTotalCons);
> > _connectionManager.setParams(params);
> > }
> >
> > Hope this information would help others.
> >
> > --
> > Regards,
> >
> > Cuong Hoang
> >
>
>
--
Regards,
Cuong Hoang
RE: Resource contention problem in Solrj
Posted by Will Johnson <wi...@gmail.com>.
Fyi: the CommonsHttpSolrServer already has method to do all of those
things:
/** set connectionTimeout on the underlying
MultiThreadedHttpConnectionManager */
public void setConnectionTimeout(int timeout) {
_connectionManager.getParams().setConnectionTimeout(timeout);
}
/** set maxConnectionsPerHost on the underlying
MultiThreadedHttpConnectionManager */
public void setDefaultMaxConnectionsPerHost(int connections) {
_connectionManager.getParams().setDefaultMaxConnectionsPerHost(connections);
}
/** set maxTotalConnection on the underlying
MultiThreadedHttpConnectionManager */
public void setMaxTotalConnections(int connections) {
_connectionManager.getParams().setMaxTotalConnections(connections);
}
You can also get the underlying connection factory if you want to do other
crazier stuff.
public MultiThreadedHttpConnectionManager getConnectionManager() {
return _connectionManager;
}
- will
-----Original Message-----
From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik Seeley
Sent: Monday, December 17, 2007 9:18 PM
To: solr-dev@lucene.apache.org
Subject: Re: Resource contention problem in Solrj
Excellent! Thanks for diagnosing this!
-Yonik
On Dec 17, 2007 9:00 PM, climbingrose <cl...@gmail.com> wrote:
> There seems to be resource contention problem with Solrj under load. To
> reproduce the problem: set up a sample webapp with solrj connect to a HTTP
> Solr instance and hammer the webapp with Apache ab (say 10 concurrent
> connection with 100 requests). You'll notice that the webapp's servlet
> container quickly consumes 100% CPU and stays there unless you restart it.
I
> can confirm that this happens with both Tomcat and Jetty. Meanwhile, the
> server that Solr is deployed on seems to be running fine.
>
> From this observation, I suspect that Solrj has connection contention
> problem. And this seems to be the case if you look at
CommonHttpSolrServer.
> This class uses MultiThreadedHttpConnectionManager which has
> maxConnectionsPerHost set to 2 by default. When the number of thread
> increases, this is obviously not enough and leads to connection contention
> problem. I quickly solve problem by adding another constructor to
> CommonHttpSolrServer that allows setting maxConnectionsPerHost and
> maxTotalConnections:
>
> public CommonsHttpSolrServer(int maxConsPerHost, int maxTotalCons, String
> solrServerUrl) throws MalformedURLException {
> this(solrServerUrl);
> this.maxConsPerHost = maxConsPerHost;
> this.maxTotalCons = maxTotalCons;
> HttpConnectionManagerParams params = new
HttpConnectionManagerParams();
> params.setDefaultMaxConnectionsPerHost(maxConsPerHost);
> params.setMaxTotalConnections(maxTotalCons);
> _connectionManager.setParams(params);
> }
>
> Hope this information would help others.
>
> --
> Regards,
>
> Cuong Hoang
>
Re: Resource contention problem in Solrj
Posted by Yonik Seeley <yo...@apache.org>.
Excellent! Thanks for diagnosing this!
-Yonik
On Dec 17, 2007 9:00 PM, climbingrose <cl...@gmail.com> wrote:
> There seems to be resource contention problem with Solrj under load. To
> reproduce the problem: set up a sample webapp with solrj connect to a HTTP
> Solr instance and hammer the webapp with Apache ab (say 10 concurrent
> connection with 100 requests). You'll notice that the webapp's servlet
> container quickly consumes 100% CPU and stays there unless you restart it. I
> can confirm that this happens with both Tomcat and Jetty. Meanwhile, the
> server that Solr is deployed on seems to be running fine.
>
> From this observation, I suspect that Solrj has connection contention
> problem. And this seems to be the case if you look at CommonHttpSolrServer.
> This class uses MultiThreadedHttpConnectionManager which has
> maxConnectionsPerHost set to 2 by default. When the number of thread
> increases, this is obviously not enough and leads to connection contention
> problem. I quickly solve problem by adding another constructor to
> CommonHttpSolrServer that allows setting maxConnectionsPerHost and
> maxTotalConnections:
>
> public CommonsHttpSolrServer(int maxConsPerHost, int maxTotalCons, String
> solrServerUrl) throws MalformedURLException {
> this(solrServerUrl);
> this.maxConsPerHost = maxConsPerHost;
> this.maxTotalCons = maxTotalCons;
> HttpConnectionManagerParams params = new HttpConnectionManagerParams();
> params.setDefaultMaxConnectionsPerHost(maxConsPerHost);
> params.setMaxTotalConnections(maxTotalCons);
> _connectionManager.setParams(params);
> }
>
> Hope this information would help others.
>
> --
> Regards,
>
> Cuong Hoang
>