You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bob Sandiford <bo...@sirsidynix.com> on 2011/06/29 17:45:43 UTC
Solr just 'hangs' under load test - ideas?
Hi, all.
I'm hoping someone has some thoughts here.
We're running Solr 3.1 (with the patch for SolrQueryParser.java to not do the getLuceneVersion() calls, but use luceneMatchVersion directly).
We're running in a Tomcat instance, 64 bit Java. CATALINA_OPTS are: -Xmx7168m -Xms7168m -XX:MaxPermSize=256M
We're running 2 Solr cores, with the same schema.
We use SolrJ to run our searches from a Java app running in JBoss.
JBoss, Tomcat, and the Solr Index folders are all on the same server.
In case it's relevant, we're using JMeter as a load test harness.
We're running on Solaris, a 16 processor box with 48GB physical memory.
I've run a successful load test at a 100 user load (at that rate there are about 5-10 solr searches / second), and solr search responses were coming in under 100ms.
When I tried to ramp up, as far as I can tell, Solr is just hanging. (We have some logging statements around the SolrJ calls - just before, we log how long our query construction takes, then we run the SolrJ query and log the search times. We're getting a number of the query construction logs, but no corresponding search time logs).
Symptoms:
The Tomcat and JBoss processes show as well under 1% CPU, and they are still the top processes. CPU states show around 99% idle. RES usage for the two Java processes around 3GB each. LWP under 120 for each. STATE just shows as sleep. JBoss is still 'alive', as I can get into a piece of software that talks to our JBoss app to get data.
We set things up to use log4j logging for Solr - the log isn't showing any errors or exceptions.
We're not indexing - just searching.
Back in January, we did load testing on a prototype, and had no problems (though that was Solr 1.4 at the time). It ramped up beautifully - bottle necks were our apps, not Solr. What I'm benchmarking now is a descendent of that prototyping - a bit more complex on searches and more fields in the schema, but same basic search logic as far as SolrJ usage.
Any ideas? What else to look at? Ringing any bells?
I can send more details if anyone wants specifics...
Bob Sandiford | Lead Software Engineer | SirsiDynix
P: 800.288.8020 X6943 | Bob.Sandiford@sirsidynix.com
www.sirsidynix.com<http://www.sirsidynix.com/>
RE: Solr just 'hangs' under load test - ideas?
Posted by Bob Sandiford <bo...@sirsidynix.com>.
OK - I figured it out. It's not solr at all (and I'm not really surprised).
In the prototype benchmarks, we used a different instance of tomcat than we're using for production load tests. Our prototype tomcat instance had no maxThreads value set, so was using the default value of 200. The production tomcat environment has a maxThreads value of 15 - we were just running out of threads and getting connection refused exceptions thrown when we ramped up the Solr hits past a certain level.
Thanks for considering, Yonik (and any others waiting to see any reply I made)...
(As others have said - this listserv is great!)
Bob Sandiford | Lead Software Engineer | SirsiDynix
P: 800.288.8020 X6943 | Bob.Sandiford@sirsidynix.com
www.sirsidynix.com
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
> Seeley
> Sent: Wednesday, June 29, 2011 12:18 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Solr just 'hangs' under load test - ideas?
>
> Can you get a thread dump to see what is hanging?
>
> -Yonik
> http://www.lucidimagination.com
>
> On Wed, Jun 29, 2011 at 11:45 AM, Bob Sandiford
> <bo...@sirsidynix.com> wrote:
> > Hi, all.
> >
> > I'm hoping someone has some thoughts here.
> >
> > We're running Solr 3.1 (with the patch for SolrQueryParser.java to
> not do the getLuceneVersion() calls, but use luceneMatchVersion
> directly).
> >
> > We're running in a Tomcat instance, 64 bit Java. CATALINA_OPTS are:
> -Xmx7168m -Xms7168m -XX:MaxPermSize=256M
> >
> > We're running 2 Solr cores, with the same schema.
> >
> > We use SolrJ to run our searches from a Java app running in JBoss.
> >
> > JBoss, Tomcat, and the Solr Index folders are all on the same server.
> >
> > In case it's relevant, we're using JMeter as a load test harness.
> >
> > We're running on Solaris, a 16 processor box with 48GB physical
> memory.
> >
> > I've run a successful load test at a 100 user load (at that rate
> there are about 5-10 solr searches / second), and solr search responses
> were coming in under 100ms.
> >
> > When I tried to ramp up, as far as I can tell, Solr is just hanging.
> (We have some logging statements around the SolrJ calls - just before,
> we log how long our query construction takes, then we run the SolrJ
> query and log the search times. We're getting a number of the query
> construction logs, but no corresponding search time logs).
> >
> > Symptoms:
> > The Tomcat and JBoss processes show as well under 1% CPU, and they
> are still the top processes. CPU states show around 99% idle. RES
> usage for the two Java processes around 3GB each. LWP under 120 for
> each. STATE just shows as sleep. JBoss is still 'alive', as I can get
> into a piece of software that talks to our JBoss app to get data.
> >
> > We set things up to use log4j logging for Solr - the log isn't
> showing any errors or exceptions.
> >
> > We're not indexing - just searching.
> >
> > Back in January, we did load testing on a prototype, and had no
> problems (though that was Solr 1.4 at the time). It ramped up
> beautifully - bottle necks were our apps, not Solr. What I'm
> benchmarking now is a descendent of that prototyping - a bit more
> complex on searches and more fields in the schema, but same basic
> search logic as far as SolrJ usage.
> >
> > Any ideas? What else to look at? Ringing any bells?
> >
> > I can send more details if anyone wants specifics...
> >
> > Bob Sandiford | Lead Software Engineer | SirsiDynix
> > P: 800.288.8020 X6943 | Bob.Sandiford@sirsidynix.com
> > www.sirsidynix.com<http://www.sirsidynix.com/>
> >
> >
Re: Solr just 'hangs' under load test - ideas?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
Can you get a thread dump to see what is hanging?
-Yonik
http://www.lucidimagination.com
On Wed, Jun 29, 2011 at 11:45 AM, Bob Sandiford
<bo...@sirsidynix.com> wrote:
> Hi, all.
>
> I'm hoping someone has some thoughts here.
>
> We're running Solr 3.1 (with the patch for SolrQueryParser.java to not do the getLuceneVersion() calls, but use luceneMatchVersion directly).
>
> We're running in a Tomcat instance, 64 bit Java. CATALINA_OPTS are: -Xmx7168m -Xms7168m -XX:MaxPermSize=256M
>
> We're running 2 Solr cores, with the same schema.
>
> We use SolrJ to run our searches from a Java app running in JBoss.
>
> JBoss, Tomcat, and the Solr Index folders are all on the same server.
>
> In case it's relevant, we're using JMeter as a load test harness.
>
> We're running on Solaris, a 16 processor box with 48GB physical memory.
>
> I've run a successful load test at a 100 user load (at that rate there are about 5-10 solr searches / second), and solr search responses were coming in under 100ms.
>
> When I tried to ramp up, as far as I can tell, Solr is just hanging. (We have some logging statements around the SolrJ calls - just before, we log how long our query construction takes, then we run the SolrJ query and log the search times. We're getting a number of the query construction logs, but no corresponding search time logs).
>
> Symptoms:
> The Tomcat and JBoss processes show as well under 1% CPU, and they are still the top processes. CPU states show around 99% idle. RES usage for the two Java processes around 3GB each. LWP under 120 for each. STATE just shows as sleep. JBoss is still 'alive', as I can get into a piece of software that talks to our JBoss app to get data.
>
> We set things up to use log4j logging for Solr - the log isn't showing any errors or exceptions.
>
> We're not indexing - just searching.
>
> Back in January, we did load testing on a prototype, and had no problems (though that was Solr 1.4 at the time). It ramped up beautifully - bottle necks were our apps, not Solr. What I'm benchmarking now is a descendent of that prototyping - a bit more complex on searches and more fields in the schema, but same basic search logic as far as SolrJ usage.
>
> Any ideas? What else to look at? Ringing any bells?
>
> I can send more details if anyone wants specifics...
>
> Bob Sandiford | Lead Software Engineer | SirsiDynix
> P: 800.288.8020 X6943 | Bob.Sandiford@sirsidynix.com
> www.sirsidynix.com<http://www.sirsidynix.com/>
>
>