You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Nate Fox <nf...@neogov.com> on 2013/03/26 18:24:55 UTC
Loadtesting solr/tomcat7 and tomcat stops responding entirely
I'm new to solr and I'm load testing our setup to see what we can handle.
I'm using solrmeter and my problem is a bit odd:
* When I set solrmeter to run 4000 queries/min, it will handle a few
hundred queries and then tomcat will stop responding completely to requests
(even though according to lsof -i it is still listening and the java
process is still running).
* When I set solrmeter to run 1000 queries/min it runs fine. I can stop
solrmeter after a couple of minutes at that pace and then run at 4000/min
without issue.
It's as if it needs a ramp up time? Also, I noticed (regardless of ramp up)
that my setup cannot handle 8000/min. The reaction at 8k/min is the same as
if I were to run 4k/min without the ramp up. Of note, only the shard that
solrmeter is pointed to stops responding. The other shard hums along
without incident.
Setup (everything in AWS):
- 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0
(open-jdk-7-headless) : Ubuntu 12.04
- 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04
I have ~30k documents in each node (~300Mb on each node)
The vast majority of my solr/tomcat7 config is default from ubuntu's
packages/solr's example dir. Here's the configs and the end of the
catalina.out file:https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0
My main question is two fold:
1. Is this normal behavior for tomcat (to just stop responding completely)
when it gets overwhelmed? And the only option is to restart it? I guess I
dont know what it looks like when tomcat/solr cant keep up.
2. Why does it handle better when I give it a lower number of queries and
then ramp it up? It concerns me that if I have to restart a server in the
cluster and it gets thrown into the pool of machines that things will blow
up.
As an aside, does this seem like a normal amount of queries (~4k/min) that
this kind of environment should be able to handle?
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Posted by Furkan KAMACI <fu...@gmail.com>.
Hi Nate;
This may be out of topic however could you explain that why you want to use
Tomcat instead of Jetty or Embedded Jetty?
2013/3/27 Michael Della Bitta <mi...@appinions.com>
> You're using the blocking IO connector, which isn't so great for heavy
> loads.
>
> Give this a shot... You'll end up with 8192 max connections by
> default, although this is tunable too:
>
> Run:
> apt-get install libapr1 libtcnative-1
>
> Add this to the list of Listeners at the top of server.xml:
>
> <Listener className="org.apache.catalina.core.AprLifecycleListener"
> SSLEngine="off" />
>
> These instructions assume you're running Tomcat 6 or 7.
>
> Here's some documentation:
> http://tomcat.apache.org/tomcat-7.0-doc/apr.html
> http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
>
>
> Michael Della Bitta
>
> ------------------------------------------------
> Appinions
> 18 East 41st Street, 2nd Floor
> New York, NY 10017-6271
>
> www.appinions.com
>
> Where Influence Isn’t a Game
>
>
> On Tue, Mar 26, 2013 at 5:31 PM, Nate Fox <nf...@neogov.com> wrote:
> > We're not using ELB and I have no idea which connector I'm using - I'm
> > guessing whatever is default (I'm a total noob). This is from my
> server.xml:
> > <Connector port="8080" protocol="HTTP/1.1"
> connectionTimeout="60000"
> > URIEncoding="UTF-8" redirectPort="8443" />
> >
> >
> >
> > --
> > Nate Fox
> > Sr Systems Engineer
> >
> > o: 310.658.5775
> > m: 714.248.5350
> >
> > Follow us @NEOGOV <http://twitter.com/NEOGOV> and on
> > Facebook<http://www.facebook.com/neogov>
> >
> > NEOGOV <http://www.neogov.com/> is among the top fastest growing
> software
> > companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and
> > the LA Business Journal. We are hiring!<
> http://www.neogov.com/#/company/careers>
> >
> >
> >
> > On Tue, Mar 26, 2013 at 1:02 PM, Michael Della Bitta <
> > michael.della.bitta@appinions.com> wrote:
> >
> >> Nate,
> >>
> >> We just cleared up a problem similar to this by ditching Elastic Load
> >> Balancer and switching over to the APR connector in Tomcat. Are you
> >> using either of those?
> >>
> >> Michael Della Bitta
> >>
> >> ------------------------------------------------
> >> Appinions
> >> 18 East 41st Street, 2nd Floor
> >> New York, NY 10017-6271
> >>
> >> www.appinions.com
> >>
> >> Where Influence Isn’t a Game
> >>
> >>
> >> On Tue, Mar 26, 2013 at 2:58 PM, Otis Gospodnetic
> >> <ot...@gmail.com> wrote:
> >> > Hi Nate,
> >> >
> >> > Try adding some warmup queries and making sure the setting for using
> >> > the cold searcher in solrconfig.xml is set to false. Your warmup
> >> > queries should use facets and sorting if your normal queries use them.
> >> > In SPM you'll actually see how much time warming up takes, so you'll
> >> > get a better idea of the "cost" of that (when you don't do it).
> >> >
> >> > Otis
> >> > --
> >> > Solr & ElasticSearch Support
> >> > http://sematext.com/
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > On Tue, Mar 26, 2013 at 2:50 PM, Nate Fox <nf...@neogov.com> wrote:
> >> >> I was wondering if the warmup stuff was one of the culprits (we dont
> >> have
> >> >> warmup's at all - the configs are pretty stock).
> >> >> As for the system, it seems capable of quite a bit more: memory
> usage is
> >> >> ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb)
> >> and
> >> >> load below 1.00.
> >> >>
> >> >> The seed data and queries were put together by one of our developers.
> >> I've
> >> >> put all the solrmeter files here:
> >> >> https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce
> >> >> Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely
> >> sure
> >> >> which file does which specifically.
> >> >>
> >> >> Does the system's reaction to a 'fast load' without a warmup sound
> >> normal?
> >> >> I would have expected the first couple hundred queries to be very
> slow
> >> >> (>500ms) and then the system catch up after a while. But it just dies
> >> very
> >> >> quickly and never recovers.
> >> >>
> >> >> I'll check out your SPM - I've seen it mentioned before. Thanks!
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Nate Fox
> >> >> Sr Systems Engineer
> >> >>
> >> >> o: 310.658.5775
> >> >> m: 714.248.5350
> >> >>
> >> >> Follow us @NEOGOV <http://twitter.com/NEOGOV> and on
> >> >> Facebook<http://www.facebook.com/neogov>
> >> >>
> >> >> NEOGOV <http://www.neogov.com/> is among the top fastest growing
> >> software
> >> >> companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500,
> and
> >> >> the LA Business Journal. We are hiring!<
> >> http://www.neogov.com/#/company/careers>
> >> >>
> >> >>
> >> >>
> >> >> On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic <
> >> >> otis.gospodnetic@gmail.com> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> In short, certain data structures need to load from index in the
> >> >>> beginning, (for sorting and faceting) caches need to warm up, JVM
> >> >>> needs to warm up, etc., so going slowly in the beginning makes
> sense.
> >> >>> Why things die after that is a different Q. Maybe it OOMs? Maybe
> >> >>> queries are very complex? What do your queries look like? I see
> >> >>> newrelic.jar in the command-line. May want to try SPM for Solr, it
> >> >>> has better Solr metrics.
> >> >>>
> >> >>> Otis
> >> >>> --
> >> >>> Solr & ElasticSearch Support
> >> >>> http://sematext.com/
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox <nf...@neogov.com> wrote:
> >> >>> > I'm new to solr and I'm load testing our setup to see what we can
> >> handle.
> >> >>> > I'm using solrmeter and my problem is a bit odd:
> >> >>> > * When I set solrmeter to run 4000 queries/min, it will handle a
> few
> >> >>> > hundred queries and then tomcat will stop responding completely to
> >> >>> requests
> >> >>> > (even though according to lsof -i it is still listening and the
> java
> >> >>> > process is still running).
> >> >>> > * When I set solrmeter to run 1000 queries/min it runs fine. I can
> >> stop
> >> >>> > solrmeter after a couple of minutes at that pace and then run at
> >> >>> 4000/min
> >> >>> > without issue.
> >> >>> >
> >> >>> > It's as if it needs a ramp up time? Also, I noticed (regardless of
> >> ramp
> >> >>> up)
> >> >>> > that my setup cannot handle 8000/min. The reaction at 8k/min is
> the
> >> same
> >> >>> as
> >> >>> > if I were to run 4k/min without the ramp up. Of note, only the
> shard
> >> that
> >> >>> > solrmeter is pointed to stops responding. The other shard hums
> along
> >> >>> > without incident.
> >> >>> >
> >> >>> > Setup (everything in AWS):
> >> >>> > - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0
> >> >>> > (open-jdk-7-headless) : Ubuntu 12.04
> >> >>> > - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04
> >> >>> > I have ~30k documents in each node (~300Mb on each node)
> >> >>> >
> >> >>> > The vast majority of my solr/tomcat7 config is default from
> ubuntu's
> >> >>> > packages/solr's example dir. Here's the configs and the end of the
> >> >>> > catalina.out file:
> >> https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0
> >> >>> >
> >> >>> > My main question is two fold:
> >> >>> > 1. Is this normal behavior for tomcat (to just stop responding
> >> >>> completely)
> >> >>> > when it gets overwhelmed? And the only option is to restart it? I
> >> guess I
> >> >>> > dont know what it looks like when tomcat/solr cant keep up.
> >> >>> > 2. Why does it handle better when I give it a lower number of
> >> queries and
> >> >>> > then ramp it up? It concerns me that if I have to restart a server
> >> in the
> >> >>> > cluster and it gets thrown into the pool of machines that things
> will
> >> >>> blow
> >> >>> > up.
> >> >>> >
> >> >>> > As an aside, does this seem like a normal amount of queries
> (~4k/min)
> >> >>> that
> >> >>> > this kind of environment should be able to handle?
> >> >>>
> >>
>
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Posted by Michael Della Bitta <mi...@appinions.com>.
You're using the blocking IO connector, which isn't so great for heavy loads.
Give this a shot... You'll end up with 8192 max connections by
default, although this is tunable too:
Run:
apt-get install libapr1 libtcnative-1
Add this to the list of Listeners at the top of server.xml:
<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="off" />
These instructions assume you're running Tomcat 6 or 7.
Here's some documentation:
http://tomcat.apache.org/tomcat-7.0-doc/apr.html
http://tomcat.apache.org/tomcat-7.0-doc/config/http.html
Michael Della Bitta
------------------------------------------------
Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271
www.appinions.com
Where Influence Isn’t a Game
On Tue, Mar 26, 2013 at 5:31 PM, Nate Fox <nf...@neogov.com> wrote:
> We're not using ELB and I have no idea which connector I'm using - I'm
> guessing whatever is default (I'm a total noob). This is from my server.xml:
> <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="60000"
> URIEncoding="UTF-8" redirectPort="8443" />
>
>
>
> --
> Nate Fox
> Sr Systems Engineer
>
> o: 310.658.5775
> m: 714.248.5350
>
> Follow us @NEOGOV <http://twitter.com/NEOGOV> and on
> Facebook<http://www.facebook.com/neogov>
>
> NEOGOV <http://www.neogov.com/> is among the top fastest growing software
> companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and
> the LA Business Journal. We are hiring!<http://www.neogov.com/#/company/careers>
>
>
>
> On Tue, Mar 26, 2013 at 1:02 PM, Michael Della Bitta <
> michael.della.bitta@appinions.com> wrote:
>
>> Nate,
>>
>> We just cleared up a problem similar to this by ditching Elastic Load
>> Balancer and switching over to the APR connector in Tomcat. Are you
>> using either of those?
>>
>> Michael Della Bitta
>>
>> ------------------------------------------------
>> Appinions
>> 18 East 41st Street, 2nd Floor
>> New York, NY 10017-6271
>>
>> www.appinions.com
>>
>> Where Influence Isn’t a Game
>>
>>
>> On Tue, Mar 26, 2013 at 2:58 PM, Otis Gospodnetic
>> <ot...@gmail.com> wrote:
>> > Hi Nate,
>> >
>> > Try adding some warmup queries and making sure the setting for using
>> > the cold searcher in solrconfig.xml is set to false. Your warmup
>> > queries should use facets and sorting if your normal queries use them.
>> > In SPM you'll actually see how much time warming up takes, so you'll
>> > get a better idea of the "cost" of that (when you don't do it).
>> >
>> > Otis
>> > --
>> > Solr & ElasticSearch Support
>> > http://sematext.com/
>> >
>> >
>> >
>> >
>> >
>> > On Tue, Mar 26, 2013 at 2:50 PM, Nate Fox <nf...@neogov.com> wrote:
>> >> I was wondering if the warmup stuff was one of the culprits (we dont
>> have
>> >> warmup's at all - the configs are pretty stock).
>> >> As for the system, it seems capable of quite a bit more: memory usage is
>> >> ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb)
>> and
>> >> load below 1.00.
>> >>
>> >> The seed data and queries were put together by one of our developers.
>> I've
>> >> put all the solrmeter files here:
>> >> https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce
>> >> Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely
>> sure
>> >> which file does which specifically.
>> >>
>> >> Does the system's reaction to a 'fast load' without a warmup sound
>> normal?
>> >> I would have expected the first couple hundred queries to be very slow
>> >> (>500ms) and then the system catch up after a while. But it just dies
>> very
>> >> quickly and never recovers.
>> >>
>> >> I'll check out your SPM - I've seen it mentioned before. Thanks!
>> >>
>> >>
>> >>
>> >> --
>> >> Nate Fox
>> >> Sr Systems Engineer
>> >>
>> >> o: 310.658.5775
>> >> m: 714.248.5350
>> >>
>> >> Follow us @NEOGOV <http://twitter.com/NEOGOV> and on
>> >> Facebook<http://www.facebook.com/neogov>
>> >>
>> >> NEOGOV <http://www.neogov.com/> is among the top fastest growing
>> software
>> >> companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and
>> >> the LA Business Journal. We are hiring!<
>> http://www.neogov.com/#/company/careers>
>> >>
>> >>
>> >>
>> >> On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic <
>> >> otis.gospodnetic@gmail.com> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> In short, certain data structures need to load from index in the
>> >>> beginning, (for sorting and faceting) caches need to warm up, JVM
>> >>> needs to warm up, etc., so going slowly in the beginning makes sense.
>> >>> Why things die after that is a different Q. Maybe it OOMs? Maybe
>> >>> queries are very complex? What do your queries look like? I see
>> >>> newrelic.jar in the command-line. May want to try SPM for Solr, it
>> >>> has better Solr metrics.
>> >>>
>> >>> Otis
>> >>> --
>> >>> Solr & ElasticSearch Support
>> >>> http://sematext.com/
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox <nf...@neogov.com> wrote:
>> >>> > I'm new to solr and I'm load testing our setup to see what we can
>> handle.
>> >>> > I'm using solrmeter and my problem is a bit odd:
>> >>> > * When I set solrmeter to run 4000 queries/min, it will handle a few
>> >>> > hundred queries and then tomcat will stop responding completely to
>> >>> requests
>> >>> > (even though according to lsof -i it is still listening and the java
>> >>> > process is still running).
>> >>> > * When I set solrmeter to run 1000 queries/min it runs fine. I can
>> stop
>> >>> > solrmeter after a couple of minutes at that pace and then run at
>> >>> 4000/min
>> >>> > without issue.
>> >>> >
>> >>> > It's as if it needs a ramp up time? Also, I noticed (regardless of
>> ramp
>> >>> up)
>> >>> > that my setup cannot handle 8000/min. The reaction at 8k/min is the
>> same
>> >>> as
>> >>> > if I were to run 4k/min without the ramp up. Of note, only the shard
>> that
>> >>> > solrmeter is pointed to stops responding. The other shard hums along
>> >>> > without incident.
>> >>> >
>> >>> > Setup (everything in AWS):
>> >>> > - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0
>> >>> > (open-jdk-7-headless) : Ubuntu 12.04
>> >>> > - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04
>> >>> > I have ~30k documents in each node (~300Mb on each node)
>> >>> >
>> >>> > The vast majority of my solr/tomcat7 config is default from ubuntu's
>> >>> > packages/solr's example dir. Here's the configs and the end of the
>> >>> > catalina.out file:
>> https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0
>> >>> >
>> >>> > My main question is two fold:
>> >>> > 1. Is this normal behavior for tomcat (to just stop responding
>> >>> completely)
>> >>> > when it gets overwhelmed? And the only option is to restart it? I
>> guess I
>> >>> > dont know what it looks like when tomcat/solr cant keep up.
>> >>> > 2. Why does it handle better when I give it a lower number of
>> queries and
>> >>> > then ramp it up? It concerns me that if I have to restart a server
>> in the
>> >>> > cluster and it gets thrown into the pool of machines that things will
>> >>> blow
>> >>> > up.
>> >>> >
>> >>> > As an aside, does this seem like a normal amount of queries (~4k/min)
>> >>> that
>> >>> > this kind of environment should be able to handle?
>> >>>
>>
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Posted by Nate Fox <nf...@neogov.com>.
We're not using ELB and I have no idea which connector I'm using - I'm
guessing whatever is default (I'm a total noob). This is from my server.xml:
<Connector port="8080" protocol="HTTP/1.1" connectionTimeout="60000"
URIEncoding="UTF-8" redirectPort="8443" />
--
Nate Fox
Sr Systems Engineer
o: 310.658.5775
m: 714.248.5350
Follow us @NEOGOV <http://twitter.com/NEOGOV> and on
Facebook<http://www.facebook.com/neogov>
NEOGOV <http://www.neogov.com/> is among the top fastest growing software
companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and
the LA Business Journal. We are hiring!<http://www.neogov.com/#/company/careers>
On Tue, Mar 26, 2013 at 1:02 PM, Michael Della Bitta <
michael.della.bitta@appinions.com> wrote:
> Nate,
>
> We just cleared up a problem similar to this by ditching Elastic Load
> Balancer and switching over to the APR connector in Tomcat. Are you
> using either of those?
>
> Michael Della Bitta
>
> ------------------------------------------------
> Appinions
> 18 East 41st Street, 2nd Floor
> New York, NY 10017-6271
>
> www.appinions.com
>
> Where Influence Isn’t a Game
>
>
> On Tue, Mar 26, 2013 at 2:58 PM, Otis Gospodnetic
> <ot...@gmail.com> wrote:
> > Hi Nate,
> >
> > Try adding some warmup queries and making sure the setting for using
> > the cold searcher in solrconfig.xml is set to false. Your warmup
> > queries should use facets and sorting if your normal queries use them.
> > In SPM you'll actually see how much time warming up takes, so you'll
> > get a better idea of the "cost" of that (when you don't do it).
> >
> > Otis
> > --
> > Solr & ElasticSearch Support
> > http://sematext.com/
> >
> >
> >
> >
> >
> > On Tue, Mar 26, 2013 at 2:50 PM, Nate Fox <nf...@neogov.com> wrote:
> >> I was wondering if the warmup stuff was one of the culprits (we dont
> have
> >> warmup's at all - the configs are pretty stock).
> >> As for the system, it seems capable of quite a bit more: memory usage is
> >> ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb)
> and
> >> load below 1.00.
> >>
> >> The seed data and queries were put together by one of our developers.
> I've
> >> put all the solrmeter files here:
> >> https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce
> >> Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely
> sure
> >> which file does which specifically.
> >>
> >> Does the system's reaction to a 'fast load' without a warmup sound
> normal?
> >> I would have expected the first couple hundred queries to be very slow
> >> (>500ms) and then the system catch up after a while. But it just dies
> very
> >> quickly and never recovers.
> >>
> >> I'll check out your SPM - I've seen it mentioned before. Thanks!
> >>
> >>
> >>
> >> --
> >> Nate Fox
> >> Sr Systems Engineer
> >>
> >> o: 310.658.5775
> >> m: 714.248.5350
> >>
> >> Follow us @NEOGOV <http://twitter.com/NEOGOV> and on
> >> Facebook<http://www.facebook.com/neogov>
> >>
> >> NEOGOV <http://www.neogov.com/> is among the top fastest growing
> software
> >> companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and
> >> the LA Business Journal. We are hiring!<
> http://www.neogov.com/#/company/careers>
> >>
> >>
> >>
> >> On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic <
> >> otis.gospodnetic@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> In short, certain data structures need to load from index in the
> >>> beginning, (for sorting and faceting) caches need to warm up, JVM
> >>> needs to warm up, etc., so going slowly in the beginning makes sense.
> >>> Why things die after that is a different Q. Maybe it OOMs? Maybe
> >>> queries are very complex? What do your queries look like? I see
> >>> newrelic.jar in the command-line. May want to try SPM for Solr, it
> >>> has better Solr metrics.
> >>>
> >>> Otis
> >>> --
> >>> Solr & ElasticSearch Support
> >>> http://sematext.com/
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox <nf...@neogov.com> wrote:
> >>> > I'm new to solr and I'm load testing our setup to see what we can
> handle.
> >>> > I'm using solrmeter and my problem is a bit odd:
> >>> > * When I set solrmeter to run 4000 queries/min, it will handle a few
> >>> > hundred queries and then tomcat will stop responding completely to
> >>> requests
> >>> > (even though according to lsof -i it is still listening and the java
> >>> > process is still running).
> >>> > * When I set solrmeter to run 1000 queries/min it runs fine. I can
> stop
> >>> > solrmeter after a couple of minutes at that pace and then run at
> >>> 4000/min
> >>> > without issue.
> >>> >
> >>> > It's as if it needs a ramp up time? Also, I noticed (regardless of
> ramp
> >>> up)
> >>> > that my setup cannot handle 8000/min. The reaction at 8k/min is the
> same
> >>> as
> >>> > if I were to run 4k/min without the ramp up. Of note, only the shard
> that
> >>> > solrmeter is pointed to stops responding. The other shard hums along
> >>> > without incident.
> >>> >
> >>> > Setup (everything in AWS):
> >>> > - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0
> >>> > (open-jdk-7-headless) : Ubuntu 12.04
> >>> > - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04
> >>> > I have ~30k documents in each node (~300Mb on each node)
> >>> >
> >>> > The vast majority of my solr/tomcat7 config is default from ubuntu's
> >>> > packages/solr's example dir. Here's the configs and the end of the
> >>> > catalina.out file:
> https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0
> >>> >
> >>> > My main question is two fold:
> >>> > 1. Is this normal behavior for tomcat (to just stop responding
> >>> completely)
> >>> > when it gets overwhelmed? And the only option is to restart it? I
> guess I
> >>> > dont know what it looks like when tomcat/solr cant keep up.
> >>> > 2. Why does it handle better when I give it a lower number of
> queries and
> >>> > then ramp it up? It concerns me that if I have to restart a server
> in the
> >>> > cluster and it gets thrown into the pool of machines that things will
> >>> blow
> >>> > up.
> >>> >
> >>> > As an aside, does this seem like a normal amount of queries (~4k/min)
> >>> that
> >>> > this kind of environment should be able to handle?
> >>>
>
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Posted by Michael Della Bitta <mi...@appinions.com>.
Nate,
We just cleared up a problem similar to this by ditching Elastic Load
Balancer and switching over to the APR connector in Tomcat. Are you
using either of those?
Michael Della Bitta
------------------------------------------------
Appinions
18 East 41st Street, 2nd Floor
New York, NY 10017-6271
www.appinions.com
Where Influence Isn’t a Game
On Tue, Mar 26, 2013 at 2:58 PM, Otis Gospodnetic
<ot...@gmail.com> wrote:
> Hi Nate,
>
> Try adding some warmup queries and making sure the setting for using
> the cold searcher in solrconfig.xml is set to false. Your warmup
> queries should use facets and sorting if your normal queries use them.
> In SPM you'll actually see how much time warming up takes, so you'll
> get a better idea of the "cost" of that (when you don't do it).
>
> Otis
> --
> Solr & ElasticSearch Support
> http://sematext.com/
>
>
>
>
>
> On Tue, Mar 26, 2013 at 2:50 PM, Nate Fox <nf...@neogov.com> wrote:
>> I was wondering if the warmup stuff was one of the culprits (we dont have
>> warmup's at all - the configs are pretty stock).
>> As for the system, it seems capable of quite a bit more: memory usage is
>> ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb) and
>> load below 1.00.
>>
>> The seed data and queries were put together by one of our developers. I've
>> put all the solrmeter files here:
>> https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce
>> Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely sure
>> which file does which specifically.
>>
>> Does the system's reaction to a 'fast load' without a warmup sound normal?
>> I would have expected the first couple hundred queries to be very slow
>> (>500ms) and then the system catch up after a while. But it just dies very
>> quickly and never recovers.
>>
>> I'll check out your SPM - I've seen it mentioned before. Thanks!
>>
>>
>>
>> --
>> Nate Fox
>> Sr Systems Engineer
>>
>> o: 310.658.5775
>> m: 714.248.5350
>>
>> Follow us @NEOGOV <http://twitter.com/NEOGOV> and on
>> Facebook<http://www.facebook.com/neogov>
>>
>> NEOGOV <http://www.neogov.com/> is among the top fastest growing software
>> companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and
>> the LA Business Journal. We are hiring!<http://www.neogov.com/#/company/careers>
>>
>>
>>
>> On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic <
>> otis.gospodnetic@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> In short, certain data structures need to load from index in the
>>> beginning, (for sorting and faceting) caches need to warm up, JVM
>>> needs to warm up, etc., so going slowly in the beginning makes sense.
>>> Why things die after that is a different Q. Maybe it OOMs? Maybe
>>> queries are very complex? What do your queries look like? I see
>>> newrelic.jar in the command-line. May want to try SPM for Solr, it
>>> has better Solr metrics.
>>>
>>> Otis
>>> --
>>> Solr & ElasticSearch Support
>>> http://sematext.com/
>>>
>>>
>>>
>>>
>>>
>>> On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox <nf...@neogov.com> wrote:
>>> > I'm new to solr and I'm load testing our setup to see what we can handle.
>>> > I'm using solrmeter and my problem is a bit odd:
>>> > * When I set solrmeter to run 4000 queries/min, it will handle a few
>>> > hundred queries and then tomcat will stop responding completely to
>>> requests
>>> > (even though according to lsof -i it is still listening and the java
>>> > process is still running).
>>> > * When I set solrmeter to run 1000 queries/min it runs fine. I can stop
>>> > solrmeter after a couple of minutes at that pace and then run at
>>> 4000/min
>>> > without issue.
>>> >
>>> > It's as if it needs a ramp up time? Also, I noticed (regardless of ramp
>>> up)
>>> > that my setup cannot handle 8000/min. The reaction at 8k/min is the same
>>> as
>>> > if I were to run 4k/min without the ramp up. Of note, only the shard that
>>> > solrmeter is pointed to stops responding. The other shard hums along
>>> > without incident.
>>> >
>>> > Setup (everything in AWS):
>>> > - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0
>>> > (open-jdk-7-headless) : Ubuntu 12.04
>>> > - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04
>>> > I have ~30k documents in each node (~300Mb on each node)
>>> >
>>> > The vast majority of my solr/tomcat7 config is default from ubuntu's
>>> > packages/solr's example dir. Here's the configs and the end of the
>>> > catalina.out file:https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0
>>> >
>>> > My main question is two fold:
>>> > 1. Is this normal behavior for tomcat (to just stop responding
>>> completely)
>>> > when it gets overwhelmed? And the only option is to restart it? I guess I
>>> > dont know what it looks like when tomcat/solr cant keep up.
>>> > 2. Why does it handle better when I give it a lower number of queries and
>>> > then ramp it up? It concerns me that if I have to restart a server in the
>>> > cluster and it gets thrown into the pool of machines that things will
>>> blow
>>> > up.
>>> >
>>> > As an aside, does this seem like a normal amount of queries (~4k/min)
>>> that
>>> > this kind of environment should be able to handle?
>>>
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi Nate,
Try adding some warmup queries and making sure the setting for using
the cold searcher in solrconfig.xml is set to false. Your warmup
queries should use facets and sorting if your normal queries use them.
In SPM you'll actually see how much time warming up takes, so you'll
get a better idea of the "cost" of that (when you don't do it).
Otis
--
Solr & ElasticSearch Support
http://sematext.com/
On Tue, Mar 26, 2013 at 2:50 PM, Nate Fox <nf...@neogov.com> wrote:
> I was wondering if the warmup stuff was one of the culprits (we dont have
> warmup's at all - the configs are pretty stock).
> As for the system, it seems capable of quite a bit more: memory usage is
> ~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb) and
> load below 1.00.
>
> The seed data and queries were put together by one of our developers. I've
> put all the solrmeter files here:
> https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce
> Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely sure
> which file does which specifically.
>
> Does the system's reaction to a 'fast load' without a warmup sound normal?
> I would have expected the first couple hundred queries to be very slow
> (>500ms) and then the system catch up after a while. But it just dies very
> quickly and never recovers.
>
> I'll check out your SPM - I've seen it mentioned before. Thanks!
>
>
>
> --
> Nate Fox
> Sr Systems Engineer
>
> o: 310.658.5775
> m: 714.248.5350
>
> Follow us @NEOGOV <http://twitter.com/NEOGOV> and on
> Facebook<http://www.facebook.com/neogov>
>
> NEOGOV <http://www.neogov.com/> is among the top fastest growing software
> companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and
> the LA Business Journal. We are hiring!<http://www.neogov.com/#/company/careers>
>
>
>
> On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
>> Hi,
>>
>> In short, certain data structures need to load from index in the
>> beginning, (for sorting and faceting) caches need to warm up, JVM
>> needs to warm up, etc., so going slowly in the beginning makes sense.
>> Why things die after that is a different Q. Maybe it OOMs? Maybe
>> queries are very complex? What do your queries look like? I see
>> newrelic.jar in the command-line. May want to try SPM for Solr, it
>> has better Solr metrics.
>>
>> Otis
>> --
>> Solr & ElasticSearch Support
>> http://sematext.com/
>>
>>
>>
>>
>>
>> On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox <nf...@neogov.com> wrote:
>> > I'm new to solr and I'm load testing our setup to see what we can handle.
>> > I'm using solrmeter and my problem is a bit odd:
>> > * When I set solrmeter to run 4000 queries/min, it will handle a few
>> > hundred queries and then tomcat will stop responding completely to
>> requests
>> > (even though according to lsof -i it is still listening and the java
>> > process is still running).
>> > * When I set solrmeter to run 1000 queries/min it runs fine. I can stop
>> > solrmeter after a couple of minutes at that pace and then run at
>> 4000/min
>> > without issue.
>> >
>> > It's as if it needs a ramp up time? Also, I noticed (regardless of ramp
>> up)
>> > that my setup cannot handle 8000/min. The reaction at 8k/min is the same
>> as
>> > if I were to run 4k/min without the ramp up. Of note, only the shard that
>> > solrmeter is pointed to stops responding. The other shard hums along
>> > without incident.
>> >
>> > Setup (everything in AWS):
>> > - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0
>> > (open-jdk-7-headless) : Ubuntu 12.04
>> > - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04
>> > I have ~30k documents in each node (~300Mb on each node)
>> >
>> > The vast majority of my solr/tomcat7 config is default from ubuntu's
>> > packages/solr's example dir. Here's the configs and the end of the
>> > catalina.out file:https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0
>> >
>> > My main question is two fold:
>> > 1. Is this normal behavior for tomcat (to just stop responding
>> completely)
>> > when it gets overwhelmed? And the only option is to restart it? I guess I
>> > dont know what it looks like when tomcat/solr cant keep up.
>> > 2. Why does it handle better when I give it a lower number of queries and
>> > then ramp it up? It concerns me that if I have to restart a server in the
>> > cluster and it gets thrown into the pool of machines that things will
>> blow
>> > up.
>> >
>> > As an aside, does this seem like a normal amount of queries (~4k/min)
>> that
>> > this kind of environment should be able to handle?
>>
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Posted by Nate Fox <nf...@neogov.com>.
I was wondering if the warmup stuff was one of the culprits (we dont have
warmup's at all - the configs are pretty stock).
As for the system, it seems capable of quite a bit more: memory usage is
~30%, jvm-memory (from the dashboard) is very low (~220Mb out of 3Gb) and
load below 1.00.
The seed data and queries were put together by one of our developers. I've
put all the solrmeter files here:
https://gist.github.com/natefox/ee5cef3d4fbbc73e9bce
Unfortunately I'm quite new to solr (and tomcat) so I'm not entirely sure
which file does which specifically.
Does the system's reaction to a 'fast load' without a warmup sound normal?
I would have expected the first couple hundred queries to be very slow
(>500ms) and then the system catch up after a while. But it just dies very
quickly and never recovers.
I'll check out your SPM - I've seen it mentioned before. Thanks!
--
Nate Fox
Sr Systems Engineer
o: 310.658.5775
m: 714.248.5350
Follow us @NEOGOV <http://twitter.com/NEOGOV> and on
Facebook<http://www.facebook.com/neogov>
NEOGOV <http://www.neogov.com/> is among the top fastest growing software
companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and
the LA Business Journal. We are hiring!<http://www.neogov.com/#/company/careers>
On Tue, Mar 26, 2013 at 11:12 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Hi,
>
> In short, certain data structures need to load from index in the
> beginning, (for sorting and faceting) caches need to warm up, JVM
> needs to warm up, etc., so going slowly in the beginning makes sense.
> Why things die after that is a different Q. Maybe it OOMs? Maybe
> queries are very complex? What do your queries look like? I see
> newrelic.jar in the command-line. May want to try SPM for Solr, it
> has better Solr metrics.
>
> Otis
> --
> Solr & ElasticSearch Support
> http://sematext.com/
>
>
>
>
>
> On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox <nf...@neogov.com> wrote:
> > I'm new to solr and I'm load testing our setup to see what we can handle.
> > I'm using solrmeter and my problem is a bit odd:
> > * When I set solrmeter to run 4000 queries/min, it will handle a few
> > hundred queries and then tomcat will stop responding completely to
> requests
> > (even though according to lsof -i it is still listening and the java
> > process is still running).
> > * When I set solrmeter to run 1000 queries/min it runs fine. I can stop
> > solrmeter after a couple of minutes at that pace and then run at
> 4000/min
> > without issue.
> >
> > It's as if it needs a ramp up time? Also, I noticed (regardless of ramp
> up)
> > that my setup cannot handle 8000/min. The reaction at 8k/min is the same
> as
> > if I were to run 4k/min without the ramp up. Of note, only the shard that
> > solrmeter is pointed to stops responding. The other shard hums along
> > without incident.
> >
> > Setup (everything in AWS):
> > - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0
> > (open-jdk-7-headless) : Ubuntu 12.04
> > - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04
> > I have ~30k documents in each node (~300Mb on each node)
> >
> > The vast majority of my solr/tomcat7 config is default from ubuntu's
> > packages/solr's example dir. Here's the configs and the end of the
> > catalina.out file:https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0
> >
> > My main question is two fold:
> > 1. Is this normal behavior for tomcat (to just stop responding
> completely)
> > when it gets overwhelmed? And the only option is to restart it? I guess I
> > dont know what it looks like when tomcat/solr cant keep up.
> > 2. Why does it handle better when I give it a lower number of queries and
> > then ramp it up? It concerns me that if I have to restart a server in the
> > cluster and it gets thrown into the pool of machines that things will
> blow
> > up.
> >
> > As an aside, does this seem like a normal amount of queries (~4k/min)
> that
> > this kind of environment should be able to handle?
>
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
In short, certain data structures need to load from index in the
beginning, (for sorting and faceting) caches need to warm up, JVM
needs to warm up, etc., so going slowly in the beginning makes sense.
Why things die after that is a different Q. Maybe it OOMs? Maybe
queries are very complex? What do your queries look like? I see
newrelic.jar in the command-line. May want to try SPM for Solr, it
has better Solr metrics.
Otis
--
Solr & ElasticSearch Support
http://sematext.com/
On Tue, Mar 26, 2013 at 1:24 PM, Nate Fox <nf...@neogov.com> wrote:
> I'm new to solr and I'm load testing our setup to see what we can handle.
> I'm using solrmeter and my problem is a bit odd:
> * When I set solrmeter to run 4000 queries/min, it will handle a few
> hundred queries and then tomcat will stop responding completely to requests
> (even though according to lsof -i it is still listening and the java
> process is still running).
> * When I set solrmeter to run 1000 queries/min it runs fine. I can stop
> solrmeter after a couple of minutes at that pace and then run at 4000/min
> without issue.
>
> It's as if it needs a ramp up time? Also, I noticed (regardless of ramp up)
> that my setup cannot handle 8000/min. The reaction at 8k/min is the same as
> if I were to run 4k/min without the ramp up. Of note, only the shard that
> solrmeter is pointed to stops responding. The other shard hums along
> without incident.
>
> Setup (everything in AWS):
> - 2x m1.large (7.5Gb RAM) running tomcat7 + solr 4.2.0
> (open-jdk-7-headless) : Ubuntu 12.04
> - 1x m1.micro running zookeeper 3.4.5 : Ubuntu 12.04
> I have ~30k documents in each node (~300Mb on each node)
>
> The vast majority of my solr/tomcat7 config is default from ubuntu's
> packages/solr's example dir. Here's the configs and the end of the
> catalina.out file:https://gist.github.com/anonymous/ef8fa79ecc1673d11bc0
>
> My main question is two fold:
> 1. Is this normal behavior for tomcat (to just stop responding completely)
> when it gets overwhelmed? And the only option is to restart it? I guess I
> dont know what it looks like when tomcat/solr cant keep up.
> 2. Why does it handle better when I give it a lower number of queries and
> then ramp it up? It concerns me that if I have to restart a server in the
> cluster and it gets thrown into the pool of machines that things will blow
> up.
>
> As an aside, does this seem like a normal amount of queries (~4k/min) that
> this kind of environment should be able to handle?
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Posted by Mark Miller <ma...@gmail.com>.
On Mar 27, 2013, at 3:29 PM, Shawn Heisey <so...@elyograg.org> wrote:
> The maxThreads parameter in the jetty config that's included with Solr is set to 10000
Yonik raised this at some point if I remember right - it helps avoid some distrib deadlock issue.
- Mark
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Posted by Shawn Heisey <so...@elyograg.org>.
On 3/27/2013 1:16 PM, Nate Fox wrote:
> I have a couple quick followup questions:
> - What is the downside of having a maxThreads, acceptCount and
> maxConnections really high? Obviously defaults are there for a reason - I'd
> like to know what the reasoning is.
> - Any reason I shouldnt use Tomcat? I just went with it because I figured
> it was extremely mature and was easy to use with apt-get :)
The maxThreads parameter in the jetty config that's included with Solr
is set to 10000 - this is the value chosen by Solr's development team.
Your setting of 2500 should be perfectly fine, and it is definitely not
really high. The default of 200 in your distribution is very low.
Tomcat is certainly a viable solution, one used by many. It is very
mature and has proven itself. The really nice thing with using an
OS-packaged version is that you don't have to write or change the init
script. I use the jetty that was included with Solr, and had to write
my own init script.
Jetty, especially the stripped-down version included with Solr, has a
smaller footprint than tomcat. The bells and whistles are not required.
It is not better or worse than tomcat, just another choice.
Thanks,
Shawn
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Posted by Nate Fox <nf...@neogov.com>.
Update: issue resolved!
Cranking up the maxThreads did the trick. Default is 200. I went with 2500
for grins and giggles and things work great. Now, even if I overwhelm the
box with too many requests, when the requests back off the box continues to
respond. And when I slam the server after it's been restarted (without
having warmup queries), it acts as I wanted: queries are slow to respond
(upwards of 30s) for the first couple minutes then they start to all be
under 25ms and normalize at a very fast pace (obviously as the cache is
warmed).
Christopher, I could have sworn I tried upping acceptCount, maxConnections
and maxThreads in my testing, but with your prodding I tried it again - and
that was the solution.
I have a couple quick followup questions:
- What is the downside of having a maxThreads, acceptCount and
maxConnections really high? Obviously defaults are there for a reason - I'd
like to know what the reasoning is.
- Any reason I shouldnt use Tomcat? I just went with it because I figured
it was extremely mature and was easy to use with apt-get :)
I'll probably toy with the APR as suggested by Michael, as I like the idea
of a non-blocking connector.
--
Nate Fox
Sr Systems Engineer
o: 310.658.5775
m: 714.248.5350
Follow us @NEOGOV <http://twitter.com/NEOGOV> and on
Facebook<http://www.facebook.com/neogov>
NEOGOV <http://www.neogov.com/> is among the top fastest growing software
companies in the USA, recognized by Inc 500|5000, Deloitte Fast 500, and
the LA Business Journal. We are hiring!<http://www.neogov.com/#/company/careers>
On Tue, Mar 26, 2013 at 5:56 PM, Chris Hostetter
<ho...@fucit.org>wrote:
>
> : * When I set solrmeter to run 4000 queries/min, it will handle a few
> : hundred queries and then tomcat will stop responding completely to
> requests
> : (even though according to lsof -i it is still listening and the java
> : process is still running).
>
> have you tried tacking using jstack to generate a thread dump of the
> server to see what it's doing?
>
> : * When I set solrmeter to run 1000 queries/min it runs fine. I can stop
> : solrmeter after a couple of minutes at that pace and then run at
> 4000/min
> : without issue.
> :
> : It's as if it needs a ramp up time? Also, I noticed (regardless of ramp
> up)
> : that my setup cannot handle 8000/min. The reaction at 8k/min is the same
> as
> : if I were to run 4k/min without the ramp up. Of note, only the shard that
> : solrmeter is pointed to stops responding. The other shard hums along
> : without incident.
>
> Just to clarify: you're running a 2 node SolrCloud cluster, where each
> node contains a unique shard, and pointing solrmeter at a single node for
> the queries -- correct?
>
> Here's my hunch: you are probably hitting the limit of the number of
> concurrent connections tomcat will allow (whatever it may be confiurged
> ot in your setup).
>
> In the 8000/min case, you are probably maxing out that limit with direct
> connections you issue from solrmeter to that single node.
>
> In the 4000/min case, each request you issue causes that single node to
> fire off multiple requests to each shard, and since each shard exists on
> only one node, you are garunteeing thta you double the number of
> concurrent requests hitting that first node.
>
> in the case where you start w/ 1000/min, and then later ramp up to
> 4000/min, you are probably causing enough of the queries to be warmed up
> that they are in the caches on both nodes, so they can be served really
> fast and return their results before you reach that max number of
> concurrent connections after you ramp up.
>
> I'm no tomcat expert, but skimming hte docs, you may want to look at
> settings like acceptCount, maxConnections, maxThreads, etc...
>
> -Hoss
>
Re: Loadtesting solr/tomcat7 and tomcat stops responding entirely
Posted by Chris Hostetter <ho...@fucit.org>.
: * When I set solrmeter to run 4000 queries/min, it will handle a few
: hundred queries and then tomcat will stop responding completely to requests
: (even though according to lsof -i it is still listening and the java
: process is still running).
have you tried tacking using jstack to generate a thread dump of the
server to see what it's doing?
: * When I set solrmeter to run 1000 queries/min it runs fine. I can stop
: solrmeter after a couple of minutes at that pace and then run at 4000/min
: without issue.
:
: It's as if it needs a ramp up time? Also, I noticed (regardless of ramp up)
: that my setup cannot handle 8000/min. The reaction at 8k/min is the same as
: if I were to run 4k/min without the ramp up. Of note, only the shard that
: solrmeter is pointed to stops responding. The other shard hums along
: without incident.
Just to clarify: you're running a 2 node SolrCloud cluster, where each
node contains a unique shard, and pointing solrmeter at a single node for
the queries -- correct?
Here's my hunch: you are probably hitting the limit of the number of
concurrent connections tomcat will allow (whatever it may be confiurged
ot in your setup).
In the 8000/min case, you are probably maxing out that limit with direct
connections you issue from solrmeter to that single node.
In the 4000/min case, each request you issue causes that single node to
fire off multiple requests to each shard, and since each shard exists on
only one node, you are garunteeing thta you double the number of
concurrent requests hitting that first node.
in the case where you start w/ 1000/min, and then later ramp up to
4000/min, you are probably causing enough of the queries to be warmed up
that they are in the caches on both nodes, so they can be served really
fast and return their results before you reach that max number of
concurrent connections after you ramp up.
I'm no tomcat expert, but skimming hte docs, you may want to look at
settings like acceptCount, maxConnections, maxThreads, etc...
-Hoss