SOLR Performance benchmarking

[rashi gandhi <ga...@gmail.com>]

Hi,

I am using SolrMeter for load/stress testing solr performance.
Tomcat is configured with default "maxThreads" (i.e. 200).

I set Intended Request per min in SolrMeter to 1500 and performed testing.

I found that sometimes it works with this much load on solr but sometimes
it gives error "Sever Refused Connection" in solr.
On getting this error, i increased maxThreads to some higher value, and
then it works again.

I would like to know why solr is behaving abnormally, initially when it was
working with maxThreads=200.

Please provide me some pointers.

Re: SOLR Performance benchmarking

[Umesh Prasad <um...@gmail.com>]

Hi Rashi,
Also, checkout
http://searchhub.org/2010/01/21/the-seven-deadly-sins-of-solr/ ..
It would help if you can share your solrconfig.xml and schema.xml .. Some
problems are evident from there itself.

     From our experience we have found
1. JVM Heap size (check for young gen size and new/old ratio. Default is
very low for Prod setups)
2. Solr cache tuning as Siegfried pointed out. There are 4 cache
queryCache, filterCache , documentCache and FieldValueCache. Make sure that
you have the caches populated to by defining a newSearcher and
autowarmCount is properly configured.
3. About long running queries, solr core logs are your friend, analyze the
QTime percentiles.
    The list of reasons here is big. Two that we have found killers for
performance are
     a) . A query time analyzer chain of synonym filter -->  stemmer -->
synonym filter had resulted in like 50 * 50 = 2500 terms for a single term
for us
     b) ngroups and groups.truncate are quite costly, specially if you have
large cardinality for field. And these aren't cached.
     c) Faceting/filtering on timestamp fields (with arbitrary accuracy)
     d) Deep paging




On 13 July 2014 14:48, Siegfried Goeschl <sg...@gmx.at> wrote:

> Hi Rashi,
>
> abnormal behaviour depends on your data, system and work load - I have
> seen abnormal behaviour at customers sites and it turned out to be a
> miracle that they the customer had no serious problems before :-)
>
> * running out of sockets - you might need to check if you have enough
> sockets (system quota) and that the sockets are closed properly (mostly a
> Windows/networking issue - CLOSED_WAIT)
> * understand your test setup - usually a test box is much smaller in terms
> of CPU/memory than you production box
> ** you might be forced to tweak your test configuration (e.g. production
> SOLR cache configuration can overwhelm a small server)
> * understand your work-load
> ** if you have long-running queries within your performance tests they
> tend to bring down your server under high-load and your “abnormal”
> condition looks very normal at hindsight
> ** spot your long-running queries, optimise them, re-run your tests
> ** check your cache warming and how fast you start your load injector
> threads
>
> Cheers,
>
> Siegfried Goeschl
>
>
> On 13 Jul 2014, at 09:53, rashi gandhi <ga...@gmail.com> wrote:
>
> > Hi,
> >
> > I am using SolrMeter for load/stress testing solr performance.
> > Tomcat is configured with default "maxThreads" (i.e. 200).
> >
> > I set Intended Request per min in SolrMeter to 1500 and performed
> testing.
> >
> > I found that sometimes it works with this much load on solr but sometimes
> > it gives error "Sever Refused Connection" in solr.
> > On getting this error, i increased maxThreads to some higher value, and
> > then it works again.
> >
> > I would like to know why solr is behaving abnormally, initially when it
> was
> > working with maxThreads=200.
> >
> > Please provide me some pointers.
>
>


-- 
---
Thanks & Regards
Umesh Prasad

Re: SOLR Performance benchmarking

[Siegfried Goeschl <sg...@gmx.at>]

Hi Rashi,

abnormal behaviour depends on your data, system and work load - I have seen abnormal behaviour at customers sites and it turned out to be a miracle that they the customer had no serious problems before :-)

* running out of sockets - you might need to check if you have enough sockets (system quota) and that the sockets are closed properly (mostly a Windows/networking issue - CLOSED_WAIT)
* understand your test setup - usually a test box is much smaller in terms of CPU/memory than you production box
** you might be forced to tweak your test configuration (e.g. production SOLR cache configuration can overwhelm a small server)
* understand your work-load 
** if you have long-running queries within your performance tests they tend to bring down your server under high-load and your “abnormal” condition looks very normal at hindsight 
** spot your long-running queries, optimise them, re-run your tests
** check your cache warming and how fast you start your load injector threads

Cheers,

Siegfried Goeschl


On 13 Jul 2014, at 09:53, rashi gandhi <ga...@gmail.com> wrote:

> Hi,
> 
> I am using SolrMeter for load/stress testing solr performance.
> Tomcat is configured with default "maxThreads" (i.e. 200).
> 
> I set Intended Request per min in SolrMeter to 1500 and performed testing.
> 
> I found that sometimes it works with this much load on solr but sometimes
> it gives error "Sever Refused Connection" in solr.
> On getting this error, i increased maxThreads to some higher value, and
> then it works again.
> 
> I would like to know why solr is behaving abnormally, initially when it was
> working with maxThreads=200.
> 
> Please provide me some pointers.