Re: how to improve concurrent request performance and stress testing

[zqzuk <zi...@hotmail.com>]

Hi, I see there's no response to my question, maybe its better to ask this
way...

If solr receives 10 concurrent request, does it deal with the 10 requests
simultaneously using 10 (or as many as possible) searchers, or does it deal
with each request sequentially, which implies that first request will be
served first, and in the worse case, the last request may have to wait for a
long time until all preceding requests have been answered?

Thanks




zqzuk wrote:
> 
> Hi, I am doing a stress testing of my solr application to see how many
> concurrent requests it can handle and how long it takes. But I m not sure
> if I have done it in proper way... responses seem to be very slow
> 
> My configuration:
> 1 Solr instance, using the default settings distributed in the example
> code, while I made two changes:
> <useColdSearcher>true</useColdSearcher>
> <maxWarmingSearchers>10</maxWarmingSearchers>
> As I thought the more searchers the more concurrent requests can be dealt
> with?
> 
> There are 1.1 million documents indexed, and the platform is winxp sp2,
> duo core 1.8 GB machine with ram 2GB
> 
> I used httpstone, a simple server load testing tool to create 100 workers
> (so 100 threads) each issuing one same query to the server. To deal with a
> single request of this query it took solr 2 seconds (with facet counts),
> and 7 documents are returned. I was assuming that only first request would
> take longer time and following requests should be almost instantaneous as
> the query is the same. But strange that the first response took as long as
> 20 seconds.
> 
> It looked like that the 100 workers sent same request to solr and then all
> of a sudden solr server went silent. Only after 20 seconds some of these
> workers started to receive responses, but still very slow.
> 
> clearly there I must have made something wrong with configuring solr
> server... Could you give me some pointers on how to improve the
> performance please?
> 
> Many thanks!
> 

-- 
View this message in context: http://www.nabble.com/how-to-improve-concurrent-request-performance-and-stress-testing-tp15299687p15306531.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: how to improve concurrent request performance and stress testing

[Chris Hostetter <ho...@fucit.org>]

: Thank you so much! I will look into firstSearcher configuration next! thanks

FYI: prompted by this thread, I added some blurbs about firstSearcher, 
newSearcher, and FieldCache to the SolrCaching wiki ... as a new users 
learning about this stuff, please fele free to update that wiki with any 
improvements you can think of...

	http://wiki.apache.org/solr/SolrCaching

(the best people to write documentation are new users who (unlike the 
developers) don't have intrinsic knowledge of the subject in the back of 
their mind that they take for granted).


-Hoss

Re: how to improve concurrent request performance and stress testing

[Ziqi Zhang <zi...@hotmail.com>]

Thank you so much! I will look into firstSearcher configuration next! thanks

--------------------------------------------------
From: "Chris Hostetter" <ho...@fucit.org>
Sent: Wednesday, February 06, 2008 8:56 PM
To: <so...@lucene.apache.org>
Subject: Re: how to improve concurrent request performance and stress 
testing

> : > Also make sure that common filters, sort fields, and facets have been
> : > warmed.
> :
> : I assume these are achieved by setting large cache size and large
> : autowarmcount number in solr configuration? specifically
>
> autowarming seeds the cahces of a new Searcher using hte keys of an old
> searcher -- it does nothing to help you when you first start up SOlr and
> all of hte caches are empty.
>
> for that you need to either need to manually trigger some sample queries
> externally (before your stress test) or configure something using the
> firstSearcher event listener in solrconfig.xml.
>
> If you saw all of your requests block untill the first one finished, then
> i suspect your queries involve a sort (or faceting) that use the
> FieldCache which is initialized in a single threaded mode (and can't be
> auto-warmed, you can put some simple queries that use those sort fields in
> the newSearcher listener to ensure that they get reinitialized for each
> new searcher)
>
> -Hoss
>
> 

Re: how to improve concurrent request performance and stress testing

[Chris Hostetter <ho...@fucit.org>]

: > Also make sure that common filters, sort fields, and facets have been
: > warmed.
: 
: I assume these are achieved by setting large cache size and large
: autowarmcount number in solr configuration? specifically

autowarming seeds the cahces of a new Searcher using hte keys of an old 
searcher -- it does nothing to help you when you first start up SOlr and 
all of hte caches are empty.

for that you need to either need to manually trigger some sample queries 
externally (before your stress test) or configure something using the 
firstSearcher event listener in solrconfig.xml.

If you saw all of your requests block untill the first one finished, then 
i suspect your queries involve a sort (or faceting) that use the 
FieldCache which is initialized in a single threaded mode (and can't be 
auto-warmed, you can put some simple queries that use those sort fields in 
the newSearcher listener to ensure that they get reinitialized for each 
new searcher)

-Hoss

Re: how to improve concurrent request performance and stress testing

[Ziqi Zhang <zi...@hotmail.com>]

Thanks!

> Also make sure that common filters, sort fields, and facets have been 
> warmed.

I assume these are achieved by setting large cache size and large 
autowarmcount number in solr configuration? specifically

filterCache
queryResultCache
documentCache

Thanks!


--------------------------------------------------
From: "Yonik Seeley" <yo...@apache.org>
Sent: Wednesday, February 06, 2008 7:50 AM
To: <so...@lucene.apache.org>
Subject: Re: how to improve concurrent request performance and stress 
testing

> On Feb 6, 2008 6:37 PM, Ziqi Zhang <zi...@hotmail.com> wrote:
>> I still do not understand why sending 100 request (of same query) from 
>> 100
>> threads throws solr server to silence - is it because of the 
>> computational
>> cost to deal with same query in 100 separate threads?
>
> Yes... sending a large number of requests at once can cause one to
> start hitting synchronization bottlenecks.

>
> -Yonik
> 

Re: how to improve concurrent request performance and stress testing

[Yonik Seeley <yo...@apache.org>]

On Feb 6, 2008 6:37 PM, Ziqi Zhang <zi...@hotmail.com> wrote:
> I still do not understand why sending 100 request (of same query) from 100
> threads throws solr server to silence - is it because of the computational
> cost to deal with same query in 100 separate threads?

Yes... sending a large number of requests at once can cause one to
start hitting synchronization bottlenecks.
Also make sure that common filters, sort fields, and facets have been warmed.

-Yonik

Re: how to improve concurrent request performance and stress testing

[Ziqi Zhang <zi...@hotmail.com>]

Thanks Yonik,


> It uses a thread per request, simultaneously (up to any limit
> configured by the app server)

How can I change this setting then? I suppose it is to do with Jetty or 
Tomcat whichever hosts solr application, not through the solrconfig?

I still do not understand why sending 100 request (of same query) from 100 
threads throws solr server to silence - is it because of the computational 
cost to deal with same query in 100 separate threads?

I noticed that disabling facet counts improves things a bit, but not 
significant.

Thanks in advance! 

Re: how to improve concurrent request performance and stress testing

[Yonik Seeley <yo...@apache.org>]

On Feb 6, 2008 7:53 AM, zqzuk <zi...@hotmail.com> wrote:
> If solr receives 10 concurrent request, does it deal with the 10 requests
> simultaneously

It uses a thread per request, simultaneously (up to any limit
configured by the app server)

> using 10 (or as many as possible) searchers

There is a single searcher serving all requests at a time.

-Yonik