You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by zqzuk <zi...@hotmail.com> on 2008/02/05 22:19:53 UTC
Solr search stress testing
Hi, I d like to do a stress testing of my solr application to see how many
concurrent requests it can handle and how long it takes. But I m not sure if
I have done it in proper way (likely not)... responses seem to be very slow
My configuration:
1 Solr instance, using the default settings distributed in the example code
1.1 million documents indexed
duo core 1.8 GB machine with ram 2GB
My testing approach:
I created 100 Threads, each of which producings exactly same one query to
Solr, and the query should return 7 results. The wait interval bewteen each
thread is 1 second. In other words, every 1 second there is a thread created
issuing same query to solr server.
The time taken to serve 1 request is 2 seconds.
I was expecting that response to first thread take 2 seconds, then much
shorter for the rest, however, it seems results are very random. The last
thread took even 10 seconds to complete.
In another test I created the 100 threads instantaneously and each issuing
the same query instantaneously, and for a while Solr went silent, until
about 15-20 seconds it started to send responses...
I suspect if this is the proper way to do the stress testing? if not may I
have some pointers please? If the test design makes sense, clearly there
must be something I can do to improve performance, may I have some pointers
too please?
Many thanks!
--
View this message in context: http://www.nabble.com/Solr-search-stress-testing-tp15299687p15299687.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to improve concurrent request performance and stress
testing
Posted by sunnyfr <jo...@gmail.com>.
Thanks for all this help,
But I guess it can't be optimal with a lot of update,
my slave get back from the master 20 000docs updated every 20minutes, it's
made to try to warmup have a big cache and everything go fast with that
amount of update I guess ...?
zqzuk wrote:
>
> Hi,
>
> try to firstly have a look at http://wiki.apache.org/solr/SolrCaching the
> section on firstsearcher and warming. Search engines rely on caching, so
> first searches will be slow. I think to be fair testing it is necessary to
> warm up the search engine by sending most frequently used and/or most
> costly queries, then start your stress testing.
>
> I used this tool http://code.google.com/p/httpstone/ to do stress testing.
> It allows you to create multiple threads sending queries to a server
> simultaneously, and records time taken to process each query in each
> thread.
>
> Hope it helps.
>
>
>
> sunnyfr wrote:
>>
>> Hi,
>>
>> I'm trying as well to stress test solr. I would love some advice to
>> manage it properly.
>> I'm using solr 1.3 and tomcat55.
>> Thanks a lot,
>>
>>
>> zqzuk wrote:
>>>
>>> Hi, I am doing a stress testing of my solr application to see how many
>>> concurrent requests it can handle and how long it takes. But I m not
>>> sure if I have done it in proper way... responses seem to be very slow
>>>
>>> My configuration:
>>> 1 Solr instance, using the default settings distributed in the example
>>> code, while I made two changes:
>>> <useColdSearcher>true</useColdSearcher>
>>> <maxWarmingSearchers>10</maxWarmingSearchers>
>>> As I thought the more searchers the more concurrent requests can be
>>> dealt with?
>>>
>>> There are 1.1 million documents indexed, and the platform is winxp sp2,
>>> duo core 1.8 GB machine with ram 2GB
>>>
>>> I used httpstone, a simple server load testing tool to create 100
>>> workers (so 100 threads) each issuing one same query to the server. To
>>> deal with a single request of this query it took solr 2 seconds (with
>>> facet counts), and 7 documents are returned. I was assuming that only
>>> first request would take longer time and following requests should be
>>> almost instantaneous as the query is the same. But strange that the
>>> first response took as long as 20 seconds.
>>>
>>> It looked like that the 100 workers sent same request to solr and then
>>> all of a sudden solr server went silent. Only after 20 seconds some of
>>> these workers started to receive responses, but still very slow.
>>>
>>> clearly there I must have made something wrong with configuring solr
>>> server... Could you give me some pointers on how to improve the
>>> performance please?
>>>
>>> Many thanks!
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/how-to-improve-concurrent-request-performance-and-stress-testing-tp15299687p22827717.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to improve concurrent request performance and stress
testing
Posted by sunnyfr <jo...@gmail.com>.
Dont get really with httpstone, I did my ant dist it works fine,
but then don't get ... when I do a java -jar I've an error:
~/httpstone-read-only/dist/lib$ java -jar httpstone.jar
Failed to load Main-Class manifest attribute from httpstone.jar
Any idea? Sorry I'm new in java
zqzuk wrote:
>
> Hi,
>
> try to firstly have a look at http://wiki.apache.org/solr/SolrCaching the
> section on firstsearcher and warming. Search engines rely on caching, so
> first searches will be slow. I think to be fair testing it is necessary to
> warm up the search engine by sending most frequently used and/or most
> costly queries, then start your stress testing.
>
> I used this tool http://code.google.com/p/httpstone/ to do stress testing.
> It allows you to create multiple threads sending queries to a server
> simultaneously, and records time taken to process each query in each
> thread.
>
> Hope it helps.
>
>
>
> sunnyfr wrote:
>>
>> Hi,
>>
>> I'm trying as well to stress test solr. I would love some advice to
>> manage it properly.
>> I'm using solr 1.3 and tomcat55.
>> Thanks a lot,
>>
>>
>> zqzuk wrote:
>>>
>>> Hi, I am doing a stress testing of my solr application to see how many
>>> concurrent requests it can handle and how long it takes. But I m not
>>> sure if I have done it in proper way... responses seem to be very slow
>>>
>>> My configuration:
>>> 1 Solr instance, using the default settings distributed in the example
>>> code, while I made two changes:
>>> <useColdSearcher>true</useColdSearcher>
>>> <maxWarmingSearchers>10</maxWarmingSearchers>
>>> As I thought the more searchers the more concurrent requests can be
>>> dealt with?
>>>
>>> There are 1.1 million documents indexed, and the platform is winxp sp2,
>>> duo core 1.8 GB machine with ram 2GB
>>>
>>> I used httpstone, a simple server load testing tool to create 100
>>> workers (so 100 threads) each issuing one same query to the server. To
>>> deal with a single request of this query it took solr 2 seconds (with
>>> facet counts), and 7 documents are returned. I was assuming that only
>>> first request would take longer time and following requests should be
>>> almost instantaneous as the query is the same. But strange that the
>>> first response took as long as 20 seconds.
>>>
>>> It looked like that the 100 workers sent same request to solr and then
>>> all of a sudden solr server went silent. Only after 20 seconds some of
>>> these workers started to receive responses, but still very slow.
>>>
>>> clearly there I must have made something wrong with configuring solr
>>> server... Could you give me some pointers on how to improve the
>>> performance please?
>>>
>>> Many thanks!
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/how-to-improve-concurrent-request-performance-and-stress-testing-tp15299687p20227934.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to improve concurrent request performance and stress
testing
Posted by zqzuk <zi...@hotmail.com>.
Yes, I created different workers and with same or different queries.
sorry it has been a while since I used it, and the link to its source code
(no jars) should be here:
http://code.google.com/p/httpstone/source/checkout
sunnyfr wrote:
>
> just a question about your httpstone's configuration ....?
> I would like to know how did you simulate several word search ... ??
> Did you create a lot of different workers with lof of different word
> search ?
>
> Thanks,
>
>
> zqzuk wrote:
>>
>> Hi,
>>
>> try to firstly have a look at http://wiki.apache.org/solr/SolrCaching the
>> section on firstsearcher and warming. Search engines rely on caching, so
>> first searches will be slow. I think to be fair testing it is necessary
>> to warm up the search engine by sending most frequently used and/or most
>> costly queries, then start your stress testing.
>>
>> I used this tool http://code.google.com/p/httpstone/ to do stress
>> testing. It allows you to create multiple threads sending queries to a
>> server simultaneously, and records time taken to process each query in
>> each thread.
>>
>> Hope it helps.
>>
>>
>>
>> sunnyfr wrote:
>>>
>>> Hi,
>>>
>>> I'm trying as well to stress test solr. I would love some advice to
>>> manage it properly.
>>> I'm using solr 1.3 and tomcat55.
>>> Thanks a lot,
>>>
>>>
>>> zqzuk wrote:
>>>>
>>>> Hi, I am doing a stress testing of my solr application to see how many
>>>> concurrent requests it can handle and how long it takes. But I m not
>>>> sure if I have done it in proper way... responses seem to be very slow
>>>>
>>>> My configuration:
>>>> 1 Solr instance, using the default settings distributed in the example
>>>> code, while I made two changes:
>>>> <useColdSearcher>true</useColdSearcher>
>>>> <maxWarmingSearchers>10</maxWarmingSearchers>
>>>> As I thought the more searchers the more concurrent requests can be
>>>> dealt with?
>>>>
>>>> There are 1.1 million documents indexed, and the platform is winxp sp2,
>>>> duo core 1.8 GB machine with ram 2GB
>>>>
>>>> I used httpstone, a simple server load testing tool to create 100
>>>> workers (so 100 threads) each issuing one same query to the server. To
>>>> deal with a single request of this query it took solr 2 seconds (with
>>>> facet counts), and 7 documents are returned. I was assuming that only
>>>> first request would take longer time and following requests should be
>>>> almost instantaneous as the query is the same. But strange that the
>>>> first response took as long as 20 seconds.
>>>>
>>>> It looked like that the 100 workers sent same request to solr and then
>>>> all of a sudden solr server went silent. Only after 20 seconds some of
>>>> these workers started to receive responses, but still very slow.
>>>>
>>>> clearly there I must have made something wrong with configuring solr
>>>> server... Could you give me some pointers on how to improve the
>>>> performance please?
>>>>
>>>> Many thanks!
>>>>
>>>
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/how-to-improve-concurrent-request-performance-and-stress-testing-tp15299687p20229678.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to improve concurrent request performance and stress
testing
Posted by sunnyfr <jo...@gmail.com>.
just a question about your httpstone's configuration ....?
I would like to know how did you simulate several word search ... ??
Did you create a lot of different workers with lof of different word search
?
Thanks,
zqzuk wrote:
>
> Hi,
>
> try to firstly have a look at http://wiki.apache.org/solr/SolrCaching the
> section on firstsearcher and warming. Search engines rely on caching, so
> first searches will be slow. I think to be fair testing it is necessary to
> warm up the search engine by sending most frequently used and/or most
> costly queries, then start your stress testing.
>
> I used this tool http://code.google.com/p/httpstone/ to do stress testing.
> It allows you to create multiple threads sending queries to a server
> simultaneously, and records time taken to process each query in each
> thread.
>
> Hope it helps.
>
>
>
> sunnyfr wrote:
>>
>> Hi,
>>
>> I'm trying as well to stress test solr. I would love some advice to
>> manage it properly.
>> I'm using solr 1.3 and tomcat55.
>> Thanks a lot,
>>
>>
>> zqzuk wrote:
>>>
>>> Hi, I am doing a stress testing of my solr application to see how many
>>> concurrent requests it can handle and how long it takes. But I m not
>>> sure if I have done it in proper way... responses seem to be very slow
>>>
>>> My configuration:
>>> 1 Solr instance, using the default settings distributed in the example
>>> code, while I made two changes:
>>> <useColdSearcher>true</useColdSearcher>
>>> <maxWarmingSearchers>10</maxWarmingSearchers>
>>> As I thought the more searchers the more concurrent requests can be
>>> dealt with?
>>>
>>> There are 1.1 million documents indexed, and the platform is winxp sp2,
>>> duo core 1.8 GB machine with ram 2GB
>>>
>>> I used httpstone, a simple server load testing tool to create 100
>>> workers (so 100 threads) each issuing one same query to the server. To
>>> deal with a single request of this query it took solr 2 seconds (with
>>> facet counts), and 7 documents are returned. I was assuming that only
>>> first request would take longer time and following requests should be
>>> almost instantaneous as the query is the same. But strange that the
>>> first response took as long as 20 seconds.
>>>
>>> It looked like that the 100 workers sent same request to solr and then
>>> all of a sudden solr server went silent. Only after 20 seconds some of
>>> these workers started to receive responses, but still very slow.
>>>
>>> clearly there I must have made something wrong with configuring solr
>>> server... Could you give me some pointers on how to improve the
>>> performance please?
>>>
>>> Many thanks!
>>>
>>
>>
>
>
--
View this message in context: http://www.nabble.com/how-to-improve-concurrent-request-performance-and-stress-testing-tp15299687p20229303.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to improve concurrent request performance and stress
testing
Posted by zqzuk <zi...@hotmail.com>.
Hi,
try to firstly have a look at http://wiki.apache.org/solr/SolrCaching the
section on firstsearcher and warming. Search engines rely on caching, so
first searches will be slow. I think to be fair testing it is necessary to
warm up the search engine by sending most frequently used and/or most costly
queries, then start your stress testing.
I used this tool http://code.google.com/p/httpstone/ to do stress testing.
It allows you to create multiple threads sending queries to a server
simultaneously, and records time taken to process each query in each thread.
Hope it helps.
sunnyfr wrote:
>
> Hi,
>
> I'm trying as well to stress test solr. I would love some advice to manage
> it properly.
> I'm using solr 1.3 and tomcat55.
> Thanks a lot,
>
>
> zqzuk wrote:
>>
>> Hi, I am doing a stress testing of my solr application to see how many
>> concurrent requests it can handle and how long it takes. But I m not sure
>> if I have done it in proper way... responses seem to be very slow
>>
>> My configuration:
>> 1 Solr instance, using the default settings distributed in the example
>> code, while I made two changes:
>> <useColdSearcher>true</useColdSearcher>
>> <maxWarmingSearchers>10</maxWarmingSearchers>
>> As I thought the more searchers the more concurrent requests can be dealt
>> with?
>>
>> There are 1.1 million documents indexed, and the platform is winxp sp2,
>> duo core 1.8 GB machine with ram 2GB
>>
>> I used httpstone, a simple server load testing tool to create 100 workers
>> (so 100 threads) each issuing one same query to the server. To deal with
>> a single request of this query it took solr 2 seconds (with facet
>> counts), and 7 documents are returned. I was assuming that only first
>> request would take longer time and following requests should be almost
>> instantaneous as the query is the same. But strange that the first
>> response took as long as 20 seconds.
>>
>> It looked like that the 100 workers sent same request to solr and then
>> all of a sudden solr server went silent. Only after 20 seconds some of
>> these workers started to receive responses, but still very slow.
>>
>> clearly there I must have made something wrong with configuring solr
>> server... Could you give me some pointers on how to improve the
>> performance please?
>>
>> Many thanks!
>>
>
>
--
View this message in context: http://www.nabble.com/how-to-improve-concurrent-request-performance-and-stress-testing-tp15299687p20225293.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to improve concurrent request performance and stress
testing
Posted by sunnyfr <jo...@gmail.com>.
Hi,
I'm trying as well to stress test solr. I would love some advice to manage
it properly.
I'm using solr 1.3 and tomcat55.
Thanks a lot,
zqzuk wrote:
>
> Hi, I am doing a stress testing of my solr application to see how many
> concurrent requests it can handle and how long it takes. But I m not sure
> if I have done it in proper way... responses seem to be very slow
>
> My configuration:
> 1 Solr instance, using the default settings distributed in the example
> code, while I made two changes:
> <useColdSearcher>true</useColdSearcher>
> <maxWarmingSearchers>10</maxWarmingSearchers>
> As I thought the more searchers the more concurrent requests can be dealt
> with?
>
> There are 1.1 million documents indexed, and the platform is winxp sp2,
> duo core 1.8 GB machine with ram 2GB
>
> I used httpstone, a simple server load testing tool to create 100 workers
> (so 100 threads) each issuing one same query to the server. To deal with a
> single request of this query it took solr 2 seconds (with facet counts),
> and 7 documents are returned. I was assuming that only first request would
> take longer time and following requests should be almost instantaneous as
> the query is the same. But strange that the first response took as long as
> 20 seconds.
>
> It looked like that the 100 workers sent same request to solr and then all
> of a sudden solr server went silent. Only after 20 seconds some of these
> workers started to receive responses, but still very slow.
>
> clearly there I must have made something wrong with configuring solr
> server... Could you give me some pointers on how to improve the
> performance please?
>
> Many thanks!
>
--
View this message in context: http://www.nabble.com/how-to-improve-concurrent-request-performance-and-stress-testing-tp15299687p20224787.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: how to improve concurrent request performance and stress testing
Posted by Chris Hostetter <ho...@fucit.org>.
: Thank you so much! I will look into firstSearcher configuration next! thanks
FYI: prompted by this thread, I added some blurbs about firstSearcher,
newSearcher, and FieldCache to the SolrCaching wiki ... as a new users
learning about this stuff, please fele free to update that wiki with any
improvements you can think of...
http://wiki.apache.org/solr/SolrCaching
(the best people to write documentation are new users who (unlike the
developers) don't have intrinsic knowledge of the subject in the back of
their mind that they take for granted).
-Hoss
Re: how to improve concurrent request performance and stress testing
Posted by Ziqi Zhang <zi...@hotmail.com>.
Thank you so much! I will look into firstSearcher configuration next! thanks
--------------------------------------------------
From: "Chris Hostetter" <ho...@fucit.org>
Sent: Wednesday, February 06, 2008 8:56 PM
To: <so...@lucene.apache.org>
Subject: Re: how to improve concurrent request performance and stress
testing
> : > Also make sure that common filters, sort fields, and facets have been
> : > warmed.
> :
> : I assume these are achieved by setting large cache size and large
> : autowarmcount number in solr configuration? specifically
>
> autowarming seeds the cahces of a new Searcher using hte keys of an old
> searcher -- it does nothing to help you when you first start up SOlr and
> all of hte caches are empty.
>
> for that you need to either need to manually trigger some sample queries
> externally (before your stress test) or configure something using the
> firstSearcher event listener in solrconfig.xml.
>
> If you saw all of your requests block untill the first one finished, then
> i suspect your queries involve a sort (or faceting) that use the
> FieldCache which is initialized in a single threaded mode (and can't be
> auto-warmed, you can put some simple queries that use those sort fields in
> the newSearcher listener to ensure that they get reinitialized for each
> new searcher)
>
> -Hoss
>
>
Re: how to improve concurrent request performance and stress testing
Posted by Chris Hostetter <ho...@fucit.org>.
: > Also make sure that common filters, sort fields, and facets have been
: > warmed.
:
: I assume these are achieved by setting large cache size and large
: autowarmcount number in solr configuration? specifically
autowarming seeds the cahces of a new Searcher using hte keys of an old
searcher -- it does nothing to help you when you first start up SOlr and
all of hte caches are empty.
for that you need to either need to manually trigger some sample queries
externally (before your stress test) or configure something using the
firstSearcher event listener in solrconfig.xml.
If you saw all of your requests block untill the first one finished, then
i suspect your queries involve a sort (or faceting) that use the
FieldCache which is initialized in a single threaded mode (and can't be
auto-warmed, you can put some simple queries that use those sort fields in
the newSearcher listener to ensure that they get reinitialized for each
new searcher)
-Hoss
Re: how to improve concurrent request performance and stress testing
Posted by Ziqi Zhang <zi...@hotmail.com>.
Thanks!
> Also make sure that common filters, sort fields, and facets have been
> warmed.
I assume these are achieved by setting large cache size and large
autowarmcount number in solr configuration? specifically
filterCache
queryResultCache
documentCache
Thanks!
--------------------------------------------------
From: "Yonik Seeley" <yo...@apache.org>
Sent: Wednesday, February 06, 2008 7:50 AM
To: <so...@lucene.apache.org>
Subject: Re: how to improve concurrent request performance and stress
testing
> On Feb 6, 2008 6:37 PM, Ziqi Zhang <zi...@hotmail.com> wrote:
>> I still do not understand why sending 100 request (of same query) from
>> 100
>> threads throws solr server to silence - is it because of the
>> computational
>> cost to deal with same query in 100 separate threads?
>
> Yes... sending a large number of requests at once can cause one to
> start hitting synchronization bottlenecks.
>
> -Yonik
>
Re: how to improve concurrent request performance and stress testing
Posted by Yonik Seeley <yo...@apache.org>.
On Feb 6, 2008 6:37 PM, Ziqi Zhang <zi...@hotmail.com> wrote:
> I still do not understand why sending 100 request (of same query) from 100
> threads throws solr server to silence - is it because of the computational
> cost to deal with same query in 100 separate threads?
Yes... sending a large number of requests at once can cause one to
start hitting synchronization bottlenecks.
Also make sure that common filters, sort fields, and facets have been warmed.
-Yonik
Re: how to improve concurrent request performance and stress testing
Posted by Ziqi Zhang <zi...@hotmail.com>.
Thanks Yonik,
> It uses a thread per request, simultaneously (up to any limit
> configured by the app server)
How can I change this setting then? I suppose it is to do with Jetty or
Tomcat whichever hosts solr application, not through the solrconfig?
I still do not understand why sending 100 request (of same query) from 100
threads throws solr server to silence - is it because of the computational
cost to deal with same query in 100 separate threads?
I noticed that disabling facet counts improves things a bit, but not
significant.
Thanks in advance!
Re: how to improve concurrent request performance and stress testing
Posted by Yonik Seeley <yo...@apache.org>.
On Feb 6, 2008 7:53 AM, zqzuk <zi...@hotmail.com> wrote:
> If solr receives 10 concurrent request, does it deal with the 10 requests
> simultaneously
It uses a thread per request, simultaneously (up to any limit
configured by the app server)
> using 10 (or as many as possible) searchers
There is a single searcher serving all requests at a time.
-Yonik
Re: how to improve concurrent request performance and stress
testing
Posted by zqzuk <zi...@hotmail.com>.
Hi, I see there's no response to my question, maybe its better to ask this
way...
If solr receives 10 concurrent request, does it deal with the 10 requests
simultaneously using 10 (or as many as possible) searchers, or does it deal
with each request sequentially, which implies that first request will be
served first, and in the worse case, the last request may have to wait for a
long time until all preceding requests have been answered?
Thanks
zqzuk wrote:
>
> Hi, I am doing a stress testing of my solr application to see how many
> concurrent requests it can handle and how long it takes. But I m not sure
> if I have done it in proper way... responses seem to be very slow
>
> My configuration:
> 1 Solr instance, using the default settings distributed in the example
> code, while I made two changes:
> <useColdSearcher>true</useColdSearcher>
> <maxWarmingSearchers>10</maxWarmingSearchers>
> As I thought the more searchers the more concurrent requests can be dealt
> with?
>
> There are 1.1 million documents indexed, and the platform is winxp sp2,
> duo core 1.8 GB machine with ram 2GB
>
> I used httpstone, a simple server load testing tool to create 100 workers
> (so 100 threads) each issuing one same query to the server. To deal with a
> single request of this query it took solr 2 seconds (with facet counts),
> and 7 documents are returned. I was assuming that only first request would
> take longer time and following requests should be almost instantaneous as
> the query is the same. But strange that the first response took as long as
> 20 seconds.
>
> It looked like that the 100 workers sent same request to solr and then all
> of a sudden solr server went silent. Only after 20 seconds some of these
> workers started to receive responses, but still very slow.
>
> clearly there I must have made something wrong with configuring solr
> server... Could you give me some pointers on how to improve the
> performance please?
>
> Many thanks!
>
--
View this message in context: http://www.nabble.com/how-to-improve-concurrent-request-performance-and-stress-testing-tp15299687p15306531.html
Sent from the Solr - User mailing list archive at Nabble.com.