You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Erik Hatcher <er...@ehatchersolutions.com> on 2007/02/05 11:23:17 UTC
performance testing practices
This week I'm going to be incrementally loading up to 3.7M records
into Solr, in 50k chunks.
I'd like to capture some performance numbers after each chunk to see
how she holds up.
What numbers are folks capturing? What techniques are you using to
capture numbers? I'm not looking for anything elaborate, as the goal
is really to see how faceting fares as more data is loaded. We've
got some ugly data in our initial experiment, so the faceting
concerns me.
Thanks,
Erik
Re: performance testing practices
Posted by Bertrand Delacretaz <bd...@apache.org>.
On 2/5/07, Erik Hatcher <er...@ehatchersolutions.com> wrote:
> ...What numbers are folks capturing? What techniques are you using to
> capture numbers?...
I've been using my httpstone utility
(http://code.google.com/p/httpstone/) along with ab
(http://httpd.apache.org/docs/2.2/programs/ab.html) to generate many
concurrent search requests, based on semi-random query URLs generated
by shell scripts.
The goal was to find out, on our hardware, how many typical queries
per second we could serve with acceptable response times (less than
2.5 seconds).
In our case, we found out that 100-200 requests per second were not a
problem, and stopped testing as this is much more than we need
currently. So I don't have precise numbers, but we know that we're
safe with our current load.
HTH, but it's more empirical than structured testing ;-)
-Bertrand
Re: performance testing practices
Posted by Yonik Seeley <yo...@apache.org>.
On 2/5/07, Erik Hatcher <er...@ehatchersolutions.com> wrote:
> The facets are bibliographic metadata about library holdings, such as
> genre, subject, format, published date (year), and others. Basically
> an open source think like this:
>
> <http://www2.lib.ncsu.edu/catalog/?N=201015&Ns=Call+Number+sort%
> 7c0&sort=5>
>
> (if that link didn't work, hit the main page at <http://
> www.lib.ncsu.edu/catalog/browse.html> and drill in a little)
>
> The data is real ugly, and there are typically several values per
> field, so all facets are currently set as multiValued.
You may have some difficulties with faceting by author...
What type of hardware do you have for this?
For testing, we have an in-house performance lab that collects all
sorts of cool data.
But for quick-n-dirty performance hacking, I prefer something simple
like the following.
It requires a list of queries to start with (we normally do
performance testing with data derived from real query logs).
------------------- perf.py ---------------
import urllib2
import time
import threading
import random
def geturl(url):
#print "###getting ",url
f = urllib2.urlopen(url)
data = f.read()
if verbose: print url + "\n" + data
#headers = f.info()
f.close()
#print "!!!got request"
res=[]
requests_left=0
errors=[]
def test(uri, urllist):
global res,requests_left,errors
for url in urllist:
if len(res)>=requests: break
start=time.time()
try:
geturl(uri+url+append)
except Exception,e:
errors.append((url,e))
print "CAUGHT EXCEPTION",e
elapsed=time.time()-start
res.append( (elapsed,url) )
class TestThread(threading.Thread):
def __init__ (self,*args):
threading.Thread.__init__(self)
self.args = args
def run(self): test(*self.args)
#argument defaults
clients=1; requests=1; randomize=True; queries='dict.txt'; append=''
verbose=False
uri='http://cn-ewr1-dev40-pi2:5051'
#poor man's argument parsing
import sys
for statement in sys.argv[1:]: exec(statement)
lst = [ line.strip() for line in open(queries).readlines() ]
thr=[]
for client in range(clients):
ulst=lst
if randomize:
ulst=ulst[:]
random.shuffle(ulst)
thr.append(TestThread(uri,ulst))
print "############## starting all threads"
start=time.time()
for cli in thr: cli.start()
print "############## waiting for all threads"
for cli in thr: cli.join()
elapsed=time.time()-start
res.sort()
n=len(res)
print 'Slowest Queries:'
for i in res[:-10:-1]: print '%.3f %s' % i
print
print len(errors),'Errors'
for i in errors[:10]: print i
print
print 'total requests=%d clients=%d' % (n,clients)
print 'throughput=%.1f queries/second' % (n/elapsed)
print "99.9%%=%.3f seconds" % res[int(n*.999)][0]
print "99%%=%.3f" % res[int(n*.99)][0]
print "98%%=%.3f" % res[int(n*.98)][0]
print "95%%=%.3f" % res[int(n*.95)][0]
print "75%%=%.3f" % res[int(n*.75)][0]
print "50%%=%.3f" % res[int(n*.50)][0]
print "avg%%=%.3f" % (sum([ r[0] for r in res ])/n)
Re: performance testing practices
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Feb 5, 2007, at 11:15 AM, Yonik Seeley wrote:
> On 2/5/07, Erik Hatcher <er...@ehatchersolutions.com> wrote:
>> This week I'm going to be incrementally loading up to 3.7M records
>> into Solr, in 50k chunks.
>>
>> I'd like to capture some performance numbers after each chunk to see
>> how she holds up.
>>
>> What numbers are folks capturing? What techniques are you using to
>> capture numbers? I'm not looking for anything elaborate, as the goal
>> is really to see how faceting fares as more data is loaded. We've
>> got some ugly data in our initial experiment, so the faceting
>> concerns me.
>
> Gulp... me too. That sounds like a lot of data, and the faceting code
> is still young (it will get better with age :-)
> The big performance factor for faceting will relate to the number of
> unique values in a field.
> So what are you trying to facet on?
The facets are bibliographic metadata about library holdings, such as
genre, subject, format, published date (year), and others. Basically
an open source think like this:
<http://www2.lib.ncsu.edu/catalog/?N=201015&Ns=Call+Number+sort%
7c0&sort=5>
(if that link didn't work, hit the main page at <http://
www.lib.ncsu.edu/catalog/browse.html> and drill in a little)
The data is real ugly, and there are typically several values per
field, so all facets are currently set as multiValued.
We shall see!
Erik
Re: performance testing practices
Posted by Yonik Seeley <yo...@apache.org>.
On 2/5/07, Erik Hatcher <er...@ehatchersolutions.com> wrote:
> This week I'm going to be incrementally loading up to 3.7M records
> into Solr, in 50k chunks.
>
> I'd like to capture some performance numbers after each chunk to see
> how she holds up.
>
> What numbers are folks capturing? What techniques are you using to
> capture numbers? I'm not looking for anything elaborate, as the goal
> is really to see how faceting fares as more data is loaded. We've
> got some ugly data in our initial experiment, so the faceting
> concerns me.
Gulp... me too. That sounds like a lot of data, and the faceting code
is still young (it will get better with age :-)
The big performance factor for faceting will relate to the number of
unique values in a field.
So what are you trying to facet on?
-Yonik
Re: performance testing practices
Posted by "Burkamp, Christian" <C....@Ceyoniq.com>.
Hi there,
I am working on some performance numbers too. This is part of my evaluation of solr. I'm planning to replace a legacy search engine and have to find out if this is possible with solr.
I have loaded 1,1 million documents into solr by now. Indexing speed is not a big concern for me. I had about 17 documents per second while my indexing client is still only a python prototype with a very slow filtering engine based on windows Ifilter.
I'm measuring the search performance by using a python client that is continually querying solr. It grabs a random word from the results and uses it for the next search. For every search request the time from sending the request till receiving the response is taken. Every query uses one word as search text and one word as filter query text. Highlighting is on.
Some first results:
Solr loaded with 1120000 Documents:
Max queries per second: 14,5
Average request duration with only 1 client: 0,08 s
My criteria of 90% requests completing in less than 1 second is met with a maximum of 10 parallel clients.
I suspect to serve at least 300 users with one system like this.
(Measured on a single CPU Pentium4 3GHz, 2GB RAM, internal standard ATA Drive)
Next step will be to increase the number of documents till I reach the point where no request is completed in less than 1 second. (From this point on no amount of replication can bring me back to production performance).
I have a few questions, too.
- What size is the largest known solr server
- What number of documents do you think can be handled by solr
- Solr is using only one lucene index. There has been a thread about this before but it was more related to bringing together different lucene indexes under one solr server. I potentially need a solution for up to 500 millions of documents. I believe this will not work without splitting the index. What do you think?
- Does anybody have own performance numbers they would share?
- solr was running under jetty for my performance tests. What container is best suited for high performance?
Thanks a lot for the inspiring talk going on on this mailing list.
Christian
-----Ursprüngliche Nachricht-----
Von: Erik Hatcher [mailto:erik@ehatchersolutions.com]
Gesendet: Montag, 5. Februar 2007 11:23
An: solr-user@lucene.apache.org
Betreff: performance testing practices
This week I'm going to be incrementally loading up to 3.7M records
into Solr, in 50k chunks.
I'd like to capture some performance numbers after each chunk to see
how she holds up.
What numbers are folks capturing? What techniques are you using to
capture numbers? I'm not looking for anything elaborate, as the goal
is really to see how faceting fares as more data is loaded. We've
got some ugly data in our initial experiment, so the faceting
concerns me.
Thanks,
Erik