You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by dan sutton <da...@gmail.com> on 2012/01/27 15:44:26 UTC
Solr Warm-up performance issues
Hi List,
We use Solr 4.0.2011.12.01.09.59.41 and have a dataset of roughly 40 GB.
Every day we produce a new dataset of 40 GB and have to switch one for
the other.
Once the index switch over has taken place, it takes roughly 30 min for Solr
to reach maximum performance. Are there any hardware or software solutions
to reduce the warm-up time ? We tried warm-up queries but it didn't change
much.
Our hardware specs is:
* Dell Poweredge 1950
* 2 x Quad-Core Xeon E5405 (2.00GHz)
* 48 GB RAM
* 2 x 146 GB SAS 3 Gb/s 15K RPM disk configured in RAID mirror
One thing that does seem to take a long time is un-inverting a set of
multivalued fields, are there any optimizations we might be able to
use here?
Thanks for your help.
Dan
RE: Solr Warm-up performance issues
Posted by Peter Velikin <pe...@velobit.com>.
Dan,
I can suggest a solution that should help. VeloBit enables you to add SSDs
to your servers as a cache (SSD will cost you $200, per server should be
enough). Then, assuming a 100MB/s read speed from your SAS disks, you can
read 50GB data into the VeloBit HyperCache cache in about 9 mins (this
happens automatically, all you need to do is add the SSD to your server and
install Velobit one time, which takes 2 minutes). Solr should run much
faster after that. The added benefit of the solution is that you would have
also boosted the steady state performance by 4x.
Let me know if you are interested in trying it out and I'll set you up to
talk with my engineers.
Best regards,
Peter Velikin
VP Online Marketing, VeloBit, Inc.
peter@velobit.com
tel. 978-263-4800
mob. 617-306-7165
VeloBit provides plug & play SSD caching software that dramatically
accelerates applications at a remarkably low cost. The software installs
seamlessly in less than 10 minutes and automatically tunes for fastest
application speed. Visit www.velobit.com for details.
-----Original Message-----
From: dan sutton [mailto:danbsutton@gmail.com]
Sent: Friday, January 27, 2012 9:44 AM
To: solr-user
Subject: Solr Warm-up performance issues
Hi List,
We use Solr 4.0.2011.12.01.09.59.41 and have a dataset of roughly 40 GB.
Every day we produce a new dataset of 40 GB and have to switch one for the
other.
Once the index switch over has taken place, it takes roughly 30 min for Solr
to reach maximum performance. Are there any hardware or software solutions
to reduce the warm-up time ? We tried warm-up queries but it didn't change
much.
Our hardware specs is:
* Dell Poweredge 1950
* 2 x Quad-Core Xeon E5405 (2.00GHz)
* 48 GB RAM
* 2 x 146 GB SAS 3 Gb/s 15K RPM disk configured in RAID mirror
One thing that does seem to take a long time is un-inverting a set of
multivalued fields, are there any optimizations we might be able to use
here?
Thanks for your help.
Dan
Re: Solr Warm-up performance issues
Posted by Lance Norskog <go...@gmail.com>.
Another trick is to read in the parts of the index file that you
search against: term dictionary and maybe a few others. (The Lucene
wiki describes the various files.) That is, you copy the new index to
the server and then say "cat files > /dev/null". This pre-caches the
interesting files into memory.
This leads to: how large is you JVM and how much space do you leave to
the OS? The OS is much better at managing memory against the hard disk
that Solr/JVM is. The JVM should have enough memory to run your Solr
comfortably without slowdowns, and that is the most it should get.
You might find autowarming less useful than just picking a series of
queries that warm what you want to get rolling early: sort on the
fields you want, do a series of facet queries, search for words you
get a lot, etc.
Another problem is that you might be fighting garbage collection when
switching from the old to new collection. Just shut down Solr, switch
the index directory, cat the files mentioned above, and restart.
On Fri, Jan 27, 2012 at 10:36 PM, Otis Gospodnetic
<ot...@yahoo.com> wrote:
> Hi Dan,
>
> I think this may be your problem:
>
>> Every day we produce a new dataset of 40 GB and have to switch one for the othe
>
> If you really replace an index with a new index one a day, you throw away all the hard work the OS has been doing to cache hot parts of your index in memory. It takes it 30 minutes apparently in your cache to re-cache things. Check the link in my signature. If you use that and if I'm right about this, you will see a big spike in Disk Reads after you switch to the new index. You want to minimize that spike.
>
>
> So see if you can avoid replacing the whole index and if that is really not doable, you can try warmup queries, but of course while you run them, if they are expensive, they will hurt system performance somewhat.
>
> Otis
>
> ----
> Performance Monitoring SaaS for Solr - http://sematext.com/spm/solr-performance-monitoring/index.html
>
>
>
>
>>________________________________
>> From: dan sutton <da...@gmail.com>
>>To: solr-user <so...@lucene.apache.org>
>>Sent: Friday, January 27, 2012 9:44 AM
>>Subject: Solr Warm-up performance issues
>>
>>Hi List,
>>
>>We use Solr 4.0.2011.12.01.09.59.41 and have a dataset of roughly 40 GB.
>>Every day we produce a new dataset of 40 GB and have to switch one for
>>the other.
>>
>>Once the index switch over has taken place, it takes roughly 30 min for Solr
>>to reach maximum performance. Are there any hardware or software solutions
>>to reduce the warm-up time ? We tried warm-up queries but it didn't change
>>much.
>>
>>Our hardware specs is:
>> * Dell Poweredge 1950
>> * 2 x Quad-Core Xeon E5405 (2.00GHz)
>> * 48 GB RAM
>> * 2 x 146 GB SAS 3 Gb/s 15K RPM disk configured in RAID mirror
>>
>>One thing that does seem to take a long time is un-inverting a set of
>>multivalued fields, are there any optimizations we might be able to
>>use here?
>>
>>Thanks for your help.
>>Dan
>>
>>
>>
--
Lance Norskog
goksron@gmail.com
Re: Solr Warm-up performance issues
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi Dan,
I think this may be your problem:
> Every day we produce a new dataset of 40 GB and have to switch one for the othe
If you really replace an index with a new index one a day, you throw away all the hard work the OS has been doing to cache hot parts of your index in memory. It takes it 30 minutes apparently in your cache to re-cache things. Check the link in my signature. If you use that and if I'm right about this, you will see a big spike in Disk Reads after you switch to the new index. You want to minimize that spike.
So see if you can avoid replacing the whole index and if that is really not doable, you can try warmup queries, but of course while you run them, if they are expensive, they will hurt system performance somewhat.
Otis
----
Performance Monitoring SaaS for Solr - http://sematext.com/spm/solr-performance-monitoring/index.html
>________________________________
> From: dan sutton <da...@gmail.com>
>To: solr-user <so...@lucene.apache.org>
>Sent: Friday, January 27, 2012 9:44 AM
>Subject: Solr Warm-up performance issues
>
>Hi List,
>
>We use Solr 4.0.2011.12.01.09.59.41 and have a dataset of roughly 40 GB.
>Every day we produce a new dataset of 40 GB and have to switch one for
>the other.
>
>Once the index switch over has taken place, it takes roughly 30 min for Solr
>to reach maximum performance. Are there any hardware or software solutions
>to reduce the warm-up time ? We tried warm-up queries but it didn't change
>much.
>
>Our hardware specs is:
> * Dell Poweredge 1950
> * 2 x Quad-Core Xeon E5405 (2.00GHz)
> * 48 GB RAM
> * 2 x 146 GB SAS 3 Gb/s 15K RPM disk configured in RAID mirror
>
>One thing that does seem to take a long time is un-inverting a set of
>multivalued fields, are there any optimizations we might be able to
>use here?
>
>Thanks for your help.
>Dan
>
>
>
Re: Solr Warm-up performance issues
Posted by Tomás Fernández Löbbe <to...@gmail.com>.
You say warming queries didn't help? How do those look like? Make sure you
facet and sort in all of the fields that your application allow
faceting/sorting. The same with the filters. Uninversion of fields is done
only when you commit, but warming queries should help you here.
Tomás
On Fri, Jan 27, 2012 at 11:44 AM, dan sutton <da...@gmail.com> wrote:
> Hi List,
>
> We use Solr 4.0.2011.12.01.09.59.41 and have a dataset of roughly 40 GB.
> Every day we produce a new dataset of 40 GB and have to switch one for
> the other.
>
> Once the index switch over has taken place, it takes roughly 30 min for
> Solr
> to reach maximum performance. Are there any hardware or software solutions
> to reduce the warm-up time ? We tried warm-up queries but it didn't change
> much.
>
> Our hardware specs is:
> * Dell Poweredge 1950
> * 2 x Quad-Core Xeon E5405 (2.00GHz)
> * 48 GB RAM
> * 2 x 146 GB SAS 3 Gb/s 15K RPM disk configured in RAID mirror
>
> One thing that does seem to take a long time is un-inverting a set of
> multivalued fields, are there any optimizations we might be able to
> use here?
>
> Thanks for your help.
> Dan
>