You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Walter Underwood <wu...@netflix.com> on 2008/02/28 19:15:45 UTC
How long does optimize take on your Solr installation?
Please answer with the size of your index (post-optimize) and how long
an optimize takes. I'll collect the data and see if I can draw a line
through it.
190 MB, 55 seconds
$ du -sk /apps/wss/solr_home/data/index
191592 /apps/wss/solr_home/data/index
$ grep commit /apps/wss/tomcat/logs/stdout.log
Feb 28, 2008 11:55:11 AM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=true,waitFlush=false,waitSearcher=true)
Feb 28, 2008 11:56:06 AM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: end_commit_flush
$ uname -a
Linux spiderman4 2.6.9-22.EL #1 SMP Mon Sep 19 17:52:20 EDT 2005 ppc64 ppc64
ppc64 GNU/Linux
wunder
Re: How long does optimize take on your Solr installation?
Posted by Walter Underwood <wu...@netflix.com>.
And I could collect disk subsystem, JVM, processor, and so on, but
we'd have a seven dimensional rule of thumb, which is kinda scary.
wunder
On 2/28/08 12:14 PM, "Grant Ingersoll" <gs...@apache.org> wrote:
> You might want to get info about mergeFactors, and Lucene/Solr
> versions in use.
>
> On Feb 28, 2008, at 1:15 PM, Walter Underwood wrote:
>
>> Please answer with the size of your index (post-optimize) and how long
>> an optimize takes. I'll collect the data and see if I can draw a line
>> through it.
>>
>> 190 MB, 55 seconds
>>
>> $ du -sk /apps/wss/solr_home/data/index
>> 191592 /apps/wss/solr_home/data/index
>> $ grep commit /apps/wss/tomcat/logs/stdout.log
>> Feb 28, 2008 11:55:11 AM org.apache.solr.update.DirectUpdateHandler2
>> commit
>> INFO: start commit(optimize=true,waitFlush=false,waitSearcher=true)
>> Feb 28, 2008 11:56:06 AM org.apache.solr.update.DirectUpdateHandler2
>> commit
>> INFO: end_commit_flush
>> $ uname -a
>> Linux spiderman4 2.6.9-22.EL #1 SMP Mon Sep 19 17:52:20 EDT 2005
>> ppc64 ppc64
>> ppc64 GNU/Linux
>>
>> wunder
>>
>
>
Re: How long does optimize take on your Solr installation?
Posted by Grant Ingersoll <gs...@apache.org>.
You might want to get info about mergeFactors, and Lucene/Solr
versions in use.
On Feb 28, 2008, at 1:15 PM, Walter Underwood wrote:
> Please answer with the size of your index (post-optimize) and how long
> an optimize takes. I'll collect the data and see if I can draw a line
> through it.
>
> 190 MB, 55 seconds
>
> $ du -sk /apps/wss/solr_home/data/index
> 191592 /apps/wss/solr_home/data/index
> $ grep commit /apps/wss/tomcat/logs/stdout.log
> Feb 28, 2008 11:55:11 AM org.apache.solr.update.DirectUpdateHandler2
> commit
> INFO: start commit(optimize=true,waitFlush=false,waitSearcher=true)
> Feb 28, 2008 11:56:06 AM org.apache.solr.update.DirectUpdateHandler2
> commit
> INFO: end_commit_flush
> $ uname -a
> Linux spiderman4 2.6.9-22.EL #1 SMP Mon Sep 19 17:52:20 EDT 2005
> ppc64 ppc64
> ppc64 GNU/Linux
>
> wunder
>
Re: How long does optimize take on your Solr installation?
Posted by sfox <sf...@carleton.edu>.
767 MB 76 seconds
(single, local SATA 7200rpm disk, unloaded XServe G5)
Sean Fox
Walter Underwood wrote:
> Please answer with the size of your index (post-optimize) and how long
> an optimize takes. I'll collect the data and see if I can draw a line
> through it.
>
> 190 MB, 55 seconds
>
> $ du -sk /apps/wss/solr_home/data/index
> 191592 /apps/wss/solr_home/data/index
> $ grep commit /apps/wss/tomcat/logs/stdout.log
> Feb 28, 2008 11:55:11 AM org.apache.solr.update.DirectUpdateHandler2 commit
> INFO: start commit(optimize=true,waitFlush=false,waitSearcher=true)
> Feb 28, 2008 11:56:06 AM org.apache.solr.update.DirectUpdateHandler2 commit
> INFO: end_commit_flush
> $ uname -a
> Linux spiderman4 2.6.9-22.EL #1 SMP Mon Sep 19 17:52:20 EDT 2005 ppc64 ppc64
> ppc64 GNU/Linux
>
> wunder
>
--
sfox@carleton.edu | Technical Director | SERC | Carleton College
Re: How long does optimize take on your Solr installation?
Posted by Norberto Meijome <fr...@meijome.net>.
On Fri, 29 Feb 2008 13:02:21 -0500
"Yonik Seeley" <yo...@apache.org> wrote:
> On Fri, Feb 29, 2008 at 12:45 AM, Walter Underwood
> <wu...@netflix.com> wrote:
> > Good point. My numbers are from a full rebuild. Let's collect maximum
> > times, to keep it simple. --wunder
>
> You may see more variation than you expect since optimization is done
> in stages of mergeFactor segments. In the same environment, you could
> add a single extra doc, and then an optimize would be faster than a
> previous run because that add happened to force a bunch of merges.
Hi all,
Does providing "my optimise takes x minutes on this hardware on a data set this
big" actually tell us useful information, other than rough ideas of how long an
optimise operation could take for {those variables}? I mean, the data ,
configuration, etc you are working on makes quite a bit of difference.
As Walter mentioned, so many variables at hand can get scary...but they
could be grouped as :
1) your SOLR setup (schema, # of docs, configuration)
2) your hardware and OS configuration.
I would guess that to get a proper understanding and provide most useful
information they could be treated separately.
For example, for test 2), a sample SOLR configuration and data be provided and
a set of test scripts be provided. Then anyone can provide information back on
how fast their hardware / config works on SOLR-PERF-TEST_1 (optimised for
overall speed) vs SOLR-PERF-TEST_2 (optimised for commit times) vs ... whatever.
I am not too sure how to have a standard test for the first group... maybe the
data and configuration examples from 2) would be useful enough for finetuning ,
as examples (similar to MySQL 'large' and 'huge' configurations)...
just a thought...
B
_________________________
{Beto|Norberto|Numard} Meijome
Do not take away the camels hump, you may be stopping him from being a camel.
I speak for myself, not my employer. Contents may be hot. Slippery when wet.
Reading disclaimers makes you go blind. Writing them is worse. You have been
Warned.
Re: How long does optimize take on your Solr installation?
Posted by Yonik Seeley <yo...@apache.org>.
On Fri, Feb 29, 2008 at 12:45 AM, Walter Underwood
<wu...@netflix.com> wrote:
> Good point. My numbers are from a full rebuild. Let's collect maximum
> times, to keep it simple. --wunder
You may see more variation than you expect since optimization is done
in stages of mergeFactor segments. In the same environment, you could
add a single extra doc, and then an optimize would be faster than a
previous run because that add happened to force a bunch of merges.
-Yonik
Re: How long does optimize take on your Solr installation?
Posted by Walter Underwood <wu...@netflix.com>.
Good point. My numbers are from a full rebuild. Let's collect maximum
times, to keep it simple. --wunder
On 2/28/08 7:28 PM, "Alex Benjamen" <ab...@friendster.com> wrote:
> It mostly depends on whether or not the index is completely new or incremental
>
> 4Gb, 28MM docs, ~30min (new index)
> 4Gb, 28MM docs, 30s (incremental)
>
>
RE: How long does optimize take on your Solr installation?
Posted by Alex Benjamen <ab...@friendster.com>.
It mostly depends on whether or not the index is completely new or incremental
4Gb, 28MM docs, ~30min (new index)
4Gb, 28MM docs, 30s (incremental)