You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by maureen tanuwidjaja <au...@yahoo.com> on 2007/02/11 16:44:16 UTC

about merge factor

Hi all,
  
  I just wondering wheter is it sensible and possible if I have 660,000  documents to be indexed,I set the merge factor to 660,000 instead of  the default value 10 (...and this means no merge while indexing) and  later after closing the index,I use the IndexWriter to optimize/merge  the whole index file...
  
  
  Thanks and Regards,
  Maureen
  
 
---------------------------------
We won't tell. Get more on shows you hate to love
(and love to hate): Yahoo! TV's Guilty Pleasures list.

Re: about merge factor

Posted by Mark Miller <ma...@gmail.com>.
Found a mistake in my reponse...when I was talking about max merge docs, 
I meant max buffered docs. If your going to optimize anyway, the key 
setting appears to be max buffered docs, and I have yet to see the merge 
factor affect anything (again, only if you optimize). Oddly, performance 
seems to decrease as you up max buffered docs far before you are even 
close to running out of available ram. I do not know why this is, but 
you should certainly test to see what your prime settings are.

Also, the knew benchmarking stuff is awesome.

- Mark

Grant Ingersoll wrote:
> You may find contrib/Benchmark useful in your testing.  Doron Cohen 
> has added a nice framework for scripting benchmarking tests.
>
> -Grant
>
> On Feb 11, 2007, at 12:14 PM, Mark Miller wrote:
>
>> Not sensible at all. First, a merge above something like 90 most 
>> likely never makes since. Second, I have done some testing and my 
>> results show that if you optimize the index after loading, the merge 
>> factor really doesn't matter so keep it at 10 (I never used a max 
>> merge docs below 50. 100 worked best, 1,000 and 2,000 slowed things 
>> down even though the test had access to 600MB RAM and the docs where 
>> around 10-20k each). Setting up a test harness that automatically 
>> indexes a good amount of docs (I did 20,000) with a variety of 
>> settings will tell you a lot. Things will obviously bend based on 
>> your setup.
>>
>> - Mark
>>
>> maureen tanuwidjaja wrote:
>>> Hi all,
>>>     I just wondering wheter is it sensible and possible if I have 
>>> 660,000  documents to be indexed,I set the merge factor to 660,000 
>>> instead of  the default value 10 (...and this means no merge while 
>>> indexing) and  later after closing the index,I use the IndexWriter 
>>> to optimize/merge  the whole index file...
>>>       Thanks and Regards,
>>>   Maureen
>>>    ---------------------------------
>>> We won't tell. Get more on shows you hate to love
>>> (and love to hate): Yahoo! TV's Guilty Pleasures list.
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
> --------------------------
> Grant Ingersoll
> Center for Natural Language Processing
> http://www.cnlp.org
>
> Read the Lucene Java FAQ at 
> http://wiki.apache.org/jakarta-lucene/LuceneFAQ
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: about merge factor

Posted by Grant Ingersoll <gs...@apache.org>.
You may find contrib/Benchmark useful in your testing.  Doron Cohen  
has added a nice framework for scripting benchmarking tests.

-Grant

On Feb 11, 2007, at 12:14 PM, Mark Miller wrote:

> Not sensible at all. First, a merge above something like 90 most  
> likely never makes since. Second, I have done some testing and my  
> results show that if you optimize the index after loading, the  
> merge factor really doesn't matter so keep it at 10 (I never used a  
> max merge docs below 50. 100 worked best, 1,000 and 2,000 slowed  
> things down even though the test had access to 600MB RAM and the  
> docs where around 10-20k each). Setting up a test harness that  
> automatically indexes a good amount of docs (I did 20,000) with a  
> variety of settings will tell you a lot. Things will obviously bend  
> based on your setup.
>
> - Mark
>
> maureen tanuwidjaja wrote:
>> Hi all,
>>     I just wondering wheter is it sensible and possible if I have  
>> 660,000  documents to be indexed,I set the merge factor to 660,000  
>> instead of  the default value 10 (...and this means no merge while  
>> indexing) and  later after closing the index,I use the IndexWriter  
>> to optimize/merge  the whole index file...
>>       Thanks and Regards,
>>   Maureen
>>    ---------------------------------
>> We won't tell. Get more on shows you hate to love
>> (and love to hate): Yahoo! TV's Guilty Pleasures list.
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

--------------------------
Grant Ingersoll
Center for Natural Language Processing
http://www.cnlp.org

Read the Lucene Java FAQ at http://wiki.apache.org/jakarta-lucene/ 
LuceneFAQ



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: about merge factor

Posted by Mark Miller <ma...@gmail.com>.
Not sensible at all. First, a merge above something like 90 most likely 
never makes since. Second, I have done some testing and my results show 
that if you optimize the index after loading, the merge factor really 
doesn't matter so keep it at 10 (I never used a max merge docs below 50. 
100 worked best, 1,000 and 2,000 slowed things down even though the test 
had access to 600MB RAM and the docs where around 10-20k each). Setting 
up a test harness that automatically indexes a good amount of docs (I 
did 20,000) with a variety of settings will tell you a lot. Things will 
obviously bend based on your setup.

- Mark

maureen tanuwidjaja wrote:
> Hi all,
>   
>   I just wondering wheter is it sensible and possible if I have 660,000  documents to be indexed,I set the merge factor to 660,000 instead of  the default value 10 (...and this means no merge while indexing) and  later after closing the index,I use the IndexWriter to optimize/merge  the whole index file...
>   
>   
>   Thanks and Regards,
>   Maureen
>   
>  
> ---------------------------------
> We won't tell. Get more on shows you hate to love
> (and love to hate): Yahoo! TV's Guilty Pleasures list.
>   

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: about merge factor

Posted by Damien McCarthy <da...@propylon.com>.
Hi Maureen,

If you are attempting to tune the indexing have a read of
http://www.onjava.com/pub/a/onjava/2003/03/05/lucene.html?page=2 . Otis has
some excellent comparisons and explanations of the different optimization
strategies available.  

D.

-----Original Message-----
From: maureen tanuwidjaja [mailto:autumn_musique@yahoo.com] 
Sent: 11 February 2007 15:44
To: java-user@lucene.apache.org
Subject: about merge factor

Hi all,
  
  I just wondering wheter is it sensible and possible if I have 660,000
documents to be indexed,I set the merge factor to 660,000 instead of  the
default value 10 (...and this means no merge while indexing) and  later
after closing the index,I use the IndexWriter to optimize/merge  the whole
index file...
  
  
  Thanks and Regards,
  Maureen
  
 
---------------------------------
We won't tell. Get more on shows you hate to love
(and love to hate): Yahoo! TV's Guilty Pleasures list.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org