You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Benjamin Stein <be...@shadowtv.com> on 2006/06/07 21:16:26 UTC

IndexWriter.addIndexes & optimization

I have a very large corpus that I am storing in many indexes: 200 indexes
* ~500MB each, with 10^6 very tiny documents in each.  (I could look into
optimizing this later, of course, but seems ok for now)

During indexing, I have been using a RAMDirectory to store many thousands of
documents in memory before flushing the buffer to disk using
IndexWriter.addIndexes.
For the most part this works very well, except that performance degrades
tremendously over time due to the implicit call (or two!)
to optimize() inside the addIndexes function.

I searched the archives and found that this topic has come up a number of
times over the years, but with very few answers (except for one amusing one
from Doug:  "I don't recall exactly why this was done. (I should have
written a comment!)"  :)

Is there a way to accomplish what I'm trying to do without all the calls to
optimize()?

-Ben

Re: IndexWriter.addIndexes & optimizatio

Posted by Yonik Seeley <ys...@gmail.com>.
On 6/8/06, Otis Gospodnetic <ot...@yahoo.com> wrote:
> When writing a unit test that comapres RAMDirectory and FSDirectory performance for Lucene in Action I had a very hard time showing that RAMDirectory really is faster. :)

For indexing, even if you open IndexWriter with a FSDirectory, it
internally uses RAMDirectory to buffer until it reaches
maxBufferedDocs.


-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: IndexWriter.addIndexes & optimizatio

Posted by vipin sharma <he...@gmail.com>.
 - > Just set your maxBufferedDocs to as high a number as your RAM/heap will
let you, and pick a mergeFactor that is high, but doesn't get you in trouble
with open files.

can you please explaing this in brief??

regards and thanks,

On 6/9/06, Otis Gospodnetic <ot...@yahoo.com> wrote:
>
> When writing a unit test that comapres RAMDirectory and FSDirectory
> performance for Lucene in Action I had a very hard time showing that
> RAMDirectory really is faster. :)  Just set your maxBufferedDocs to as high
> a number as your RAM/heap will let you, and pick a mergeFactor that is high,
> but doesn't get you in trouble with open files.
>
> Otis
>
> ----- Original Message ----
> From: Dan Armbrust <da...@gmail.com>
> To: java-user@lucene.apache.org
> Sent: Wednesday, June 7, 2006 4:05:49 PM
> Subject: Re: IndexWriter.addIndexes & optimization
>
> Benjamin Stein wrote:
>
> >
> > I could probably store the little RAMDirectories to disk as many
> > FSDirectories, and then addIndexes() of *all* the FSDirectories at the
> end
> > instead of every time.  That would probably be smart.
> >
> > Glad I asked myself!
> >
>
> That was what I was going to suggest - you may also want to benchmark to
> see if the RAMDirectory is buying you anything.  With the data that I am
> indexing on my hardware, I found it to be faster to index to a regular
> FSDirectory that it is to use the RAMDirectory.  Especially if you tweak
> the performance knobs on the indexer so it does its own caching before
> it writes to the Directory.
>
> I do batches of documents to FSDirectories - and then merge all of the
> FSDirectories into a new master index at the end - so I never have to
> optimize during the indexing process.
>
> Dan
>
>
> --
> ****************************
> Daniel Armbrust
> Biomedical Informatics
> Mayo Clinic Rochester
> daniel.armbrust(at)mayo.edu
> http://informatics.mayo.edu/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Re: IndexWriter.addIndexes & optimization

Posted by Otis Gospodnetic <ot...@yahoo.com>.
When writing a unit test that comapres RAMDirectory and FSDirectory performance for Lucene in Action I had a very hard time showing that RAMDirectory really is faster. :)  Just set your maxBufferedDocs to as high a number as your RAM/heap will let you, and pick a mergeFactor that is high, but doesn't get you in trouble with open files.

Otis

----- Original Message ----
From: Dan Armbrust <da...@gmail.com>
To: java-user@lucene.apache.org
Sent: Wednesday, June 7, 2006 4:05:49 PM
Subject: Re: IndexWriter.addIndexes & optimization

Benjamin Stein wrote:

> 
> I could probably store the little RAMDirectories to disk as many
> FSDirectories, and then addIndexes() of *all* the FSDirectories at the end
> instead of every time.  That would probably be smart.
> 
> Glad I asked myself!
> 

That was what I was going to suggest - you may also want to benchmark to 
see if the RAMDirectory is buying you anything.  With the data that I am 
indexing on my hardware, I found it to be faster to index to a regular 
FSDirectory that it is to use the RAMDirectory.  Especially if you tweak 
the performance knobs on the indexer so it does its own caching before 
it writes to the Directory.

I do batches of documents to FSDirectories - and then merge all of the 
FSDirectories into a new master index at the end - so I never have to 
optimize during the indexing process.

Dan


-- 
****************************
Daniel Armbrust
Biomedical Informatics
Mayo Clinic Rochester
daniel.armbrust(at)mayo.edu
http://informatics.mayo.edu/

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org





---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: IndexWriter.addIndexes & optimization

Posted by Dan Armbrust <da...@gmail.com>.
Benjamin Stein wrote:

> 
> I could probably store the little RAMDirectories to disk as many
> FSDirectories, and then addIndexes() of *all* the FSDirectories at the end
> instead of every time.  That would probably be smart.
> 
> Glad I asked myself!
> 

That was what I was going to suggest - you may also want to benchmark to 
see if the RAMDirectory is buying you anything.  With the data that I am 
indexing on my hardware, I found it to be faster to index to a regular 
FSDirectory that it is to use the RAMDirectory.  Especially if you tweak 
the performance knobs on the indexer so it does its own caching before 
it writes to the Directory.

I do batches of documents to FSDirectories - and then merge all of the 
FSDirectories into a new master index at the end - so I never have to 
optimize during the indexing process.

Dan


-- 
****************************
Daniel Armbrust
Biomedical Informatics
Mayo Clinic Rochester
daniel.armbrust(at)mayo.edu
http://informatics.mayo.edu/

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: IndexWriter.addIndexes & optimization

Posted by Grant Ingersoll <gs...@syr.edu>.
My understanding of the IndexWriter code is that it more or less manages 
this for you.  It has an internal RAMDirectory which it uses to index in 
memory and then periodically flushes to disk based on your merge factor 
settings (amongst other settings).  So I am not sure if the extra work 
you are doing is really saving you anything.  Once all this is done, 
then you can call optimize just once at the end (or wherever you feel 
you need to)

Benjamin Stein wrote:
> On 6/7/06, Benjamin Stein <be...@shadowtv.com> wrote:
>>
>> During indexing, I have been using a RAMDirectory to store many 
>> thousands
>> of documents in memory before flushing the buffer to disk using
>> IndexWriter.addIndexes.
>> For the most part this works very well, except that performance degrades
>> tremendously over time due to the implicit call (or two!)
>> to optimize() inside the addIndexes function.
>>
>
>
> I could probably store the little RAMDirectories to disk as many
> FSDirectories, and then addIndexes() of *all* the FSDirectories at the 
> end
> instead of every time.  That would probably be smart.
>
> Glad I asked myself!
>

-- 

Grant Ingersoll 
Sr. Software Engineer 
Center for Natural Language Processing 
Syracuse University 
School of Information Studies 
335 Hinds Hall 
Syracuse, NY 13244 

http://www.cnlp.org 
Voice:  315-443-5484 
Fax: 315-443-6886 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: IndexWriter.addIndexes & optimization

Posted by Benjamin Stein <be...@shadowtv.com>.
On 6/7/06, Benjamin Stein <be...@shadowtv.com> wrote:
>
> During indexing, I have been using a RAMDirectory to store many thousands
> of documents in memory before flushing the buffer to disk using
> IndexWriter.addIndexes.
> For the most part this works very well, except that performance degrades
> tremendously over time due to the implicit call (or two!)
> to optimize() inside the addIndexes function.
>


I could probably store the little RAMDirectories to disk as many
FSDirectories, and then addIndexes() of *all* the FSDirectories at the end
instead of every time.  That would probably be smart.

Glad I asked myself!