You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Max Pfingsthorn <m....@hippo.nl> on 2006/04/05 13:02:38 UTC

Optimize completely in memory with a FSDirectory?

Hi all,

I have a question about memory/fileio settings and the FSDirectory.
The setMaxBufferedDocs and related parameters help a lot already to fully exploit my RAM when indexing, but since I'm running a fairly small index of around 40000 docs and I'm optimizing it relatively often, I was wondering if there is any way to enforce complete in-memory optimization.
The stupid thing is that even with a maxBufferedDocs of 50000, it still writes lots of tiny files to disk (together almost 2-3 times the size of the index), and the disk io skyrockets for a few seconds. I have enough memory to hold the index many times over, so it really shouldn't be a problem there, and it would be so much faster (I have to think).

Any hints?

Best regards,

Max Pfingsthorn

Hippo  

Oosteinde 11
1017WT Amsterdam
The Netherlands
Tel  +31 (0)20 5224466
-------------------------------------------------------------
m.pfingsthorn@hippo.nl / www.hippo.nl
-------------------------------------------------------------

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: Optimize completely in memory with a FSDirectory?

Posted by Daniel Naber <lu...@danielnaber.de>.
On Mittwoch 05 April 2006 13:02, Max Pfingsthorn wrote:

> The setMaxBufferedDocs and related parameters help a lot already to
> fully exploit my RAM when indexing, but since I'm running a fairly small
> index of around 40000 docs and I'm optimizing it relatively often, I was
> wondering if there is any way to enforce complete in-memory
> optimization.

Maybe you could use a RAMDirectory and write it to disk using 
IndexWriter.addIndexes() from time to time?

Regards
 Daniel

-- 
http://www.danielnaber.de

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org