RAMDirectory with FSDirectory merging Versus large mergeFactor and RAMBufferSizeMB

[Maxim Terletsky <sx...@yahoo.com>]

Hi guys,
There are two approaches I see in Lucene In Action about speeding up the indexing process. 

1) Simply increase the mergeFactor and RAMBufferSizeMB.
2) Using RAMDirectory as a buffer (perhaps even several in parallel) and later merging it using addIndexes to FSDirectory.

So my question is the following:
In case I have only 1 thread with RAMDirectory - is that pretty much the same as method 1? Since it's in memory anyhow for large mergeFactor and large RAMBufferSizeMB.  
 
Maxim


________________________________

Re: RAMDirectory with FSDirectory merging Versus large mergeFactor and RAMBufferSizeMB

[Lance Norskog <go...@gmail.com>]

Ramdirectory is no longer an interesting technique for this. It makes
garbage collection do a lot of work. With memory-mapped directory the
data is cached by the OS instead of Java, and OS is very good at this.

TieredMergePolicy is much smarter about time spent merging segments.

Lucene In Action 2 might be more help than a 6-year-old book :)

On Mon, Jun 4, 2012 at 12:47 AM, Maxim Terletsky <sx...@yahoo.com> wrote:
> Hi guys,
> There are two approaches I see in Lucene In Action about speeding up the indexing process.
>
> 1) Simply increase the mergeFactor and RAMBufferSizeMB.
> 2) Using RAMDirectory as a buffer (perhaps even several in parallel) and later merging it using addIndexes to FSDirectory.
>
> So my question is the following:
> In case I have only 1 thread with RAMDirectory - is that pretty much the same as method 1? Since it's in memory anyhow for large mergeFactor and large RAMBufferSizeMB.
>
> Maxim
>
>
> ________________________________



-- 
Lance Norskog
goksron@gmail.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org