You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Cheng <zh...@gmail.com> on 2014/05/07 15:46:49 UTC
Can RAMDirectory work for gigabyte data which needs refreshing of the
index all the time?
Hi,
I have an index of multiple gigabytes which serves 5-10 threads and needs
refreshing very often. I wonder if RAMDirectory is the good candidate for
this purpose. If not, what kind of directory is better?
Thanks,
Cheng
Re: Can RAMDirectory work for gigabyte data which needs refreshing
of the index all the time?
Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Wed, 2014-05-07 at 15:46 +0200, Cheng wrote:
> I have an index of multiple gigabytes which serves 5-10 threads and needs
> refreshing very often. I wonder if RAMDirectory is the good candidate for
> this purpose. If not, what kind of directory is better?
RAMDirectory will probably give you poor performance:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Stick to MMapDirectory: As you are considering using a RAMDirectory,
your index must be smaller than the amount of free RAM, which means that
everything will be fully cached and fast.
- Toke Eskildsen, State and University Library, Denmark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Can RAMDirectory work for gigabyte data which needs refreshing of the index all the time?
Posted by Steven Schlansker <st...@likeness.com>.
On May 7, 2014, at 6:46 AM, Cheng <zh...@gmail.com> wrote:
>
> I have an index of multiple gigabytes which serves 5-10 threads and needs
> refreshing very often. I wonder if RAMDirectory is the good candidate for
> this purpose. If not, what kind of directory is better?
We found that loading and unloading RAMDirectory from the Java heap causes crushing
heap pressure. When you switch over, we regularly had GC pauses in the tens of seconds.
We switched back to using mmap-ed files (which IIRC is the default for 64-bit Linux).
As long as you have enough memory for buffer cache, it ends up in RAM and is quite fast.
We never looked back.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org