You are viewing a plain text version of this content. The canonical link for it is here.

Posted to java-user@lucene.apache.org by Cheng <zh...@gmail.com> on 2012/06/04 16:08:08 UTC

RAMDirectory unexpectedly slows

Hi,

My apps need to read from and write to some big indexes frequently. So I
use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
size.

I notice that the speed of reading and writing unexpectedly slows as the
size of the indexes increases. Since the usage of RAM is less than 20%, I
think by default the RAMDirectory doesn't take advantage of the memory I
assigned to JVM.

What are the steps to improve the reading and writing speed of
RAMDirectory?

Thanks!
Jeff

Re: RAMDirectory unexpectedly slows

Posted by Jason Rutherglen <ja...@gmail.com>.

> What about the ByteBufferDirectory? Can this specific directory utilize the
> 2GB memory I grant to the app?

BBD places the byte objects outside of the heap, so increasing the
heap size is only going to rob the system IO cache.  With Lucene the
heap is only used for field caches and the terms dictionary index.

On Mon, Jun 4, 2012 at 11:04 AM, Cheng <zh...@gmail.com> wrote:
> Please shed more insight into the difference between JVM heap size and the
> memory size used by Lucene.
>
> What I am getting at is that no matter however much ram I give my apps,
> Lucene can't utilize it. Is that right?
>
> What about the ByteBufferDirectory? Can this specific directory utilize the
> 2GB memory I grant to the app?
>
> On Mon, Jun 4, 2012 at 10:58 PM, Jason Rutherglen <
> jason.rutherglen@gmail.com> wrote:
>
>> If you want the index to be stored completely in RAM, there is the
>> ByteBuffer directory [1].  Though I do not see the point in putting an
>> index in RAM, it will be cached in RAM regardless in the OS system IO
>> cache.
>>
>> 1.
>> https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/apache/lucene/store/bytebuffer/ByteBufferDirectory.java
>>
>> On Mon, Jun 4, 2012 at 10:55 AM, Cheng <zh...@gmail.com> wrote:
>> > My indexes are 500MB+. So it seems like that RAMDirectory is not good for
>> > that big a size.
>> >
>> > My challenge, on the other side, is that I need to update the indexes
>> very
>> > frequently. So, do you think  MMapDirectory is the solution?
>> >
>> > Thanks.
>> >
>> > On Mon, Jun 4, 2012 at 10:30 PM, Jack Krupansky <jack@basetechnology.com
>> >wrote:
>> >
>> >> From the javadoc for RAMDirectory:
>> >>
>> >> "Warning: This class is not intended to work with huge indexes.
>> Everything
>> >> beyond several hundred megabytes will waste resources (GC cycles),
>> because
>> >> it uses an internal buffer size of 1024 bytes, producing millions of
>> >> byte[1024] arrays. This class is optimized for small memory-resident
>> >> indexes. It also has bad concurrency on multithreaded environments.
>> >>
>> >> It is recommended to materialize large indexes on disk and use
>> >> MMapDirectory, which is a high-performance directory implementation
>> working
>> >> directly on the file system cache of the operating system, so copying
>> data
>> >> to Java heap space is not useful."
>> >>
>> >> -- Jack Krupansky
>> >>
>> >> -----Original Message----- From: Cheng
>> >> Sent: Monday, June 04, 2012 10:08 AM
>> >> To: java-user@lucene.apache.org
>> >> Subject: RAMDirectory unexpectedly slows
>> >>
>> >>
>> >> Hi,
>> >>
>> >> My apps need to read from and write to some big indexes frequently. So I
>> >> use RAMDirectory instead of FSDirectory, and give JVM about 2GB memory
>> >> size.
>> >>
>> >> I notice that the speed of reading and writing unexpectedly slows as the
>> >> size of the indexes increases. Since the usage of RAM is less than 20%,
>> I
>> >> think by default the RAMDirectory doesn't take advantage of the memory I
>> >> assigned to JVM.
>> >>
>> >> What are the steps to improve the reading and writing speed of
>> >> RAMDirectory?
>> >>
>> >> Thanks!
>> >> Jeff
>> >>
>> >>
>> ------------------------------**------------------------------**---------
>> >> To unsubscribe, e-mail: java-user-unsubscribe@lucene.**apache.org<
>> java-user-unsubscribe@lucene.apache.org>
>> >> For additional commands, e-mail: java-user-help@lucene.apache.**org<
>> java-user-help@lucene.apache.org>
>> >>
>> >>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org