memory usage keep increase

[Yongtao Liu <yl...@commvault.com>]

Hi all,

I saw one issue is ram usage keep increase when we run query.
After look in the code, looks like Lucene use MMapDirectory to map index file to ram.

According to http://lucene.apache.org/java/3_1_0/api/core/org/apache/lucene/store/MMapDirectory.html comments, it will use lot of memory.
NOTE: memory mapping uses up a portion of the virtual memory address space in your process equal to the size of the file being mapped. Before using this class, be sure your have plenty of virtual address space, e.g. by using a 64 bit JRE, or a 32 bit JRE with indexes that are guaranteed to fit within the address space.

So, my understanding is solr request physical RAM >= index file size, is it right?

Yongtao


******************Legal Disclaimer***************************
"This communication may contain confidential and privileged
material for the sole use of the intended recipient. Any
unauthorized review, use or distribution by others is strictly
prohibited. If you have received the message in error, please
advise the sender by reply email and delete the message. Thank
you."
*********************************************************

RE: memory usage keep increase

[Yongtao Liu <yl...@commvault.com>]

Erick,

Thanks for your reply.

Yes, "virtual memory" does not mean physical memory.
But if when "virtual memory" >> physical memory, the system will change to slow, since lots for paging request happen.

Yongtao
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com] 
Sent: Tuesday, November 15, 2011 8:37 AM
To: solr-user@lucene.apache.org
Subject: Re: memory usage keep increase

I'm pretty sure not. The words "virtual memory address space" is important here, that's not physical memory...

Best
Erick

On Mon, Nov 14, 2011 at 11:55 AM, Yongtao Liu <yl...@commvault.com> wrote:
> Hi all,
>
> I saw one issue is ram usage keep increase when we run query.
> After look in the code, looks like Lucene use MMapDirectory to map index file to ram.
>
> According to http://lucene.apache.org/java/3_1_0/api/core/org/apache/lucene/store/MMapDirectory.html comments, it will use lot of memory.
> NOTE: memory mapping uses up a portion of the virtual memory address space in your process equal to the size of the file being mapped. Before using this class, be sure your have plenty of virtual address space, e.g. by using a 64 bit JRE, or a 32 bit JRE with indexes that are guaranteed to fit within the address space.
>
> So, my understanding is solr request physical RAM >= index file size, is it right?
>
> Yongtao
>
>
> ******************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message in error, please advise the sender by reply email 
> and delete the message. Thank you."
> *********************************************************

Re: memory usage keep increase

[Erick Erickson <er...@gmail.com>]

I'm pretty sure not. The words "virtual memory address space" is important
here, that's not physical memory...

Best
Erick

On Mon, Nov 14, 2011 at 11:55 AM, Yongtao Liu <yl...@commvault.com> wrote:
> Hi all,
>
> I saw one issue is ram usage keep increase when we run query.
> After look in the code, looks like Lucene use MMapDirectory to map index file to ram.
>
> According to http://lucene.apache.org/java/3_1_0/api/core/org/apache/lucene/store/MMapDirectory.html comments, it will use lot of memory.
> NOTE: memory mapping uses up a portion of the virtual memory address space in your process equal to the size of the file being mapped. Before using this class, be sure your have plenty of virtual address space, e.g. by using a 64 bit JRE, or a 32 bit JRE with indexes that are guaranteed to fit within the address space.
>
> So, my understanding is solr request physical RAM >= index file size, is it right?
>
> Yongtao
>
>
> ******************Legal Disclaimer***************************
> "This communication may contain confidential and privileged
> material for the sole use of the intended recipient. Any
> unauthorized review, use or distribution by others is strictly
> prohibited. If you have received the message in error, please
> advise the sender by reply email and delete the message. Thank
> you."
> *********************************************************