Virtual Memory usage increases beyond Xmx with Solr 3.3

[Nikhil Chhaochharia <ni...@yahoo.com>]

Hi,

I am using Ubuntu 10.04 64-bit with Sun Java (build 1.6.0_24-b07) and Tomcat (6.0.24).  Sun Java and Tomcat have been installed using apt-get from the Ubuntu/Canonical repositories.  I run Tomcat with -Xmx4g and have been using using Solr 1.4/3.0/3.1/3.2 without any problems.

However, if I upgrade to Solr 3.3, then the Virtual Memory of the Tomcat process increases to roughly the index size (70GB).  Any ideas why
 this is happening?

Thanks,
Nikhil

Re: Virtual Memory usage increases beyond Xmx with Solr 3.3

[Nikhil Chhaochharia <ni...@yahoo.com>]

Thanks, MMapDirectory was the reason - it was made the default in Lucene 3.3

http://lucene.apache.org/java/3_3_0/changes/Changes.html#3.3.0.changes_in_runtime_behavior
https://issues.apache.org/jira/browse/LUCENE-3198



________________________________
From: Toke Eskildsen
To: "solr-user@lucene.apache.org"; Nikhil Chhaochharia
Sent: Friday, 8 July 2011 2:42 PM
Subject: Re: Virtual Memory usage increases beyond Xmx with Solr 3.3

On Fri, 2011-07-08 at 07:12 +0200, Nikhil Chhaochharia wrote:
> However, if I upgrade to Solr 3.3, then the Virtual Memory of the Tomcat process increases to roughly the index size (70GB).  Any ideas why
>  this is happening?

Maybe you switched to MMapDirectory?
http://lucene.apache.org/java/3_3_0/api/core/org/apache/lucene/store/MMapDirectory.html

I haven't verified this, but it seems probable that the allocated but
unused address space that MMapDirectory requires is reported as part of
the statistics for the Java process.

It looks strange, but a large reported VM-size is not in itself a
problem. If you look in /proc/pid/status, VmRSS tells you the physical
memory usage and VmSwap how much is swapped. If the sum of those is
around the 4GB you specified, then the load on the machine is as you
requested.


Caveat lector: I am not by any means a kernel hacker. I might have
misunderstood the meaning of the values in proc/pid/status.

Re: Virtual Memory usage increases beyond Xmx with Solr 3.3

[Toke Eskildsen <te...@statsbiblioteket.dk>]

On Fri, 2011-07-08 at 07:12 +0200, Nikhil Chhaochharia wrote:
> However, if I upgrade to Solr 3.3, then the Virtual Memory of the Tomcat process increases to roughly the index size (70GB).  Any ideas why
>  this is happening?

Maybe you switched to MMapDirectory?
http://lucene.apache.org/java/3_3_0/api/core/org/apache/lucene/store/MMapDirectory.html

I haven't verified this, but it seems probable that the allocated but
unused address space that MMapDirectory requires is reported as part of
the statistics for the Java process.

It looks strange, but a large reported VM-size is not in itself a
problem. If you look in /proc/pid/status, VmRSS tells you the physical
memory usage and VmSwap how much is swapped. If the sum of those is
around the 4GB you specified, then the load on the machine is as you
requested.


Caveat lector: I am not by any means a kernel hacker. I might have
misunderstood the meaning of the values in proc/pid/status.