You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Rohit <ro...@in-rev.com> on 2011/12/10 05:54:33 UTC
Virtual Memory very high
Hi All,
Don't know if this question is directly related to this forum, I am running
Solr in Tomcat on linux server. The moment I start tomcat the virtual memory
shown using TOP command goes to its max 31.1G and then remains there.
Is this the right behaviour, why is the virtual memory usage so high. I have
36GB of ram on the server.
Tasks: 309 total, 1 running, 308 sleeping, 0 stopped, 0 zombie
Cpu(s): 19.1%us, 0.2%sy, 0.0%ni, 79.3%id, 1.2%wa, 0.0%hi, 0.2%si,
0.0%st
Mem: 49555260k total, 36152224k used, 13403036k free, 121612k buffers
Swap: 999416k total, 0k used, 999416k free, 5409052k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2741 mysql 20 0 6412m 5.8g 6380 S 182 12.3 108:07.45 mysqld
2814 root 20 0 31.1g 22g 9716 S 100 46.6 375:51.70 java
1765 root 20 0 12.2g 285m 9488 S 2 0.6 3:52.59 java
3591 root 20 0 19352 1576 1068 R 0 0.0 0:00.24 top
1 root 20 0 23684 1908 1276 S 0 0.0 0:06.21 init
Regards,
Rohit
RE: Virtual Memory very high
Posted by Rohit <ro...@in-rev.com>.
Thanks Yurykats.
Regards,
Rohit
Mobile: +91-9901768202
About Me: http://about.me/rohitg
-----Original Message-----
From: Dmitry Kan [mailto:dmitry.kan@gmail.com]
Sent: 13 December 2011 11:17
To: solr-user@lucene.apache.org
Subject: Re: Virtual Memory very high
If you allow me to chime in, is there a way to check for which
DirectoryFactory is in use, if
${solr.directoryFactory:solr.StandardDirectoryFactory} has been configured?
Dmitry
2011/12/12 Yury Kats <yu...@yahoo.com>
> On 12/11/2011 4:57 AM, Rohit wrote:
> > What are the difference in the different DirectoryFactory?
>
>
> http://lucene.apache.org/java/3_3_0/api/core/org/apache/lucene/store/MMapDirectory.html
>
> http://lucene.apache.org/java/3_3_0/api/core/org/apache/lucene/store/NIOFSDirectory.html
>
Re: Virtual Memory very high
Posted by Erick Erickson <er...@gmail.com>.
Why do you care about virtual memory? It's after all, virtual. You can
allocate as much as you want.
For instance, MMapDirectory maps a load of virtual memory, but that
has little relation to how much physical memory is being used. Consider
looking at your app with something like jConsole and seeing how much
physical memory is being used before you worry about this issue.
Best
Erick
On Mon, Apr 2, 2012 at 4:56 AM, Suneel <pa...@gmail.com> wrote:
> Hello Everyone,
>
> On window server.
>
> I am facing same problem during indexing my memory consumption going very
> high based on above discussion i checked in my Solrconfig.xml file and found
> that "directoryFactory" not configured yet. if i configuring
> directoryfactory then its will help me reduce the consumption of memory.
>
> i think below configuration used for linex server.
>
> <directoryFactory name="DirectoryFactory"
> class="${solr.directoryFactory:solr.NIOFSDirectoryFactory}"/>
>
> what will be best option for window server which solve my problem.
>
> Please suggest me.
>
>
>
>
>
>
> -----
> Regards,
>
> Suneel Pandey
> Sr. Software Developer
> --
> View this message in context: http://lucene.472066.n3.nabble.com/Virtual-Memory-very-high-tp3574817p3877097.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: Virtual Memory very high
Posted by Suneel <pa...@gmail.com>.
Hello Everyone,
On window server.
I am facing same problem during indexing my memory consumption going very
high based on above discussion i checked in my Solrconfig.xml file and found
that "directoryFactory" not configured yet. if i configuring
directoryfactory then its will help me reduce the consumption of memory.
i think below configuration used for linex server.
<directoryFactory name="DirectoryFactory"
class="${solr.directoryFactory:solr.NIOFSDirectoryFactory}"/>
what will be best option for window server which solve my problem.
Please suggest me.
-----
Regards,
Suneel Pandey
Sr. Software Developer
--
View this message in context: http://lucene.472066.n3.nabble.com/Virtual-Memory-very-high-tp3574817p3877097.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Virtual Memory very high
Posted by Yury Kats <yu...@yahoo.com>.
On 12/13/2011 6:16 AM, Dmitry Kan wrote:
> If you allow me to chime in, is there a way to check for which
> DirectoryFactory is in use, if
> ${solr.directoryFactory:solr.StandardDirectoryFactory} has been configured?
I think you can get the currently used factory in a Luke response, if you hit your Solr server with a Luke request,
eg http://localhost:8983/solr/admin/luke
>
> Dmitry
>
> 2011/12/12 Yury Kats <yu...@yahoo.com>
>
>> On 12/11/2011 4:57 AM, Rohit wrote:
>>> What are the difference in the different DirectoryFactory?
>>
>>
>> http://lucene.apache.org/java/3_3_0/api/core/org/apache/lucene/store/MMapDirectory.html
>>
>> http://lucene.apache.org/java/3_3_0/api/core/org/apache/lucene/store/NIOFSDirectory.html
>>
>
Re: Virtual Memory very high
Posted by Dmitry Kan <dm...@gmail.com>.
If you allow me to chime in, is there a way to check for which
DirectoryFactory is in use, if
${solr.directoryFactory:solr.StandardDirectoryFactory} has been configured?
Dmitry
2011/12/12 Yury Kats <yu...@yahoo.com>
> On 12/11/2011 4:57 AM, Rohit wrote:
> > What are the difference in the different DirectoryFactory?
>
>
> http://lucene.apache.org/java/3_3_0/api/core/org/apache/lucene/store/MMapDirectory.html
>
> http://lucene.apache.org/java/3_3_0/api/core/org/apache/lucene/store/NIOFSDirectory.html
>
Re: Virtual Memory very high
Posted by Yury Kats <yu...@yahoo.com>.
On 12/11/2011 4:57 AM, Rohit wrote:
> What are the difference in the different DirectoryFactory?
http://lucene.apache.org/java/3_3_0/api/core/org/apache/lucene/store/MMapDirectory.html
http://lucene.apache.org/java/3_3_0/api/core/org/apache/lucene/store/NIOFSDirectory.html
RE: Virtual Memory very high
Posted by Rohit <ro...@in-rev.com>.
What are the difference in the different DirectoryFactory?
Regards,
Rohit
Mobile: +91-9901768202
About Me: http://about.me/rohitg
-----Original Message-----
From: Yury Kats [mailto:yurykats@yahoo.com]
Sent: 10 December 2011 12:11
To: solr-user@lucene.apache.org
Subject: Re: Virtual Memory very high
On 12/9/2011 11:54 PM, Rohit wrote:
> Hi All,
>
>
>
> Don't know if this question is directly related to this forum, I am
running
> Solr in Tomcat on linux server. The moment I start tomcat the virtual
memory
> shown using TOP command goes to its max 31.1G and then remains there.
>
>
>
> Is this the right behaviour, why is the virtual memory usage so high. I
have
> 36GB of ram on the server.
To limit VIRT memory, change DirectoryFactory in the solrconfig.xml to use
solr.NIOFSDirectoryFactory.
Re: Virtual Memory very high
Posted by Yury Kats <yu...@yahoo.com>.
On 12/9/2011 11:54 PM, Rohit wrote:
> Hi All,
>
>
>
> Don't know if this question is directly related to this forum, I am running
> Solr in Tomcat on linux server. The moment I start tomcat the virtual memory
> shown using TOP command goes to its max 31.1G and then remains there.
>
>
>
> Is this the right behaviour, why is the virtual memory usage so high. I have
> 36GB of ram on the server.
To limit VIRT memory, change DirectoryFactory in the solrconfig.xml to use
solr.NIOFSDirectoryFactory.
Re: Virtual Memory very high
Posted by Michael McCandless <lu...@mikemccandless.com>.
Are you seeing a real problem here, besides just being alarmed by the
big numbers from top?
Consumption of virtual memory by itself is basically harmless, as long
as you're not running up against any of the OS limits (and, you're
running a 64 bit JVM).
This is just "top" telling you that you've mapped large files into the
virtual memory space.
It's not telling you that you don't have any RAM left... virtual
memory is different from RAM.
In my tests, generally MMapDirectory gives faster search performance
than NIOFSDirectory... so unless there's an actual issue, I would
recommend sticking with MMapDirectory.
Mike McCandless
http://blog.mikemccandless.com
On Fri, Dec 9, 2011 at 11:54 PM, Rohit <ro...@in-rev.com> wrote:
> Hi All,
>
>
>
> Don't know if this question is directly related to this forum, I am running
> Solr in Tomcat on linux server. The moment I start tomcat the virtual memory
> shown using TOP command goes to its max 31.1G and then remains there.
>
>
>
> Is this the right behaviour, why is the virtual memory usage so high. I have
> 36GB of ram on the server.
>
>
>
> Tasks: 309 total, 1 running, 308 sleeping, 0 stopped, 0 zombie
>
> Cpu(s): 19.1%us, 0.2%sy, 0.0%ni, 79.3%id, 1.2%wa, 0.0%hi, 0.2%si,
> 0.0%st
>
> Mem: 49555260k total, 36152224k used, 13403036k free, 121612k buffers
>
> Swap: 999416k total, 0k used, 999416k free, 5409052k cached
>
>
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> 2741 mysql 20 0 6412m 5.8g 6380 S 182 12.3 108:07.45 mysqld
>
> 2814 root 20 0 31.1g 22g 9716 S 100 46.6 375:51.70 java
>
> 1765 root 20 0 12.2g 285m 9488 S 2 0.6 3:52.59 java
>
> 3591 root 20 0 19352 1576 1068 R 0 0.0 0:00.24 top
>
> 1 root 20 0 23684 1908 1276 S 0 0.0 0:06.21 init
>
>
>
> Regards,
>
> Rohit
>
>
>