You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Utkarsh Sengar <ut...@gmail.com> on 2013/04/17 01:52:55 UTC
Solr's physical memory and JVM memory
Hello,
I have setup a solr4 instance (just one node) and I see this memory pattern:
[image: Inline image 1]
Physical memory is nearly full and JVM memory is ok. I have ~40M documents
(where 1 document=1KB) indexed and in production env I am planning to setup
2 solr cloud nodes.
So I have 2 questions:
1. What is the recommended memory for those 2 nodes?
2. I am not sure what does Physical memory mean in context to solr. My
understanding of the physical memory is the actual RAM in my machine and
'top' says that I have used just 4.6GB or 23.7GB. Why is Solr admin
reporting that I have used 22.84GB out of 23.7GB?
--
Thanks,
-Utkarsh
Re: Solr's physical memory and JVM memory
Posted by Utkarsh Sengar <ut...@gmail.com>.
My bad about the attachment, there you go: http://i.imgur.com/XKtw32K.png
Thanks for the details answer and that helps alot.
Thank,
-Utkarsh
On Tue, Apr 16, 2013 at 9:48 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 4/16/2013 10:01 PM, Otis Gospodnetic wrote:
> > Not sure if it's just me, but I'm not seeing your inlined image.
>
> It's not just you.
>
> > On Tue, Apr 16, 2013 at 7:52 PM, Utkarsh Sengar <utkarsh2012@gmail.com
> >wrote:
> >> So I have 2 questions:
> >> 1. What is the recommended memory for those 2 nodes?
> >> 2. I am not sure what does Physical memory mean in context to solr. My
> >> understanding of the physical memory is the actual RAM in my machine and
> >> 'top' says that I have used just 4.6GB or 23.7GB. Why is Solr admin
> >> reporting that I have used 22.84GB out of 23.7GB?
>
> Attachments don't work well on mailing lists. We can't see your image.
> Best to put the file on the Internet somewhere (like dropbox or another
> file sharing site) and include the public link. After you get an answer
> to your question, you can remove the file.
>
> Answers to your two questions:
>
> 1) A good rule of thumb is that you want to have enough RAM to equal or
> exceed the sum of two things: The amount of memory that your programs
> take (including the max heap setting you give to Solr), and the size of
> your Solr index(es) stored on that server. You may be able to get away
> with less memory than this, but you do want to have enough memory for a
> sizable chunk of your on-disk index. Example: If Solr is the only major
> program running on the machine, you give Solr a 4GB heap, and your index
> is 20GB, an ideal setup would have at least 24GB of RAM.
>
> 2) You are seeing the result of the way that all modern operating
> systems work. The extra memory that is not being currently used by
> programs is borrowed by the operating system to cache data from your
> disk into RAM, so that frequently accessed data will not have be read
> from the disk. Reading from main memory is many orders of magnitude
> faster than reading from disk. The memory that is being used for the
> disk cache (on top it shows up as 'cached') is instantly made available
> to programs that request it.
>
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> 2a) Operating systems like Linux tell you the truth about the OS using
> excess memory for the disk cache. With the most basic information
> tools, Windows tells you a semi-lie and will report that memory as free.
> The newest versions of Windows seem to have gotten the hint and do
> include tools that will give you the true picture.
>
> 2b) For good performance, Solr is extremely reliant on having a big
> enough disk cache so that reads from disk are rare. This is the case
> for most other programs too, actually.
>
> Thanks,
> Shawn
>
>
--
Thanks,
-Utkarsh
Re: Solr's physical memory and JVM memory
Posted by Shawn Heisey <so...@elyograg.org>.
On 4/16/2013 10:01 PM, Otis Gospodnetic wrote:
> Not sure if it's just me, but I'm not seeing your inlined image.
It's not just you.
> On Tue, Apr 16, 2013 at 7:52 PM, Utkarsh Sengar <ut...@gmail.com>wrote:
>> So I have 2 questions:
>> 1. What is the recommended memory for those 2 nodes?
>> 2. I am not sure what does Physical memory mean in context to solr. My
>> understanding of the physical memory is the actual RAM in my machine and
>> 'top' says that I have used just 4.6GB or 23.7GB. Why is Solr admin
>> reporting that I have used 22.84GB out of 23.7GB?
Attachments don't work well on mailing lists. We can't see your image.
Best to put the file on the Internet somewhere (like dropbox or another
file sharing site) and include the public link. After you get an answer
to your question, you can remove the file.
Answers to your two questions:
1) A good rule of thumb is that you want to have enough RAM to equal or
exceed the sum of two things: The amount of memory that your programs
take (including the max heap setting you give to Solr), and the size of
your Solr index(es) stored on that server. You may be able to get away
with less memory than this, but you do want to have enough memory for a
sizable chunk of your on-disk index. Example: If Solr is the only major
program running on the machine, you give Solr a 4GB heap, and your index
is 20GB, an ideal setup would have at least 24GB of RAM.
2) You are seeing the result of the way that all modern operating
systems work. The extra memory that is not being currently used by
programs is borrowed by the operating system to cache data from your
disk into RAM, so that frequently accessed data will not have be read
from the disk. Reading from main memory is many orders of magnitude
faster than reading from disk. The memory that is being used for the
disk cache (on top it shows up as 'cached') is instantly made available
to programs that request it.
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
2a) Operating systems like Linux tell you the truth about the OS using
excess memory for the disk cache. With the most basic information
tools, Windows tells you a semi-lie and will report that memory as free.
The newest versions of Windows seem to have gotten the hint and do
include tools that will give you the true picture.
2b) For good performance, Solr is extremely reliant on having a big
enough disk cache so that reads from disk are rare. This is the case
for most other programs too, actually.
Thanks,
Shawn
Re: Solr's physical memory and JVM memory
Posted by Otis Gospodnetic <ot...@gmail.com>.
Not sure if it's just me, but I'm not seeing your inlined image.
Otis
--
SOLR Performance Monitoring - http://sematext.com/spm/index.html
On Tue, Apr 16, 2013 at 7:52 PM, Utkarsh Sengar <ut...@gmail.com>wrote:
> Hello,
>
> I have setup a solr4 instance (just one node) and I see this memory
> pattern:
>
> [image: Inline image 1]
>
> Physical memory is nearly full and JVM memory is ok. I have ~40M documents
> (where 1 document=1KB) indexed and in production env I am planning to setup
> 2 solr cloud nodes.
>
> So I have 2 questions:
> 1. What is the recommended memory for those 2 nodes?
> 2. I am not sure what does Physical memory mean in context to solr. My
> understanding of the physical memory is the actual RAM in my machine and
> 'top' says that I have used just 4.6GB or 23.7GB. Why is Solr admin
> reporting that I have used 22.84GB out of 23.7GB?
>
> --
> Thanks,
> -Utkarsh
>