You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Steven White <sw...@gmail.com> on 2015/12/08 20:54:01 UTC
Solr memory usage
Hi folks,
My index size on disk (optimized) is 20 GB (single core, single index). I
have a system with 64 GB of RAM. I start Solr with 24 GB of RAM.
I have run load tests (up to 100 concurrent users) for hours where each
user issuing unique searches (the same search is never executed again for
at least 30 minute since it was last executed). In all tests I run, Solr's
JVM memory never goes over 10 GB (monitoring http://localhost:8983/).
I read over and over, for optimal performance, Solr should be given enough
RAM to hold the index in memory. Well, I have done that and some but yet I
don't see Solr using up that whole RAM. What am I doing wrong? Is my test
at fault? I doubled the test load (number of users) and didn't see much of
a difference with RAM usage but yet my search performance went down (takes
about 40% longer now). I run my tests again but this time with only 12 GB
of RAM given to Solr. Test result didn't differ much from the 24 GB run
and Solr never used more than 10 GB of RAM.
Can someone help me understand this? I don't want to give Solr RAM that it
won't use.
PS: This is simply search tests, there is no update to the index at all.
Thanks in advanced.
Steve
Re: Solr memory usage
Posted by Otis Gospodnetić <ot...@gmail.com>.
Hi Steve,
Fluctuation is OK. 100% utilization for more than a moment is not :)
Not sure what tool(s) you use for monitoring your Solr servers, but look
under "JVM Pool Utilization" in SPM if you're using SPM.
Or this live demo of a Solr system:
* click on https://apps.sematext.com/demo to get into the demo account
* look at "JVM Pool Utilization" on
https://apps.sematext.com/spm-reports/mainPage.do?selectedApplication=1704&r=poolReportPage×tamp=1449865787801&stickyFiltersOff=false
And on that JVM Pool Size chart on top of the page you will see giant saw
pattern.... which is a healthy sign :)
HTH
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
On Wed, Dec 9, 2015 at 9:56 AM, Steven White <sw...@gmail.com> wrote:
> Thanks Erick!! Your summary and the blog by Uwe (thank you too Uwe) are
> very helpful.
>
> A follow up question. I also noticed the "JVM-Memory" report off Solr's
> home page is fluctuating. I expect some fluctuation, but it kinda worries
> me when it fluctuates up / down in a range of 4 GB and maybe more. I.e.:
> at times it is at 5 GB and other times it is at 10 GB (this is while I'm
> running my search tests). What does such high fluctuation means?
>
> If it helps, Solr's "JVM-Memory" report states 2.5 GB usage when Solr is
> first started and before I run any search on it. I'm taking this as my
> base startup memory usage.
>
> Steve
>
> On Tue, Dec 8, 2015 at 3:17 PM, Erick Erickson <er...@gmail.com>
> wrote:
>
> > You're doing nothing wrong, that particular bit of advice has
> > always needed a bit of explanation.
> >
> > Solr (well, actually Lucene) uses MMapDirectory for much of
> > the index structure which uses the OS memory rather than
> > the JVM heap. See Uwe's excellent:
> >
> > http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
> >
> > Plus, the size on disk includes the stored data, which is in the *.fdt
> > files in data/index. Very little of the stored data is kept in the JVM
> > so that's another reason your Java heap may be smaller than
> > your raw index size on disk.
> >
> > The advice about fitting your entire index into memory really has
> > the following caveats (at least).
> > 1> "memory" includes the OS memory available to the process
> > 2> The size of the index on disk is misleading, the *.fdt files
> > should be subtracted in order to get a truer picture.
> > 3> Both Solr and Lucene create structures in the Java JVM
> > that are _not_ reflected in the size on disk.
> >
> > <1> and <2> mean the JVM memory necessary is smaller
> > than the size on disk.
> >
> > <3> means the JVM memory will be larger than.
> >
> > So you're doing the right thing, testing and seeing what you
> > _really_ need. I'd pretty much take your test, add some
> > padding and consider it good. You're _not_ doing the
> > really bad thing of using the same query over and over
> > again and hoping <G>.
> >
> > Best,
> > Erick
> >
> >
> > On Tue, Dec 8, 2015 at 11:54 AM, Steven White <sw...@gmail.com>
> > wrote:
> > > Hi folks,
> > >
> > > My index size on disk (optimized) is 20 GB (single core, single index).
> > I
> > > have a system with 64 GB of RAM. I start Solr with 24 GB of RAM.
> > >
> > > I have run load tests (up to 100 concurrent users) for hours where each
> > > user issuing unique searches (the same search is never executed again
> for
> > > at least 30 minute since it was last executed). In all tests I run,
> > Solr's
> > > JVM memory never goes over 10 GB (monitoring http://localhost:8983/).
> > >
> > > I read over and over, for optimal performance, Solr should be given
> > enough
> > > RAM to hold the index in memory. Well, I have done that and some but
> > yet I
> > > don't see Solr using up that whole RAM. What am I doing wrong? Is my
> > test
> > > at fault? I doubled the test load (number of users) and didn't see
> much
> > of
> > > a difference with RAM usage but yet my search performance went down
> > (takes
> > > about 40% longer now). I run my tests again but this time with only 12
> > GB
> > > of RAM given to Solr. Test result didn't differ much from the 24 GB
> run
> > > and Solr never used more than 10 GB of RAM.
> > >
> > > Can someone help me understand this? I don't want to give Solr RAM
> that
> > it
> > > won't use.
> > >
> > > PS: This is simply search tests, there is no update to the index at
> all.
> > >
> > > Thanks in advanced.
> > >
> > > Steve
> >
>
Re: Solr memory usage
Posted by Shawn Heisey <ap...@elyograg.org>.
On 12/9/2015 7:56 AM, Steven White wrote:
> Thanks Erick!! Your summary and the blog by Uwe (thank you too Uwe) are
> very helpful.
>
> A follow up question. I also noticed the "JVM-Memory" report off Solr's
> home page is fluctuating. I expect some fluctuation, but it kinda worries
> me when it fluctuates up / down in a range of 4 GB and maybe more. I.e.:
> at times it is at 5 GB and other times it is at 10 GB (this is while I'm
> running my search tests). What does such high fluctuation means?
>
> If it helps, Solr's "JVM-Memory" report states 2.5 GB usage when Solr is
> first started and before I run any search on it. I'm taking this as my
> base startup memory usage.
The heap usage at any particular instant in time (even right after
startup) is nearly useless information. To reach any useful conclusions
and change your heap size based on those conclusions, heap usage must be
tracked (and ideally graphed) for several minutes or hours, sampling no
less frequently than about every five or ten seconds -- exactly what
programs like JConsole (included with the Java JDK) do. You will want
to do the tracking/graphing during your your heaviest usage for both
queries and indexing.
See the "How much heap space do I need?" section here for some
relatively vague pointers:
https://wiki.apache.org/solr/SolrPerformanceProblems#How_much_heap_space_do_I_need.3F
Thanks,
Shawn
Re: Solr memory usage
Posted by Steven White <sw...@gmail.com>.
Thanks Erick!! Your summary and the blog by Uwe (thank you too Uwe) are
very helpful.
A follow up question. I also noticed the "JVM-Memory" report off Solr's
home page is fluctuating. I expect some fluctuation, but it kinda worries
me when it fluctuates up / down in a range of 4 GB and maybe more. I.e.:
at times it is at 5 GB and other times it is at 10 GB (this is while I'm
running my search tests). What does such high fluctuation means?
If it helps, Solr's "JVM-Memory" report states 2.5 GB usage when Solr is
first started and before I run any search on it. I'm taking this as my
base startup memory usage.
Steve
On Tue, Dec 8, 2015 at 3:17 PM, Erick Erickson <er...@gmail.com>
wrote:
> You're doing nothing wrong, that particular bit of advice has
> always needed a bit of explanation.
>
> Solr (well, actually Lucene) uses MMapDirectory for much of
> the index structure which uses the OS memory rather than
> the JVM heap. See Uwe's excellent:
>
> http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
>
> Plus, the size on disk includes the stored data, which is in the *.fdt
> files in data/index. Very little of the stored data is kept in the JVM
> so that's another reason your Java heap may be smaller than
> your raw index size on disk.
>
> The advice about fitting your entire index into memory really has
> the following caveats (at least).
> 1> "memory" includes the OS memory available to the process
> 2> The size of the index on disk is misleading, the *.fdt files
> should be subtracted in order to get a truer picture.
> 3> Both Solr and Lucene create structures in the Java JVM
> that are _not_ reflected in the size on disk.
>
> <1> and <2> mean the JVM memory necessary is smaller
> than the size on disk.
>
> <3> means the JVM memory will be larger than.
>
> So you're doing the right thing, testing and seeing what you
> _really_ need. I'd pretty much take your test, add some
> padding and consider it good. You're _not_ doing the
> really bad thing of using the same query over and over
> again and hoping <G>.
>
> Best,
> Erick
>
>
> On Tue, Dec 8, 2015 at 11:54 AM, Steven White <sw...@gmail.com>
> wrote:
> > Hi folks,
> >
> > My index size on disk (optimized) is 20 GB (single core, single index).
> I
> > have a system with 64 GB of RAM. I start Solr with 24 GB of RAM.
> >
> > I have run load tests (up to 100 concurrent users) for hours where each
> > user issuing unique searches (the same search is never executed again for
> > at least 30 minute since it was last executed). In all tests I run,
> Solr's
> > JVM memory never goes over 10 GB (monitoring http://localhost:8983/).
> >
> > I read over and over, for optimal performance, Solr should be given
> enough
> > RAM to hold the index in memory. Well, I have done that and some but
> yet I
> > don't see Solr using up that whole RAM. What am I doing wrong? Is my
> test
> > at fault? I doubled the test load (number of users) and didn't see much
> of
> > a difference with RAM usage but yet my search performance went down
> (takes
> > about 40% longer now). I run my tests again but this time with only 12
> GB
> > of RAM given to Solr. Test result didn't differ much from the 24 GB run
> > and Solr never used more than 10 GB of RAM.
> >
> > Can someone help me understand this? I don't want to give Solr RAM that
> it
> > won't use.
> >
> > PS: This is simply search tests, there is no update to the index at all.
> >
> > Thanks in advanced.
> >
> > Steve
>
Re: Solr memory usage
Posted by Erick Erickson <er...@gmail.com>.
You're doing nothing wrong, that particular bit of advice has
always needed a bit of explanation.
Solr (well, actually Lucene) uses MMapDirectory for much of
the index structure which uses the OS memory rather than
the JVM heap. See Uwe's excellent:
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
Plus, the size on disk includes the stored data, which is in the *.fdt
files in data/index. Very little of the stored data is kept in the JVM
so that's another reason your Java heap may be smaller than
your raw index size on disk.
The advice about fitting your entire index into memory really has
the following caveats (at least).
1> "memory" includes the OS memory available to the process
2> The size of the index on disk is misleading, the *.fdt files
should be subtracted in order to get a truer picture.
3> Both Solr and Lucene create structures in the Java JVM
that are _not_ reflected in the size on disk.
<1> and <2> mean the JVM memory necessary is smaller
than the size on disk.
<3> means the JVM memory will be larger than.
So you're doing the right thing, testing and seeing what you
_really_ need. I'd pretty much take your test, add some
padding and consider it good. You're _not_ doing the
really bad thing of using the same query over and over
again and hoping <G>.
Best,
Erick
On Tue, Dec 8, 2015 at 11:54 AM, Steven White <sw...@gmail.com> wrote:
> Hi folks,
>
> My index size on disk (optimized) is 20 GB (single core, single index). I
> have a system with 64 GB of RAM. I start Solr with 24 GB of RAM.
>
> I have run load tests (up to 100 concurrent users) for hours where each
> user issuing unique searches (the same search is never executed again for
> at least 30 minute since it was last executed). In all tests I run, Solr's
> JVM memory never goes over 10 GB (monitoring http://localhost:8983/).
>
> I read over and over, for optimal performance, Solr should be given enough
> RAM to hold the index in memory. Well, I have done that and some but yet I
> don't see Solr using up that whole RAM. What am I doing wrong? Is my test
> at fault? I doubled the test load (number of users) and didn't see much of
> a difference with RAM usage but yet my search performance went down (takes
> about 40% longer now). I run my tests again but this time with only 12 GB
> of RAM given to Solr. Test result didn't differ much from the 24 GB run
> and Solr never used more than 10 GB of RAM.
>
> Can someone help me understand this? I don't want to give Solr RAM that it
> won't use.
>
> PS: This is simply search tests, there is no update to the index at all.
>
> Thanks in advanced.
>
> Steve