Solr not using allocated memory?

[Tristan Roddis <tr...@cogapp.com>]

We have a Solr server running on 64-bit Windows 2008, and under the 
default Tomcat settings it seems to be using a small fraction of the 
available memory (fluctuating between 100Mb and 300Mb: see trace from 
VisualVM at http://imgur.com/4Om2El&tqLKW <http://imgur.com/4Om2El&tqLKW> )

I was advised that, to increase performance, we should increase the 
amount of memory allocated to Tomcat so that Solr could use this to 
cache query results and so reduce CPU usage and disk I/O. So, I set both 
the min and max RAM allocation for Tomcat to 10Gb, which is well within 
the amount allocated to the server (16Gb). I tried this on another 
machine, set a simulated load of lots of concurrent users running on our 
application, and the VisualVM results can be seen at 
http://imgur.com/4Om2E&tqLKWl <http://imgur.com/4Om2E&tqLKWl>

These results were very different: in the first half of the graph, when 
the system was under load, the amount of allocated heap size was fixed 
at 10Gb, which I would expect, but the amount of used heap climbed 
gradually to around 3Gb, but then shot right down again to something 
close to zero every minute or so, presumably due to the GC kicking in, 
giving the sawtooth pattern you can see in the screenshot above.

My question is: is this normal and expected? I would have thought that 
the amount of used heap would climb to somewhere much nearer the maximum 
available of 10Gb, rather than peaking at a mere 3Gb. And also that any 
garbage collection would only deallocate part of the memory rather than 
knocking it right back to zero.

Essentially, why isn't Solr using more of the memory assigned, and using 
it constantly rather than continually clearing the cache?

Thanks,

-Tristan.

Re: Solr not using allocated memory?

[Walter Underwood <wu...@wunderwood.org>]

When I have seen that sawtooth pattern before, it was because the cache hit rate was low. Items ejected from the cache become garbage. If that happens at a fairly constant rate, you get that slope of increasing memory usage.

wunder

On Oct 7, 2011, at 8:48 AM, Tristan Roddis wrote:

> We have a Solr server running on 64-bit Windows 2008, and under the default Tomcat settings it seems to be using a small fraction of the available memory (fluctuating between 100Mb and 300Mb: see trace from VisualVM at http://imgur.com/4Om2El&tqLKW <http://imgur.com/4Om2El&tqLKW> )
> 
> I was advised that, to increase performance, we should increase the amount of memory allocated to Tomcat so that Solr could use this to cache query results and so reduce CPU usage and disk I/O. So, I set both the min and max RAM allocation for Tomcat to 10Gb, which is well within the amount allocated to the server (16Gb). I tried this on another machine, set a simulated load of lots of concurrent users running on our application, and the VisualVM results can be seen at http://imgur.com/4Om2E&tqLKWl <http://imgur.com/4Om2E&tqLKWl>
> 
> These results were very different: in the first half of the graph, when the system was under load, the amount of allocated heap size was fixed at 10Gb, which I would expect, but the amount of used heap climbed gradually to around 3Gb, but then shot right down again to something close to zero every minute or so, presumably due to the GC kicking in, giving the sawtooth pattern you can see in the screenshot above.
> 
> My question is: is this normal and expected? I would have thought that the amount of used heap would climb to somewhere much nearer the maximum available of 10Gb, rather than peaking at a mere 3Gb. And also that any garbage collection would only deallocate part of the memory rather than knocking it right back to zero.
> 
> Essentially, why isn't Solr using more of the memory assigned, and using it constantly rather than continually clearing the cache?
> 
> Thanks,
> 
> -Tristan.

Re: Solr not using allocated memory?

[Shawn Heisey <so...@elyograg.org>]

On 10/7/2011 9:48 AM, Tristan Roddis wrote:
> I was advised that, to increase performance, we should increase the 
> amount of memory allocated to Tomcat so that Solr could use this to 
> cache query results and so reduce CPU usage and disk I/O. So, I set 
> both the min and max RAM allocation for Tomcat to 10Gb, which is well 
> within the amount allocated to the server (16Gb). I tried this on 
> another machine, set a simulated load of lots of concurrent users 
> running on our application, and the VisualVM results can be seen at 
> http://imgur.com/4Om2E&tqLKWl <http://imgur.com/4Om2E&tqLKWl>
>
> These results were very different: in the first half of the graph, 
> when the system was under load, the amount of allocated heap size was 
> fixed at 10Gb, which I would expect, but the amount of used heap 
> climbed gradually to around 3Gb, but then shot right down again to 
> something close to zero every minute or so, presumably due to the GC 
> kicking in, giving the sawtooth pattern you can see in the screenshot 
> above.
>
> My question is: is this normal and expected? I would have thought that 
> the amount of used heap would climb to somewhere much nearer the 
> maximum available of 10Gb, rather than peaking at a mere 3Gb. And also 
> that any garbage collection would only deallocate part of the memory 
> rather than knocking it right back to zero.
>
> Essentially, why isn't Solr using more of the memory assigned, and 
> using it constantly rather than continually clearing the cache?

Tristan,

Just assigning more memory to the JVM willl not make Solr utilize it for 
caches.  You must also increase the cache sizes in solrconfig.xml.  That 
may not be the best use for your RAM, though.  Here are my cache 
settings for the cores that handle queries.  My shards are nearing 20GB 
and have more than 10 million documents in them.  I use low 
autowarmCount values so that commit times are short.

<filterCache
     class="solr.FastLRUCache"
     size="64"
     initialSize="64"
     autowarmCount="4"
     cleanupThread="true"
   />

<queryResultCache
     class="solr.FastLRUCache"
     size="512"
     initialSize="512"
     autowarmCount="8"
     cleanupThread="true"
   />

<documentCache
     class="solr.FastLRUCache"
     size="16384"
     initialSize="4096"
     cleanupThread="true"
   />

Although Solr's built in caching does offer performance improvements, 
beyond a certain size it won't provide any actual benefit.  The best 
thing to do (in most cases) is to allocate enough RAM to the JVM to keep 
it from ever reaching an out of memory condition, then leave the rest of 
it available for OS disk cache.  It sounds like 2-3GB might be a good 
max heap size for you.  If the default tomcat settings did not lead to 
any out of memory errors, you might even be able to use 1GB.

Ideally the OS disk cache will be big enough to cache your entire index, 
which would mean that during normal operation the system never has to 
actually access the disk.  That isn't always possible, but you should 
get as close to that ideal as you can.

Thanks,
Shawn

Re: Solr not using allocated memory?

[Lan <du...@gmail.com>]

What are you trying to speed up and what are the timings you are getting?

Giving solr a lot of memory does not always speed things up. You must leave
some room to allow the O/S to cache the index files.

--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-not-using-allocated-memory-tp3403176p3403524.html
Sent from the Solr - User mailing list archive at Nabble.com.