Solr Memory Usage Question

[Jamie Johnson <je...@gmail.com>]

I am not currently running into memory issues, but I was wondering if
anyone could explain to me Solrs memory usage?  What does Solr
actually store in memory?  What are some of the largest memory
consumers (i.e. faceting, sorting, etc).  Is the best way to start
addressing questions like this to just fire up JConsole?

Re: Solr Memory Usage Question

[Erick Erickson <er...@gmail.com>]

Faceting and sorting are the two biggest places people get into trouble.
You've been asking questions about Solr Cloud, so I assume you're
working on a trunk release. Note that most everything people know
about memory consumption painfully gained over the years is...wrong
on trunk.

Or at least may be wrong. Just for yucks I took an 11M document
Wiki dump and indexed it as just a few (10 or so) fields. Then
I faceted (as I remember) on the text field (a seriously bad thing to
do). Memory consumption dropped 70+%. Restated: I was using
less than 1/3 of the memory. Additionally, as I understand
it the memory allocation is very significantly different for strings and
things. There should be many, many fewer objects created when
searching in Solr and the garbage collection should be lots less
intensive.

Now, I wasn't involved in writing the code, so my understanding may be
way off base.

I've also seen problems with garbage collection kicking in and not
doing much, but I expect trunk alters that problem for the above reasons
as well.

But yes, jConsole or similar is about the only way to measure this. If
you haven't read Mark Miller's excellent article on GC, you might
want to:
http://www.lucidimagination.com/blog/2011/03/27/garbage-collection-bootcamp-1-0/

The caches are where the memory consumption really shows up, BTW, see:
http://wiki.apache.org/solr/SolrCaching
The queryResultCache and documentCache are generally not where problems
show up, but watching the stats on the other caches to see if you're getting
decent hit ratios can be a pointer too.

And if you're stressing things with trunk I'll be very interested in
your results!

Erick

On Mon, Mar 19, 2012 at 3:15 PM, Jamie Johnson <je...@gmail.com> wrote:
> I am not currently running into memory issues, but I was wondering if
> anyone could explain to me Solrs memory usage?  What does Solr
> actually store in memory?  What are some of the largest memory
> consumers (i.e. faceting, sorting, etc).  Is the best way to start
> addressing questions like this to just fire up JConsole?