You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Matt Mitchell <go...@gmail.com> on 2010/01/16 16:01:07 UTC
java heap space error when faceting
I have an index with more than 6 million docs. All is well, until I turn on
faceting and specify a facet.field. There is only about unique 20 values for
this particular facet throughout the entire index. I was able to make things
a little better by using facet.method=enum. That seems to work, until I add
another facet.field to the request, which is another facet that doesn't have
that many unique values. I utlimately end up running out of heap space
memory. I should also mention that in every case, the "rows" param is set to
0.
I've thrown as much memory as I can at the JVM (+3G for start-up and max),
tweaked filter cache settings etc.. I can't seem to get this error to go
away. Anyone have any tips to throw my way?
-- using a recent nighlty build of solr 1.5 dev and Jetty as my servlet
container.
Thanks!
Matt
Re: java heap space error when faceting
Posted by Matt Mitchell <go...@gmail.com>.
I'm embarrassed (but hugely relieved) to say that, the script I had for
starting Jetty had a bug in the way it set java options! So, my heap
start/max was always set at the default. I did end up using jconsole and
learned quite a bit from that too.
Thanks for your help Yonik :)
Matt
On Sat, Jan 16, 2010 at 11:13 AM, Yonik Seeley
<yo...@lucidimagination.com>wrote:
> On Sat, Jan 16, 2010 at 11:04 AM, Matt Mitchell <go...@gmail.com>
> wrote:
> > These are single valued fields. Strings and integers. Is there more
> specific
> > info I could post to help diagnose what might be happening?
>
> Faceting on either should currently take ~24MB (6M docs @ 4 bytes per
> doc + size_of_unique_values)
> With that small number of values, facet.enum may be faster in general
> (and take up less room: 6M/8*20 or 15MB).
> But you certainly shouldn't be running out of space with the heap
> sizes you mentioned.
>
> Perhaps look at the stats.jsp page in the admin and see what's listed
> in the fieldCache?
> And verify that your heap is really as big as you think it is.
> You can also use something like jconsole that ships with the JDK to
> manually do a GC and check out how much of the heap is in use before
> you try to facet.
>
> -Yonik
> http://www.lucidimagination.com
>
Re: java heap space error when faceting
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Sat, Jan 16, 2010 at 11:04 AM, Matt Mitchell <go...@gmail.com> wrote:
> These are single valued fields. Strings and integers. Is there more specific
> info I could post to help diagnose what might be happening?
Faceting on either should currently take ~24MB (6M docs @ 4 bytes per
doc + size_of_unique_values)
With that small number of values, facet.enum may be faster in general
(and take up less room: 6M/8*20 or 15MB).
But you certainly shouldn't be running out of space with the heap
sizes you mentioned.
Perhaps look at the stats.jsp page in the admin and see what's listed
in the fieldCache?
And verify that your heap is really as big as you think it is.
You can also use something like jconsole that ships with the JDK to
manually do a GC and check out how much of the heap is in use before
you try to facet.
-Yonik
http://www.lucidimagination.com
Re: java heap space error when faceting
Posted by Matt Mitchell <go...@gmail.com>.
These are single valued fields. Strings and integers. Is there more specific
info I could post to help diagnose what might be happening?
Thanks!
Matt
On Sat, Jan 16, 2010 at 10:42 AM, Yonik Seeley
<yo...@lucidimagination.com>wrote:
> On Sat, Jan 16, 2010 at 10:01 AM, Matt Mitchell <go...@gmail.com>
> wrote:
> > I have an index with more than 6 million docs. All is well, until I turn
> on
> > faceting and specify a facet.field. There is only about unique 20 values
> for
> > this particular facet throughout the entire index.
>
> Hmmm, that doesn't sound right... unless you're already near max
> memory usage due to other things.
> Is this a single-valued or multi-valued field? If multi-valued, how
> many values does each document have on average?
>
> -Yonik
> http://www.lucidimagination.com
>
Re: java heap space error when faceting
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Sat, Jan 16, 2010 at 10:01 AM, Matt Mitchell <go...@gmail.com> wrote:
> I have an index with more than 6 million docs. All is well, until I turn on
> faceting and specify a facet.field. There is only about unique 20 values for
> this particular facet throughout the entire index.
Hmmm, that doesn't sound right... unless you're already near max
memory usage due to other things.
Is this a single-valued or multi-valued field? If multi-valued, how
many values does each document have on average?
-Yonik
http://www.lucidimagination.com