You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Zheng Lin Edwin Yeo <ed...@gmail.com> on 2016/08/17 03:02:13 UTC

Increasing filterCache size and Java Heap size

Hi,

Would like to check, do I need to increase my Java Heap size for Solr, if I
plan to increase my filterCache size in solrconfig.xml?

I'm using Solr 6.1.0

Regards,
Edwin

Re: Increasing filterCache size and Java Heap size

Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Hi Toke,

Thanks for the explanation.
I will prefer the memory-based limit too. At first I got confused with that
too, thinking that the setting of 2000 means 2GB.

Regards,
Edwin


On 17 August 2016 at 17:40, Toke Eskildsen <te...@statsbiblioteket.dk> wrote:

> On Wed, 2016-08-17 at 11:02 +0800, Zheng Lin Edwin Yeo wrote:
> > Would like to check, do I need to increase my Java Heap size for
> > Solr, if I plan to increase my filterCache size in solrconfig.xml?
> >
> > I'm using Solr 6.1.0
>
> It _seems_ that you can specify a limit in megabytes when using
> LRUCache in Solr 5.2+: https://issues.apache.org/jira/browse/SOLR-7372
>
> The documentation only mentions it for queryResultCache, but I do not
> know if that is intentional (i.e. it does not work for filterCache) or
> a shortcoming of the documentation:
> https://cwiki.apache.org/confluence/display/solr/Query+Settings+in+Solr
> Config
>
> If it does work for filterCache too (using LRUCache, I guess), then
> that would be a much better way of limiting cache size than the highly
> insufficient count-based limiter.
>
>
> I say "highly insufficient" because filter cache entries are not of
> equal size. With small sets they are stored as sparse, using a
> relatively small amount of memory. For larger sets they are stored as
> bitmaps, taking up ~1K + maxdoc/8 bytes as Erick describes.
>
> So a fixed upper limit measured in counts needs to be adjusted to worst
> case, meaning maxdoc/8, to ensure stability. In reality most of the
> filter cache entries are small, meaning that there is plenty of heap
> not being used. This leads people to over-allocate the max size for the
> filterCache (very understandable) , resulting in setups that are only
> stable as long as there are not too many large filter sets stores.
> Leaving it to chance really.
>
> I would prefer the count-based limit to be deprecated for the
> filterCache, or at least warned against, in favour of memory-based.
>
> - Toke Eskildsen, State and University Library, Denmark
>
>

Re: Increasing filterCache size and Java Heap size

Posted by Toke Eskildsen <te...@statsbiblioteket.dk>.
On Wed, 2016-08-17 at 11:02 +0800, Zheng Lin Edwin Yeo wrote:
> Would like to check, do I need to increase my Java Heap size for
> Solr, if I plan to increase my filterCache size in solrconfig.xml?
> 
> I'm using Solr 6.1.0

It _seems_ that you can specify a limit in megabytes when using
LRUCache in Solr 5.2+:�https://issues.apache.org/jira/browse/SOLR-7372

The documentation only mentions it for queryResultCache, but I do not
know if that is intentional (i.e. it does not work for filterCache) or
a shortcoming of the documentation:
https://cwiki.apache.org/confluence/display/solr/Query+Settings+in+Solr
Config

If it does work for filterCache too (using LRUCache, I guess), then
that would be a much better way of limiting cache size than the highly
insufficient count-based limiter.


I say "highly insufficient" because filter cache entries are not of
equal size. With small sets they are stored as sparse, using a
relatively small amount of memory. For larger sets they are stored as
bitmaps, taking up ~1K + maxdoc/8 bytes as Erick describes.

So a fixed upper limit measured in counts needs to be adjusted to worst
case, meaning maxdoc/8, to ensure stability. In reality most of the
filter cache entries are small, meaning that there is plenty of heap
not being used. This leads people to over-allocate the max size for the
filterCache (very understandable)�, resulting in setups that are only
stable as long as there are not too many large filter sets stores.
Leaving it to chance really.

I would prefer the count-based limit to be deprecated for the
filterCache, or at least warned against, in favour of memory-based.

- Toke Eskildsen, State and University Library, Denmark

Re: Increasing filterCache size and Java Heap size

Posted by Zheng Lin Edwin Yeo <ed...@gmail.com>.
Hi Erick,

Thanks for your reply.

But do we have to set the Java Heap size based on all the collections
available (if I were to increase the filterCache size for all my
collections)?

I come across this from StackOverFlow,
http://stackoverflow.com/questions/20999904/solr-filter-cache-fastlrucache-takes-too-much-memory-and-results-in-out-of-mem
it says that if we want to have a filterCache size of 2000, we will need
12GB of memory.

Let's say if we have 3 of the collections, which all the filterCache size
are set to 2000, do we need 36GB of Java Heap space memory? Or just 12GB
will be sufficient?

Regards,
Edwin


On 17 August 2016 at 14:09, Erick Erickson <er...@gmail.com> wrote:

> Yes. Each entry is roughly 1K + maxdoc/8 bytes. The maxdoc/8 is the
> bitmap that holds the result set and the 1K is just overhead for the
> text of the query itself and cache overhead. Usually it's safe to
> ignore since the maxdoc/8 usually dominates by a wide margin.
>
> Best,
> Erick
>
> On Tue, Aug 16, 2016 at 8:02 PM, Zheng Lin Edwin Yeo
> <ed...@gmail.com> wrote:
> > Hi,
> >
> > Would like to check, do I need to increase my Java Heap size for Solr,
> if I
> > plan to increase my filterCache size in solrconfig.xml?
> >
> > I'm using Solr 6.1.0
> >
> > Regards,
> > Edwin
>

Re: Increasing filterCache size and Java Heap size

Posted by Erick Erickson <er...@gmail.com>.
Yes. Each entry is roughly 1K + maxdoc/8 bytes. The maxdoc/8 is the
bitmap that holds the result set and the 1K is just overhead for the
text of the query itself and cache overhead. Usually it's safe to
ignore since the maxdoc/8 usually dominates by a wide margin.

Best,
Erick

On Tue, Aug 16, 2016 at 8:02 PM, Zheng Lin Edwin Yeo
<ed...@gmail.com> wrote:
> Hi,
>
> Would like to check, do I need to increase my Java Heap size for Solr, if I
> plan to increase my filterCache size in solrconfig.xml?
>
> I'm using Solr 6.1.0
>
> Regards,
> Edwin