You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Andrew Lundgren <lu...@familysearch.org> on 2011/11/19 00:49:55 UTC
Solr filterCache size settings...
I am new to solr in general and trying to get a handle on the memory requirements for caching. Specifically I am looking at the filterCache right now. The documentation on size setting seems to indicate that it is the number of values to be cached. Did I read that correctly, or is it really the amount of memory that will be set aside for the cache?
How do you determine how much cache each fq will consume?
Thank you!
--
Andrew Lundgren
lundgren@familysearch.org
NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
RE: Solr filterCache size settings...
Posted by Chris Hostetter <ho...@fucit.org>.
: One clarification, is the maxdocs the max docs in the set, or the matched docs from the set?
:
: If there are 1000 docs and 19 of them match, is the maxdocs 1000, or 19?
Erick ment the maxDocs of the index -- but that's really just a rule of
thumb approximation that applies when many docs match throughout the
entire index.
The objects stored in the cache are "DocSet" instances, and there are
optimizations in the code to choose between differnet DocSet
implementations based on how many documents actually match. Even when
the bitset based implementation is choosen, there are additional
optimizations to use a compact bitset when there are large "gaps" in
between set bits.
-Hoss
RE: Solr filterCache size settings...
Posted by Andrew Lundgren <lu...@familysearch.org>.
Thank you for your reply.
One clarification, is the maxdocs the max docs in the set, or the matched docs from the set?
If there are 1000 docs and 19 of them match, is the maxdocs 1000, or 19?
--
Andrew
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Sunday, November 20, 2011 8:10 PM
To: solr-user@lucene.apache.org
Subject: Re: Solr filterCache size settings...
Each fq will create a bitmap that is bounded by (maxdocs / 8) bytes.
You can think of the entries in the fiterCache as a map where the key is the
filter query you specify and the value is the aforementioned bitmap. The
number of entries specified in the config file is the number of entries
in that map. So the cache can take up roughly (assuming the size if 512)
512 * maxDocs / 8 bytes.
Best
Erick
On Fri, Nov 18, 2011 at 6:49 PM, Andrew Lundgren
<lu...@familysearch.org> wrote:
> I am new to solr in general and trying to get a handle on the memory requirements for caching. Specifically I am looking at the filterCache right now. The documentation on size setting seems to indicate that it is the number of values to be cached. Did I read that correctly, or is it really the amount of memory that will be set aside for the cache?
>
> How do you determine how much cache each fq will consume?
>
> Thank you!
>
> --
> Andrew Lundgren
> lundgren@familysearch.org
>
>
> NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
>
>
>
Re: Solr filterCache size settings...
Posted by Markus Jelsma <ma...@openindex.io>.
ignore, i misread :)
> > Each fq will create a bitmap that is bounded by (maxdocs / 8) bytes.
> >
> > You can think of the entries in the fiterCache as a map where the key is
> > the filter query you specify and the value is the aforementioned bitmap.
> > The number of entries specified in the config file is the number of
> > entries in that map. So the cache can take up roughly (assuming the size
> > if 512) 512 * maxDocs / 8 bytes.
>
> Don't you mean 512 * (maxDocs / 8 bytes) since you first say that each
> filter is maxDoc / 8?
>
> > Best
> > Erick
> >
> > On Fri, Nov 18, 2011 at 6:49 PM, Andrew Lundgren
> >
> > <lu...@familysearch.org> wrote:
> > > I am new to solr in general and trying to get a handle on the memory
> > > requirements for caching. Specifically I am looking at the
> > > filterCache right now. The documentation on size setting seems to
> > > indicate that it is the number of values to be cached. Did I read
> > > that correctly, or is it really the amount of memory that will be set
> > > aside for the cache?
> > >
> > > How do you determine how much cache each fq will consume?
> > >
> > > Thank you!
> > >
> > > --
> > > Andrew Lundgren
> > > lundgren@familysearch.org
> > >
> > > NOTICE: This email message is for the sole use of the intended
> > >
> > > recipient(s) and may contain confidential and privileged information.
> > > Any unauthorized review, use, disclosure or distribution is prohibited.
> > > If you are not the intended recipient, please contact the sender by
> > > reply email and destroy all copies of the original message.
Re: Solr filterCache size settings...
Posted by Markus Jelsma <ma...@openindex.io>.
> Each fq will create a bitmap that is bounded by (maxdocs / 8) bytes.
>
> You can think of the entries in the fiterCache as a map where the key is
> the filter query you specify and the value is the aforementioned bitmap.
> The number of entries specified in the config file is the number of
> entries in that map. So the cache can take up roughly (assuming the size
> if 512) 512 * maxDocs / 8 bytes.
Don't you mean 512 * (maxDocs / 8 bytes) since you first say that each filter
is maxDoc / 8?
>
> Best
> Erick
>
> On Fri, Nov 18, 2011 at 6:49 PM, Andrew Lundgren
>
> <lu...@familysearch.org> wrote:
> > I am new to solr in general and trying to get a handle on the memory
> > requirements for caching. Specifically I am looking at the filterCache
> > right now. The documentation on size setting seems to indicate that it
> > is the number of values to be cached. Did I read that correctly, or is
> > it really the amount of memory that will be set aside for the cache?
> >
> > How do you determine how much cache each fq will consume?
> >
> > Thank you!
> >
> > --
> > Andrew Lundgren
> > lundgren@familysearch.org
> >
> >
> > NOTICE: This email message is for the sole use of the intended
> > recipient(s) and may contain confidential and privileged information.
> > Any unauthorized review, use, disclosure or distribution is prohibited.
> > If you are not the intended recipient, please contact the sender by
> > reply email and destroy all copies of the original message.
Re: Solr filterCache size settings...
Posted by Erick Erickson <er...@gmail.com>.
Each fq will create a bitmap that is bounded by (maxdocs / 8) bytes.
You can think of the entries in the fiterCache as a map where the key is the
filter query you specify and the value is the aforementioned bitmap. The
number of entries specified in the config file is the number of entries
in that map. So the cache can take up roughly (assuming the size if 512)
512 * maxDocs / 8 bytes.
Best
Erick
On Fri, Nov 18, 2011 at 6:49 PM, Andrew Lundgren
<lu...@familysearch.org> wrote:
> I am new to solr in general and trying to get a handle on the memory requirements for caching. Specifically I am looking at the filterCache right now. The documentation on size setting seems to indicate that it is the number of values to be cached. Did I read that correctly, or is it really the amount of memory that will be set aside for the cache?
>
> How do you determine how much cache each fq will consume?
>
> Thank you!
>
> --
> Andrew Lundgren
> lundgren@familysearch.org
>
>
> NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
>
>
>