You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Stephen Duncan Jr <st...@gmail.com> on 2009/07/31 23:06:26 UTC
Questions on FieldValueCache
I have a couple more questions on the FieldValueCache. I see that the
number of items in the cache is basically the number of multi-valued fields
facets have been requested for. What does each entry in the cache actually
contain? How does it's size grow as the number of total documents
increases?
Ok, so that was the curiosity question. More critical:
When we first ask for facets for multi-valued fields, it can take up to 25
seconds to get the response, although after that it's very fast (1.5 seconds
or less even for a lot of different facets) since the FieldValueCache has
been populated. Obviously, we'd like to get the FieldValueCache populated,
hopefully without an end-user being the one to wait 25 seconds. I think
this is what the autowarming settings are about, but I'm not sure how to
warm the cache with the specific fields we expect facets to be requested
for. Any pointers?
--
Stephen Duncan Jr
www.stephenduncanjr.com
Re: Questions on FieldValueCache
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Stephen,
Yes, *:* will work, or at least it did last time I tried it a few months ago. This should quickly warm up your OS disk cache.
Yes, if searcher warming takes too long, you may need to commit less frequently to avoid searcher overlap.
Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
----- Original Message ----
> From: Stephen Duncan Jr <st...@gmail.com>
> To: solr-user@lucene.apache.org; yonik@lucidimagination.com
> Sent: Friday, July 31, 2009 5:54:39 PM
> Subject: Re: Questions on FieldValueCache
>
> On Fri, Jul 31, 2009 at 5:23 PM, Yonik Seeley wrote:
>
> > > Ok, so that was the curiosity question. More critical:
> > >
> > > When we first ask for facets for multi-valued fields, it can take up to
> > 25
> > > seconds to get the response, although after that it's very fast (1.5
> > seconds
> > > or less even for a lot of different facets) since the FieldValueCache has
> > > been populated. Obviously, we'd like to get the FieldValueCache
> > populated,
> > > hopefully without an end-user being the one to wait 25 seconds. I think
> > > this is what the autowarming settings are about, but I'm not sure how to
> > > warm the cache with the specific fields we expect facets to be requested
> > > for. Any pointers?
> >
> > Use static warming queries. autowarming doesn't hit the fieldValueCache.
> >
> > -Yonik
> > http://www.lucidimagination.com
> >
>
> So in solrconfig.xml update the queries in newSearcher and/or(?)
> firstSearcher to include all the settings for a facet query with the desired
> facets? Does the cache get populated if the query string is *:* (I don't
> want to risk a setup that doesn't hit any results and fails to populate the
> cache..). If the warming query might now take 25 seconds (maybe more for
> more documents), I guess we might need to decrease the autocommit rate to
> cut down on chance of overlapping warmers?
>
> --
> Stephen Duncan Jr
> www.stephenduncanjr.com
Re: Questions on FieldValueCache
Posted by Stephen Duncan Jr <st...@gmail.com>.
On Fri, Jul 31, 2009 at 5:23 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
> > Ok, so that was the curiosity question. More critical:
> >
> > When we first ask for facets for multi-valued fields, it can take up to
> 25
> > seconds to get the response, although after that it's very fast (1.5
> seconds
> > or less even for a lot of different facets) since the FieldValueCache has
> > been populated. Obviously, we'd like to get the FieldValueCache
> populated,
> > hopefully without an end-user being the one to wait 25 seconds. I think
> > this is what the autowarming settings are about, but I'm not sure how to
> > warm the cache with the specific fields we expect facets to be requested
> > for. Any pointers?
>
> Use static warming queries. autowarming doesn't hit the fieldValueCache.
>
> -Yonik
> http://www.lucidimagination.com
>
So in solrconfig.xml update the queries in newSearcher and/or(?)
firstSearcher to include all the settings for a facet query with the desired
facets? Does the cache get populated if the query string is *:* (I don't
want to risk a setup that doesn't hit any results and fails to populate the
cache..). If the warming query might now take 25 seconds (maybe more for
more documents), I guess we might need to decrease the autocommit rate to
cut down on chance of overlapping warmers?
--
Stephen Duncan Jr
www.stephenduncanjr.com
Re: Questions on FieldValueCache
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Mon, Aug 3, 2009 at 5:45 PM, Stephen Duncan
Jr<st...@gmail.com> wrote:
> What will cause me to exceed the limit and fail during faceting using the fc
> method is if the documents within a block of 65536 combine to take up too
> much space. And this (generally speaking) going to be a function on the
> average number of unique terms in the documents?
Yep... pretty much. The comment at the head of UnInvertedField tries
to lay out the data structure.
Also subtract out "big terms" that match > 5% of the index... they
aren't uninverted.
And the block of 65536 docs wraps around after 256 blocks (since there
are only 256 arrays).
If you needed just a little more headroom, it wouldn't be to hard for
you to modify from 256 arrays to 512 (which would double the number of
terms allowed per block). Or if you had much less than 16M docs in
the index, you could also lower the number of docs per block (all
custom code of course... it can't be done with a parameter today).
-Yonik
http://www.lucidimagination.com
Re: Questions on FieldValueCache
Posted by Stephen Duncan Jr <st...@gmail.com>.
On Mon, Aug 3, 2009 at 4:56 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
> On Mon, Aug 3, 2009 at 4:18 PM, Stephen Duncan
> Jr<st...@gmail.com> wrote:
> > On Mon, Aug 3, 2009 at 2:43 PM, Yonik Seeley <yonik@lucidimagination.com
> >wrote:
> > Hmm, that's a hard thing to sell to the user and my boss, as it makes the
> > query time go from nearly always being sub-second (frequently less than
> 60
> > ms), to ranging up to nearly 4 seconds for a new query not already in the
> > cache. (My test was with 100 facets being requested, which may be
> > reasonable, as one reason to facet on a full-text field to provide a
> dynamic
> > world-cloud).
>
> Could you possibly profile it to find out what the hotspot is?
> We don't really have a good algorithm for faceting text fields, but it
> would be nice to see what the current bottleneck is.
I'll put in my TODO list to try that out soon. I'll let you know the
results if I manage.
>
> > I guess I should update my code to use the enum method on all the fields
> > that are likely to risk crossing this line. Should I be looking at the
> > termInstances property on the fields that are displayed in the
> > FieldValueCache on the stats page, and figuring those on the order of 10
> > million are likely to grow past the limit?
>
> For an index over 16M docs, it's perhaps closer to
> 16M/avg_bytes_per_term*256.
>
> The storage space for terms that aren't "big terms" (which come from
> the fieldCache) is 256 byte arrays, each which can be up to 16MB in
> size. Every 65536 block of documents shares one of those byte arrays
> (or more if you have more than 16M documents). So the average
> document can't take up more than 256 bytes in the array. That doesn't
> mean 256 term instances though... that's the max. The list is delta
> encoded vints, so if there are many terms, each vint could be bigger.
>
> More details in UnInvertedField after the comment:
> //
> // transform intermediate form into the final form, building a
> single byte[]
> // at a time, and releasing the intermediate byte[]s as we go to avoid
> // increasing the memory footprint.
> //
>
> -Yonik
> http://www.lucidimagination.com
>
Ok, a lot of that is going over-my-head for the moment. I'll try to digest
this info a little further, but for now let's see if my minimal
understanding is correct:
What will cause me to exceed the limit and fail during faceting using the fc
method is if the documents within a block of 65536 combine to take up too
much space. And this (generally speaking) going to be a function on the
average number of unique terms in the documents?
--
Stephen Duncan Jr
www.stephenduncanjr.com
Re: Questions on FieldValueCache
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Mon, Aug 3, 2009 at 4:18 PM, Stephen Duncan
Jr<st...@gmail.com> wrote:
> On Mon, Aug 3, 2009 at 2:43 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
> Hmm, that's a hard thing to sell to the user and my boss, as it makes the
> query time go from nearly always being sub-second (frequently less than 60
> ms), to ranging up to nearly 4 seconds for a new query not already in the
> cache. (My test was with 100 facets being requested, which may be
> reasonable, as one reason to facet on a full-text field to provide a dynamic
> world-cloud).
Could you possibly profile it to find out what the hotspot is?
We don't really have a good algorithm for faceting text fields, but it
would be nice to see what the current bottleneck is.
> How can I mitigate the time it takes with the enum method? Do I need to ask
> for more facet values in my facet-warming query (I set facet.limit to 1 as
> it didn't seem to matter to the FieldValueCache)?
Yes, it matters for the enum method because of the smart
short-circuiting that takes place.
Use a base query that matches fewer documents than the size of the
sets you want cached.
Set the limit higher to avoid short circuiting
> And/Or do I need to up the
> autowarmCount on the FilterCache?
Not if you have a static warming query that includes the facets you
are interested in.
> If speed is the primary concern vs
> memory, should I bother with the minDf setting?
minDf is pretty much just for memory savings. But if you turn it down
or eliminate it, make sure your filterCache is big enough to hold a
filter for each possible term.
> I guess I should update my code to use the enum method on all the fields
> that are likely to risk crossing this line. Should I be looking at the
> termInstances property on the fields that are displayed in the
> FieldValueCache on the stats page, and figuring those on the order of 10
> million are likely to grow past the limit?
For an index over 16M docs, it's perhaps closer to 16M/avg_bytes_per_term*256.
The storage space for terms that aren't "big terms" (which come from
the fieldCache) is 256 byte arrays, each which can be up to 16MB in
size. Every 65536 block of documents shares one of those byte arrays
(or more if you have more than 16M documents). So the average
document can't take up more than 256 bytes in the array. That doesn't
mean 256 term instances though... that's the max. The list is delta
encoded vints, so if there are many terms, each vint could be bigger.
More details in UnInvertedField after the comment:
//
// transform intermediate form into the final form, building a
single byte[]
// at a time, and releasing the intermediate byte[]s as we go to avoid
// increasing the memory footprint.
//
-Yonik
http://www.lucidimagination.com
Re: Questions on FieldValueCache
Posted by Stephen Duncan Jr <st...@gmail.com>.
On Mon, Aug 3, 2009 at 2:43 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
>
> >
> > I'm getting the following warning in my logs: 2009-08-03 13:41:40,114
> > [http-127.0.0.1-8080-1] WARN org.apache.solr.core.SolrCore - Approaching
> > too many values for UnInvertedField faceting on field 'originaltext' :
> > bucket size=15802492
> >
> > What's the impact of that? If the number of values (number of unique
> terms
> > for that field, or some other "values"?) exceeds that limit, will
> faceting
> > for that field go back to a different technique and be slower, or...?
>
> It will throw an exception.
>
> This method of faceting wasn't really designed for big full-text fields.
> The enum method should work better for this... try something like the
> following:
>
> f.originaltext.facet.method=enum
> facet.enum.cache.minDf=10000
>
> -Yonik
> http://www.lucidimagination.com
>
Hmm, that's a hard thing to sell to the user and my boss, as it makes the
query time go from nearly always being sub-second (frequently less than 60
ms), to ranging up to nearly 4 seconds for a new query not already in the
cache. (My test was with 100 facets being requested, which may be
reasonable, as one reason to facet on a full-text field to provide a dynamic
world-cloud).
How can I mitigate the time it takes with the enum method? Do I need to ask
for more facet values in my facet-warming query (I set facet.limit to 1 as
it didn't seem to matter to the FieldValueCache)? And/Or do I need to up the
autowarmCount on the FilterCache? If speed is the primary concern vs
memory, should I bother with the minDf setting?
I guess I should update my code to use the enum method on all the fields
that are likely to risk crossing this line. Should I be looking at the
termInstances property on the fields that are displayed in the
FieldValueCache on the stats page, and figuring those on the order of 10
million are likely to grow past the limit?
--
Stephen Duncan Jr
www.stephenduncanjr.com
Re: Questions on FieldValueCache
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Mon, Aug 3, 2009 at 2:18 PM, Stephen Duncan
Jr<st...@gmail.com> wrote:
> On Fri, Jul 31, 2009 at 5:23 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
>
>> On Fri, Jul 31, 2009 at 5:06 PM, Stephen Duncan
>> Jr<st...@gmail.com> wrote:
>> > I have a couple more questions on the FieldValueCache. I see that the
>> > number of items in the cache is basically the number of multi-valued
>> fields
>> > facets have been requested for. What does each entry in the cache
>> actually
>> > contain? How does it's size grow as the number of total documents
>> > increases?
>>
>> It's basically an array of int[maxDoc] that contain the list of
>> delta-coded vint values or optionally point out to shared byte arrays
>> if the list of values don't fit in an int. See the javadoc for
>> UnInvertedField for more details.
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>
> I'm getting the following warning in my logs: 2009-08-03 13:41:40,114
> [http-127.0.0.1-8080-1] WARN org.apache.solr.core.SolrCore - Approaching
> too many values for UnInvertedField faceting on field 'originaltext' :
> bucket size=15802492
>
> What's the impact of that? If the number of values (number of unique terms
> for that field, or some other "values"?) exceeds that limit, will faceting
> for that field go back to a different technique and be slower, or...?
It will throw an exception.
This method of faceting wasn't really designed for big full-text fields.
The enum method should work better for this... try something like the following:
f.originaltext.facet.method=enum
facet.enum.cache.minDf=10000
-Yonik
http://www.lucidimagination.com
Re: Questions on FieldValueCache
Posted by Stephen Duncan Jr <st...@gmail.com>.
On Fri, Jul 31, 2009 at 5:23 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
> On Fri, Jul 31, 2009 at 5:06 PM, Stephen Duncan
> Jr<st...@gmail.com> wrote:
> > I have a couple more questions on the FieldValueCache. I see that the
> > number of items in the cache is basically the number of multi-valued
> fields
> > facets have been requested for. What does each entry in the cache
> actually
> > contain? How does it's size grow as the number of total documents
> > increases?
>
> It's basically an array of int[maxDoc] that contain the list of
> delta-coded vint values or optionally point out to shared byte arrays
> if the list of values don't fit in an int. See the javadoc for
> UnInvertedField for more details.
>
> -Yonik
> http://www.lucidimagination.com
>
I'm getting the following warning in my logs: 2009-08-03 13:41:40,114
[http-127.0.0.1-8080-1] WARN org.apache.solr.core.SolrCore - Approaching
too many values for UnInvertedField faceting on field 'originaltext' :
bucket size=15802492
What's the impact of that? If the number of values (number of unique terms
for that field, or some other "values"?) exceeds that limit, will faceting
for that field go back to a different technique and be slower, or...?
--
Stephen Duncan Jr
www.stephenduncanjr.com
Re: Questions on FieldValueCache
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Fri, Jul 31, 2009 at 5:06 PM, Stephen Duncan
Jr<st...@gmail.com> wrote:
> I have a couple more questions on the FieldValueCache. I see that the
> number of items in the cache is basically the number of multi-valued fields
> facets have been requested for. What does each entry in the cache actually
> contain? How does it's size grow as the number of total documents
> increases?
It's basically an array of int[maxDoc] that contain the list of
delta-coded vint values or optionally point out to shared byte arrays
if the list of values don't fit in an int. See the javadoc for
UnInvertedField for more details.
> Ok, so that was the curiosity question. More critical:
>
> When we first ask for facets for multi-valued fields, it can take up to 25
> seconds to get the response, although after that it's very fast (1.5 seconds
> or less even for a lot of different facets) since the FieldValueCache has
> been populated. Obviously, we'd like to get the FieldValueCache populated,
> hopefully without an end-user being the one to wait 25 seconds. I think
> this is what the autowarming settings are about, but I'm not sure how to
> warm the cache with the specific fields we expect facets to be requested
> for. Any pointers?
Use static warming queries. autowarming doesn't hit the fieldValueCache.
-Yonik
http://www.lucidimagination.com