You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Manepalli, Kalyan" <KA...@orbitz.com> on 2009/06/18 01:53:14 UTC
FilterCache issue
Hi,
I am seeing an issue with the filtercache setting on my solr app which is causing slower faceting.
Here is the configuration.
<filterCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="256"/>
Statistics:
description: LRU Cache(maxSize=512, initialSize=512, autowarmCount=256, regenerator=org.apache.solr.search.SolrIndexSearcher$1@8d41f2)
stats: lookups : 979692
hits : 6904
hitratio : 0.00
inserts : 973531
evictions : 972978
size : 512
warmupTime : 1479
cumulative_lookups : 61660491
cumulative_hits : 516057
cumulative_hitratio : 0.00
cumulative_inserts : 61170111
cumulative_evictions : 61153787
As we can see the cache hit ratio is almost zero. How do I improve the filter cache.
Also wanted to know what does the size mean. Is it number of documents or the memory size (kb/mb)
Any suggestions in this regard will be very helpful.
Thanks,
Kalyan Manepalli
Re: FilterCache issue
Posted by Michael Ludwig <ml...@as-guides.com>.
Manepalli, Kalyan schrieb:
> I am seeing an issue with the filtercache setting on my solr app
> which is causing slower faceting.
>
> Here is the configuration.
> <filterCache class="solr.LRUCache" size="512" initialSize="512"
> autowarmCount="256"/>
> hitratio : 0.00
> inserts : 973531
> evictions : 972978
> size : 512
> cumulative_hitratio : 0.00
> cumulative_inserts : 61170111
> cumulative_evictions : 61153787
>
> As we can see the cache hit ratio is almost zero. How do I improve the
> filter cache.
Maybe these pages add some ideas to the mix:
http://wiki.apache.org/solr/FilterQueryGuidance
https://issues.apache.org/jira/browse/SOLR-475
Michael Ludwig
Re: FilterCache issue
Posted by Yonik Seeley <ys...@gmail.com>.
On Thu, Jun 18, 2009 at 1:22 PM, Mark Miller<ma...@gmail.com> wrote:
> Maybe he is not using the FieldCache method?
It occurs to me that this might be nice info to add to debugging info
(the exact method used + perhaps some other info).
-Yonik
http://www.lucidimagination.com
RE: FilterCache issue
Posted by "Manepalli, Kalyan" <KA...@orbitz.com>.
Got that. Since I am still using Solr 1.3, the defaults should work fine, field cache for single value and enum for multi-valued fields.
Thanks,
Kalyan Manepalli
-----Original Message-----
From: Mark Miller [mailto:markrmiller@gmail.com]
Sent: Thursday, June 18, 2009 3:01 PM
To: solr-user@lucene.apache.org
Subject: Re: FilterCache issue
Its the facet.method param:
http://wiki.apache.org/solr/SimpleFacetParameters#head-7574cb658563f6de3ad54cd99a793cd73d593caa
--
- Mark
http://www.lucidimagination.com
Manepalli, Kalyan wrote:
> Mark,
> Where do we specify the method? fieldCache or otherwise
>
> Thanks,
> Kalyan Manepalli
>
> -----Original Message-----
> From: Mark Miller [mailto:markrmiller@gmail.com]
> Sent: Thursday, June 18, 2009 12:22 PM
> To: solr-user@lucene.apache.org
> Subject: Re: FilterCache issue
>
> Maybe he is not using the FieldCache method?
>
> Yonik Seeley wrote:
>
>> On Thu, Jun 18, 2009 at 12:19 PM, Manepalli,
>> Kalyan<KA...@orbitz.com> wrote:
>>
>>
>>> The fields are defined as single valued and they are non tokenized for.
>>> I am using solr 1.3 waiting for release of solr 1.4.
>>>
>>>
>> Then the filterCache won't be used for faceting, just for filters.
>> You should be able to verify this by looking at how the cache stats
>> change for a single faceting request.
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>>
>>
>>
>>> Thanks,
>>> Kalyan Manepalli
>>> -----Original Message-----
>>> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik Seeley
>>> Sent: Thursday, June 18, 2009 10:15 AM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: FilterCache issue
>>>
>>> On Thu, Jun 18, 2009 at 10:59 AM, Manepalli,
>>> Kalyan<KA...@orbitz.com> wrote:
>>>
>>>
>>>> I am faceting on the single values only.
>>>>
>>>>
>>> You may have only added a single value to each field, but is the field
>>> defined to be single valued or multi valued?
>>>
>>> Also, what version of Solr are you using?
>>>
>>>
>>>
>>>
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
Re: FilterCache issue
Posted by Mark Miller <ma...@gmail.com>.
Its the facet.method param:
http://wiki.apache.org/solr/SimpleFacetParameters#head-7574cb658563f6de3ad54cd99a793cd73d593caa
--
- Mark
http://www.lucidimagination.com
Manepalli, Kalyan wrote:
> Mark,
> Where do we specify the method? fieldCache or otherwise
>
> Thanks,
> Kalyan Manepalli
>
> -----Original Message-----
> From: Mark Miller [mailto:markrmiller@gmail.com]
> Sent: Thursday, June 18, 2009 12:22 PM
> To: solr-user@lucene.apache.org
> Subject: Re: FilterCache issue
>
> Maybe he is not using the FieldCache method?
>
> Yonik Seeley wrote:
>
>> On Thu, Jun 18, 2009 at 12:19 PM, Manepalli,
>> Kalyan<KA...@orbitz.com> wrote:
>>
>>
>>> The fields are defined as single valued and they are non tokenized for.
>>> I am using solr 1.3 waiting for release of solr 1.4.
>>>
>>>
>> Then the filterCache won't be used for faceting, just for filters.
>> You should be able to verify this by looking at how the cache stats
>> change for a single faceting request.
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>>
>>
>>
>>> Thanks,
>>> Kalyan Manepalli
>>> -----Original Message-----
>>> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik Seeley
>>> Sent: Thursday, June 18, 2009 10:15 AM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: FilterCache issue
>>>
>>> On Thu, Jun 18, 2009 at 10:59 AM, Manepalli,
>>> Kalyan<KA...@orbitz.com> wrote:
>>>
>>>
>>>> I am faceting on the single values only.
>>>>
>>>>
>>> You may have only added a single value to each field, but is the field
>>> defined to be single valued or multi valued?
>>>
>>> Also, what version of Solr are you using?
>>>
>>>
>>>
>>>
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
>
RE: FilterCache issue
Posted by "Manepalli, Kalyan" <KA...@orbitz.com>.
Mark,
Where do we specify the method? fieldCache or otherwise
Thanks,
Kalyan Manepalli
-----Original Message-----
From: Mark Miller [mailto:markrmiller@gmail.com]
Sent: Thursday, June 18, 2009 12:22 PM
To: solr-user@lucene.apache.org
Subject: Re: FilterCache issue
Maybe he is not using the FieldCache method?
Yonik Seeley wrote:
> On Thu, Jun 18, 2009 at 12:19 PM, Manepalli,
> Kalyan<KA...@orbitz.com> wrote:
>
>> The fields are defined as single valued and they are non tokenized for.
>> I am using solr 1.3 waiting for release of solr 1.4.
>>
>
> Then the filterCache won't be used for faceting, just for filters.
> You should be able to verify this by looking at how the cache stats
> change for a single faceting request.
>
> -Yonik
> http://www.lucidimagination.com
>
>
>
>
>> Thanks,
>> Kalyan Manepalli
>> -----Original Message-----
>> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik Seeley
>> Sent: Thursday, June 18, 2009 10:15 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: FilterCache issue
>>
>> On Thu, Jun 18, 2009 at 10:59 AM, Manepalli,
>> Kalyan<KA...@orbitz.com> wrote:
>>
>>> I am faceting on the single values only.
>>>
>> You may have only added a single value to each field, but is the field
>> defined to be single valued or multi valued?
>>
>> Also, what version of Solr are you using?
>>
>>
>>
--
- Mark
http://www.lucidimagination.com
Re: FilterCache issue
Posted by Mark Miller <ma...@gmail.com>.
Maybe he is not using the FieldCache method?
Yonik Seeley wrote:
> On Thu, Jun 18, 2009 at 12:19 PM, Manepalli,
> Kalyan<KA...@orbitz.com> wrote:
>
>> The fields are defined as single valued and they are non tokenized for.
>> I am using solr 1.3 waiting for release of solr 1.4.
>>
>
> Then the filterCache won't be used for faceting, just for filters.
> You should be able to verify this by looking at how the cache stats
> change for a single faceting request.
>
> -Yonik
> http://www.lucidimagination.com
>
>
>
>
>> Thanks,
>> Kalyan Manepalli
>> -----Original Message-----
>> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik Seeley
>> Sent: Thursday, June 18, 2009 10:15 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: FilterCache issue
>>
>> On Thu, Jun 18, 2009 at 10:59 AM, Manepalli,
>> Kalyan<KA...@orbitz.com> wrote:
>>
>>> I am faceting on the single values only.
>>>
>> You may have only added a single value to each field, but is the field
>> defined to be single valued or multi valued?
>>
>> Also, what version of Solr are you using?
>>
>>
>>
--
- Mark
http://www.lucidimagination.com
Re: FilterCache issue
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Thu, Jun 18, 2009 at 12:19 PM, Manepalli,
Kalyan<KA...@orbitz.com> wrote:
> The fields are defined as single valued and they are non tokenized for.
> I am using solr 1.3 waiting for release of solr 1.4.
Then the filterCache won't be used for faceting, just for filters.
You should be able to verify this by looking at how the cache stats
change for a single faceting request.
-Yonik
http://www.lucidimagination.com
> Thanks,
> Kalyan Manepalli
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik Seeley
> Sent: Thursday, June 18, 2009 10:15 AM
> To: solr-user@lucene.apache.org
> Subject: Re: FilterCache issue
>
> On Thu, Jun 18, 2009 at 10:59 AM, Manepalli,
> Kalyan<KA...@orbitz.com> wrote:
>> I am faceting on the single values only.
>
> You may have only added a single value to each field, but is the field
> defined to be single valued or multi valued?
>
> Also, what version of Solr are you using?
>
>
RE: FilterCache issue
Posted by "Manepalli, Kalyan" <KA...@orbitz.com>.
The fields are defined as single valued and they are non tokenized for.
I am using solr 1.3 waiting for release of solr 1.4.
Thanks,
Kalyan Manepalli
-----Original Message-----
From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik Seeley
Sent: Thursday, June 18, 2009 10:15 AM
To: solr-user@lucene.apache.org
Subject: Re: FilterCache issue
On Thu, Jun 18, 2009 at 10:59 AM, Manepalli,
Kalyan<KA...@orbitz.com> wrote:
> I am faceting on the single values only.
You may have only added a single value to each field, but is the field
defined to be single valued or multi valued?
Also, what version of Solr are you using?
-Yonik
http://www.lucidimagination.com
Re: FilterCache issue
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Thu, Jun 18, 2009 at 10:59 AM, Manepalli,
Kalyan<KA...@orbitz.com> wrote:
> I am faceting on the single values only.
You may have only added a single value to each field, but is the field
defined to be single valued or multi valued?
Also, what version of Solr are you using?
-Yonik
http://www.lucidimagination.com
RE: FilterCache issue
Posted by "Manepalli, Kalyan" <KA...@orbitz.com>.
I am faceting on the single values only. I ran load test against solr app and found that under increased load the faceting just gets slower and slower. That is why I wanted to investigate filtercache and any other features to tweak the performance.
As suggested by Mark in the earlier email, I increased the size of filtercache and the performance has improved. I need to further test to see the impact on other areas.
Thanks,
Kalyan Manepalli
-----Original Message-----
From: Mark Miller [mailto:markrmiller@gmail.com]
Sent: Thursday, June 18, 2009 9:15 AM
To: solr-user@lucene.apache.org
Subject: Re: FilterCache issue
Mark Miller wrote:
> Yonik Seeley wrote:
>> On Thu, Jun 18, 2009 at 8:35 AM, Mark Miller<ma...@gmail.com>
>> wrote:
>>
>>> Thats why I asked about multi-valued terms. If hes not using the enum
>>> faceting method (which only makes sense with fewer uniques), and the
>>> fields
>>> are not multi-valued, than it is using the FieldCache method. Which of
>>> course does use the filterCache,
>>>
>>
>> The FieldCache method for single-valued fields does not use the
>> filterCache... that's only for big terms on multi-valued fields.
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
> Ah, I think the wiki is incorrect then.
>
> SolrCaching
>
> If you use faceting with the fieldCache method (see
> SolrFacetingOverview
> <http://wiki.apache.org/solr/SolrFacetingOverview>), it is recommended
> that you set the filterCache size to be greater than the number of
> unique values in all of your faceted fields.
>
Thats some pretty misleading info. I was wondering how the heck the
filterCache played into counting off a FieldCache.
--
- Mark
http://www.lucidimagination.com
Re: FilterCache issue
Posted by Mark Miller <ma...@gmail.com>.
Mark Miller wrote:
> Yonik Seeley wrote:
>> On Thu, Jun 18, 2009 at 8:35 AM, Mark Miller<ma...@gmail.com>
>> wrote:
>>
>>> Thats why I asked about multi-valued terms. If hes not using the enum
>>> faceting method (which only makes sense with fewer uniques), and the
>>> fields
>>> are not multi-valued, than it is using the FieldCache method. Which of
>>> course does use the filterCache,
>>>
>>
>> The FieldCache method for single-valued fields does not use the
>> filterCache... that's only for big terms on multi-valued fields.
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
> Ah, I think the wiki is incorrect then.
>
> SolrCaching
>
> If you use faceting with the fieldCache method (see
> SolrFacetingOverview
> <http://wiki.apache.org/solr/SolrFacetingOverview>), it is recommended
> that you set the filterCache size to be greater than the number of
> unique values in all of your faceted fields.
>
Thats some pretty misleading info. I was wondering how the heck the
filterCache played into counting off a FieldCache.
--
- Mark
http://www.lucidimagination.com
Re: FilterCache issue
Posted by Mark Miller <ma...@gmail.com>.
Yonik Seeley wrote:
> On Thu, Jun 18, 2009 at 8:35 AM, Mark Miller<ma...@gmail.com> wrote:
>
>> Thats why I asked about multi-valued terms. If hes not using the enum
>> faceting method (which only makes sense with fewer uniques), and the fields
>> are not multi-valued, than it is using the FieldCache method. Which of
>> course does use the filterCache,
>>
>
> The FieldCache method for single-valued fields does not use the
> filterCache... that's only for big terms on multi-valued fields.
>
> -Yonik
> http://www.lucidimagination.com
>
Ah, I think the wiki is incorrect then.
SolrCaching
If you use faceting with the fieldCache method (see SolrFacetingOverview
<http://wiki.apache.org/solr/SolrFacetingOverview>), it is recommended
that you set the filterCache size to be greater than the number of
unique values in all of your faceted fields.
--
- Mark
http://www.lucidimagination.com
Re: FilterCache issue
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Thu, Jun 18, 2009 at 8:35 AM, Mark Miller<ma...@gmail.com> wrote:
> Thats why I asked about multi-valued terms. If hes not using the enum
> faceting method (which only makes sense with fewer uniques), and the fields
> are not multi-valued, than it is using the FieldCache method. Which of
> course does use the filterCache,
The FieldCache method for single-valued fields does not use the
filterCache... that's only for big terms on multi-valued fields.
-Yonik
http://www.lucidimagination.com
Re: FilterCache issue
Posted by Mark Miller <ma...@gmail.com>.
Thats why I asked about multi-valued terms. If hes not using the enum
faceting method (which only makes sense with fewer uniques), and the
fields are not multi-valued, than it is using the FieldCache method.
Which of course does use the filterCache, and works best when the
filterCache size is the size of all the unique terms in all the fields
you are faceting on.
Perhaps his machine can't handle that, but certainly he would benefit
heavily from the cache. And if you had tons of uniques and a small
cache, you would see exactly what he is seeing. Probably best to see how
it goes before seeing if you have to optimize based on garbage
collection (and I beleive that he could run in to resource issues, whats
why I said give it a shot :) ). He may just need more resources than he
can get, but I'm fairly sure he needs the resources. Unless its
multivalued fields and its using the UninvertedField - not sure how the
filterCache plays into that.
- Mark
Grant Ingersoll wrote:
>
> On Jun 17, 2009, at 10:32 PM, Mark Miller wrote:
>
>> Right, so if you are on 1.3 or early 1.4 dev, with so many uniques,
>> you should be using the FieldCache method of faceting. The RAM
>> depends on the number of documents and number of uniques terms mostly.
>>
>> With 1.4 you may be using an UninvertedField though (are your facet
>> fields multivalued or tokenized?), and I know much less about that.
>>
>> I'd try a cache size of 10,000 and see how it goes.
>
> I'm not so sure about that, my guess is your going to get hammered on
> garbage collection when you do commits with something that big.
>
> Let's take a step back. These are LRU cache's, the fact that you have
> a zero hit ratio does not mean caching isn't working or that you
> necessarily need a bigger cache. It suggests to me that your
> application is not the type that can benefit from caching of
> filters. My understanding is that in certain cases with the new 1.4
> faceting, it ends up using the filterCache as well. I believe the
> admin will give stats on the number of big terms, etc.
>
> Perhaps you can give a bit more detail about your application and why
> you think that cache ratio is causing slower faceting.
>
> Have you actually done some profiling/timings on the faceting?
>
>
>>
>> - Mark
>>
>> Manepalli, Kalyan wrote:
>>> Got that, if its number of cache entries, definitely its very low. I
>>> have around 10,000 unique items to facet on. Does the RAM size
>>> depend on Document size.
>>>
>>> Thanks,
>>> Kalyan Manepalli
>>> -----Original Message-----
>>> From: Mark Miller [mailto:markrmiller@gmail.com]
>>> Sent: Wednesday, June 17, 2009 7:13 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: FilterCache issue
>>>
>>> Its been a while since I've thought about this sort of thing, but it
>>> looks like your cache is way too small and things get evicted before
>>> being used. How many uniques are you faceting on? 512 is the number of
>>> cache entries, not the size in kb/mb.
>>>
>>> Try raising it - perhaps a lot ;) But consider that you have to have
>>> the
>>> RAM to accommodate as well ...
>>>
>>> What version of Solr are you using?
>>>
>>> --
>>> - Mark
>>>
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>> Manepalli, Kalyan wrote:
>>>
>>>> Hi,
>>>> I am seeing an issue with the filtercache setting on my
>>>> solr app which is causing slower faceting.
>>>>
>>>> Here is the configuration.
>>>> <filterCache class="solr.LRUCache" size="512" initialSize="512"
>>>> autowarmCount="256"/>
>>>>
>>>> Statistics:
>>>> description: LRU Cache(maxSize=512, initialSize=512,
>>>> autowarmCount=256,
>>>> regenerator=org.apache.solr.search.SolrIndexSearcher$1@8d41f2)
>>>> stats: lookups : 979692
>>>> hits : 6904
>>>> hitratio : 0.00
>>>> inserts : 973531
>>>> evictions : 972978
>>>> size : 512
>>>> warmupTime : 1479
>>>> cumulative_lookups : 61660491
>>>> cumulative_hits : 516057
>>>> cumulative_hitratio : 0.00
>>>> cumulative_inserts : 61170111
>>>> cumulative_evictions : 61153787
>>>>
>>>> As we can see the cache hit ratio is almost zero. How do I improve
>>>> the filter cache.
>>>> Also wanted to know what does the size mean. Is it number of
>>>> documents or the memory size (kb/mb)
>>>>
>>>> Any suggestions in this regard will be very helpful.
>>>>
>>>> Thanks,
>>>> Kalyan Manepalli
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>
>
> --------------------------
> Grant Ingersoll
> http://www.lucidimagination.com/
>
> Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
> using Solr/Lucene:
> http://www.lucidimagination.com/search
>
--
- Mark
http://www.lucidimagination.com
Re: FilterCache issue
Posted by Grant Ingersoll <gs...@apache.org>.
On Jun 17, 2009, at 10:32 PM, Mark Miller wrote:
> Right, so if you are on 1.3 or early 1.4 dev, with so many uniques,
> you should be using the FieldCache method of faceting. The RAM
> depends on the number of documents and number of uniques terms mostly.
>
> With 1.4 you may be using an UninvertedField though (are your facet
> fields multivalued or tokenized?), and I know much less about that.
>
> I'd try a cache size of 10,000 and see how it goes.
I'm not so sure about that, my guess is your going to get hammered on
garbage collection when you do commits with something that big.
Let's take a step back. These are LRU cache's, the fact that you have
a zero hit ratio does not mean caching isn't working or that you
necessarily need a bigger cache. It suggests to me that your
application is not the type that can benefit from caching of
filters. My understanding is that in certain cases with the new 1.4
faceting, it ends up using the filterCache as well. I believe the
admin will give stats on the number of big terms, etc.
Perhaps you can give a bit more detail about your application and why
you think that cache ratio is causing slower faceting.
Have you actually done some profiling/timings on the faceting?
>
> - Mark
>
> Manepalli, Kalyan wrote:
>> Got that, if its number of cache entries, definitely its very low.
>> I have around 10,000 unique items to facet on. Does the RAM size
>> depend on Document size.
>>
>> Thanks,
>> Kalyan Manepalli
>> -----Original Message-----
>> From: Mark Miller [mailto:markrmiller@gmail.com]
>> Sent: Wednesday, June 17, 2009 7:13 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: FilterCache issue
>>
>> Its been a while since I've thought about this sort of thing, but it
>> looks like your cache is way too small and things get evicted before
>> being used. How many uniques are you faceting on? 512 is the number
>> of
>> cache entries, not the size in kb/mb.
>>
>> Try raising it - perhaps a lot ;) But consider that you have to
>> have the
>> RAM to accommodate as well ...
>>
>> What version of Solr are you using?
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>
>> Manepalli, Kalyan wrote:
>>
>>> Hi,
>>> I am seeing an issue with the filtercache setting on my
>>> solr app which is causing slower faceting.
>>>
>>> Here is the configuration.
>>> <filterCache class="solr.LRUCache" size="512" initialSize="512"
>>> autowarmCount="256"/>
>>>
>>> Statistics:
>>> description: LRU Cache(maxSize=512, initialSize=512,
>>> autowarmCount=256,
>>> regenerator=org.apache.solr.search.SolrIndexSearcher$1@8d41f2)
>>> stats: lookups : 979692
>>> hits : 6904
>>> hitratio : 0.00
>>> inserts : 973531
>>> evictions : 972978
>>> size : 512
>>> warmupTime : 1479
>>> cumulative_lookups : 61660491
>>> cumulative_hits : 516057
>>> cumulative_hitratio : 0.00
>>> cumulative_inserts : 61170111
>>> cumulative_evictions : 61153787
>>>
>>> As we can see the cache hit ratio is almost zero. How do I improve
>>> the filter cache.
>>> Also wanted to know what does the size mean. Is it number of
>>> documents or the memory size (kb/mb)
>>>
>>> Any suggestions in this regard will be very helpful.
>>>
>>> Thanks,
>>> Kalyan Manepalli
>>>
>>>
>>>
>>>
>>
>>
>>
>>
>>
>
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search
Re: FilterCache issue
Posted by Mark Miller <ma...@gmail.com>.
Right, so if you are on 1.3 or early 1.4 dev, with so many uniques, you
should be using the FieldCache method of faceting. The RAM depends on
the number of documents and number of uniques terms mostly.
With 1.4 you may be using an UninvertedField though (are your facet
fields multivalued or tokenized?), and I know much less about that.
I'd try a cache size of 10,000 and see how it goes.
- Mark
Manepalli, Kalyan wrote:
> Got that, if its number of cache entries, definitely its very low. I have around 10,000 unique items to facet on. Does the RAM size depend on Document size.
>
> Thanks,
> Kalyan Manepalli
> -----Original Message-----
> From: Mark Miller [mailto:markrmiller@gmail.com]
> Sent: Wednesday, June 17, 2009 7:13 PM
> To: solr-user@lucene.apache.org
> Subject: Re: FilterCache issue
>
> Its been a while since I've thought about this sort of thing, but it
> looks like your cache is way too small and things get evicted before
> being used. How many uniques are you faceting on? 512 is the number of
> cache entries, not the size in kb/mb.
>
> Try raising it - perhaps a lot ;) But consider that you have to have the
> RAM to accommodate as well ...
>
> What version of Solr are you using?
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
>
>
> Manepalli, Kalyan wrote:
>
>> Hi,
>> I am seeing an issue with the filtercache setting on my solr app which is causing slower faceting.
>>
>> Here is the configuration.
>> <filterCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="256"/>
>>
>> Statistics:
>> description: LRU Cache(maxSize=512, initialSize=512, autowarmCount=256, regenerator=org.apache.solr.search.SolrIndexSearcher$1@8d41f2)
>> stats: lookups : 979692
>> hits : 6904
>> hitratio : 0.00
>> inserts : 973531
>> evictions : 972978
>> size : 512
>> warmupTime : 1479
>> cumulative_lookups : 61660491
>> cumulative_hits : 516057
>> cumulative_hitratio : 0.00
>> cumulative_inserts : 61170111
>> cumulative_evictions : 61153787
>>
>> As we can see the cache hit ratio is almost zero. How do I improve the filter cache.
>> Also wanted to know what does the size mean. Is it number of documents or the memory size (kb/mb)
>>
>> Any suggestions in this regard will be very helpful.
>>
>> Thanks,
>> Kalyan Manepalli
>>
>>
>>
>>
>
>
>
>
>
--
- Mark
http://www.lucidimagination.com
RE: FilterCache issue
Posted by "Manepalli, Kalyan" <KA...@orbitz.com>.
Got that, if its number of cache entries, definitely its very low. I have around 10,000 unique items to facet on. Does the RAM size depend on Document size.
Thanks,
Kalyan Manepalli
-----Original Message-----
From: Mark Miller [mailto:markrmiller@gmail.com]
Sent: Wednesday, June 17, 2009 7:13 PM
To: solr-user@lucene.apache.org
Subject: Re: FilterCache issue
Its been a while since I've thought about this sort of thing, but it
looks like your cache is way too small and things get evicted before
being used. How many uniques are you faceting on? 512 is the number of
cache entries, not the size in kb/mb.
Try raising it - perhaps a lot ;) But consider that you have to have the
RAM to accommodate as well ...
What version of Solr are you using?
--
- Mark
http://www.lucidimagination.com
Manepalli, Kalyan wrote:
> Hi,
> I am seeing an issue with the filtercache setting on my solr app which is causing slower faceting.
>
> Here is the configuration.
> <filterCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="256"/>
>
> Statistics:
> description: LRU Cache(maxSize=512, initialSize=512, autowarmCount=256, regenerator=org.apache.solr.search.SolrIndexSearcher$1@8d41f2)
> stats: lookups : 979692
> hits : 6904
> hitratio : 0.00
> inserts : 973531
> evictions : 972978
> size : 512
> warmupTime : 1479
> cumulative_lookups : 61660491
> cumulative_hits : 516057
> cumulative_hitratio : 0.00
> cumulative_inserts : 61170111
> cumulative_evictions : 61153787
>
> As we can see the cache hit ratio is almost zero. How do I improve the filter cache.
> Also wanted to know what does the size mean. Is it number of documents or the memory size (kb/mb)
>
> Any suggestions in this regard will be very helpful.
>
> Thanks,
> Kalyan Manepalli
>
>
>
Re: FilterCache issue
Posted by Mark Miller <ma...@gmail.com>.
Its been a while since I've thought about this sort of thing, but it
looks like your cache is way too small and things get evicted before
being used. How many uniques are you faceting on? 512 is the number of
cache entries, not the size in kb/mb.
Try raising it - perhaps a lot ;) But consider that you have to have the
RAM to accommodate as well ...
What version of Solr are you using?
--
- Mark
http://www.lucidimagination.com
Manepalli, Kalyan wrote:
> Hi,
> I am seeing an issue with the filtercache setting on my solr app which is causing slower faceting.
>
> Here is the configuration.
> <filterCache class="solr.LRUCache" size="512" initialSize="512" autowarmCount="256"/>
>
> Statistics:
> description: LRU Cache(maxSize=512, initialSize=512, autowarmCount=256, regenerator=org.apache.solr.search.SolrIndexSearcher$1@8d41f2)
> stats: lookups : 979692
> hits : 6904
> hitratio : 0.00
> inserts : 973531
> evictions : 972978
> size : 512
> warmupTime : 1479
> cumulative_lookups : 61660491
> cumulative_hits : 516057
> cumulative_hitratio : 0.00
> cumulative_inserts : 61170111
> cumulative_evictions : 61153787
>
> As we can see the cache hit ratio is almost zero. How do I improve the filter cache.
> Also wanted to know what does the size mean. Is it number of documents or the memory size (kb/mb)
>
> Any suggestions in this regard will be very helpful.
>
> Thanks,
> Kalyan Manepalli
>
>
>