How to warm up filter queries for a category field with 1000 possible values ?

[user 01 <us...@gmail.com>]

what's the way to warm up filter queries for a category field with 1000
possible values. Would I need to write 1000 lines manually in the
solrconig.xml or what is the format?

Re: How to warm up filter queries for a category field with 1000 possible values ?

[Erick Erickson <er...@gmail.com>]

That's what the "autowarm" number for filterCache is about. It
re-executes the last N fq clauses and caches them. Similarly
for some of the other autowarm.

But don't go wild here. Measure _then_ fix. Usually autowarming
just a few  (< 32) is sufficient. And remember that autowarming
is done whenever you open a new searcher, so if you have your
soft commits configured to be, say, 5 seconds you'll get minimal
benefit here.

Are you saying you have almost 1,000 differently-named fields in
your documents? Or 1,000 _values_ in your category field? In either
case, please measure your query performance before assuming
you need to autowarm excessively. If you don't autowarm at all
and your first few queries are acceptable after you open a new searcher
then don't worry about it.

Often the biggest win is just filling the lower-level Lucene caches
which you can do with a very few queries.

Best
Erick

On Mon, Oct 7, 2013 at 2:49 AM, user 01 <us...@gmail.com> wrote:
> Sorry, didn't get you exactly.
>
> I need to warm up my queries after the newSearcher & firstSearcher are
> initialized. I am trying to warm up the filter caches for a category field
> but I have almost 1000 categories(& changing with time), which make it
> impossible to list them in solrConfig.xml. Is there any way to iterate over
> all categories & warm up the query for each ?
>
>
> On Mon, Oct 7, 2013 at 12:10 PM, Furkan KAMACI <fu...@gmail.com>wrote:
>
>> If you are asking to read from a file for warm up and if there is not a
>> capability for what you want I can open a Jira issue and send a patch.
>>
>>
>> 2013/10/7 user 01 <us...@gmail.com>
>>
>> > what's the way to warm up filter queries for a category field with 1000
>> > possible values. Would I need to write 1000 lines manually in the
>> > solrconig.xml or what is the format?
>> >
>>

Re: How to warm up filter queries for a category field with 1000 possible values ?

[user 01 <us...@gmail.com>]

Sorry, didn't get you exactly.

I need to warm up my queries after the newSearcher & firstSearcher are
initialized. I am trying to warm up the filter caches for a category field
but I have almost 1000 categories(& changing with time), which make it
impossible to list them in solrConfig.xml. Is there any way to iterate over
all categories & warm up the query for each ?


On Mon, Oct 7, 2013 at 12:10 PM, Furkan KAMACI <fu...@gmail.com>wrote:

> If you are asking to read from a file for warm up and if there is not a
> capability for what you want I can open a Jira issue and send a patch.
>
>
> 2013/10/7 user 01 <us...@gmail.com>
>
> > what's the way to warm up filter queries for a category field with 1000
> > possible values. Would I need to write 1000 lines manually in the
> > solrconig.xml or what is the format?
> >
>

Re: How to warm up filter queries for a category field with 1000 possible values ?

[Furkan KAMACI <fu...@gmail.com>]

If you are asking to read from a file for warm up and if there is not a
capability for what you want I can open a Jira issue and send a patch.


2013/10/7 user 01 <us...@gmail.com>

> what's the way to warm up filter queries for a category field with 1000
> possible values. Would I need to write 1000 lines manually in the
> solrconig.xml or what is the format?
>

Re: How to warm up filter queries for a category field with 1000 possible values ?

[Shawn Heisey <so...@elyograg.org>]

On 10/7/2013 12:36 AM, user 01 wrote:
> what's the way to warm up filter queries for a category field with 1000
> possible values. Would I need to write 1000 lines manually in the
> solrconig.xml or what is the format?

Erick has given you awesome advice.  Here's something a little bit 
different that doesn't invalidate his advice:

If you have enough free RAM (not used by programs) for good OS disk 
caching, then as soon as you do one query that checks this field, then 
all 1000 values for that field are likely to be in RAM, and the next 
query against that field is going to be lightning fast, because the 
operating system will not have to read the disk to get the information.  
Although it is slightly faster to get informatin out of Solr's caches 
than the OS disk cache, the operating system is far better at managing 
huge caches than Solr and Java are.

http://wiki.apache.org/solr/SolrPerformanceProblems#General_information

Thanks,
Shawn