newSearcher vs. firstSearcher

[Jonathan Rochkind <ro...@jhu.edu>]

I understand the difference between newSearcher and firstSearcher in 
terms of when they are run.

But thinking about warming queries, which is my use of new/firstSearcher 
(and probably the most common use?), I can't think of any case but ones 
where I'd want newSearcher and firstSearcher warming queries to be 
identical.

Is there something I'm missing, are there reasons/cases where you'd want 
your newSearcher warming queries to be different than your firstSearcher 
ones?

If at least I'm write that _commonly_ you want them to be the same...  
then it seems like it would be nice if there were a way to specify 
warming queries that would be run for _both_ newSearcher and 
firstSearcher, rather than have to duplicate the warming queries in both 
places.  The duplication means every time I change a warming query, I 
need to remember to change it in both places. Is there a way to avoid 
this duplication that I'm missing?

Jonathan

Re: newSearcher vs. firstSearcher

[Chris Hostetter <ho...@fucit.org>]

: But thinking about warming queries, which is my use of new/firstSearcher (and
: probably the most common use?), I can't think of any case but ones where I'd
: want newSearcher and firstSearcher warming queries to be identical.

a firstSearcher event is one in which there is no previous searcher, so 
the *only* warming is the warming you do explicitly.

a newSearcher event happens when there is an "old" searcher, and 
"autowarming" will be happening for the caches that support it -- so you 
may not need as much explicit warming.

It's common to have a lot of warming configured to explicitly happen on 
firstSearcher, but thenhave much less warming configured to explicitly 
happen on newSearcher because of the implicit warming.

-Hoss

Re: newSearcher vs. firstSearcher

[Jonathan Rochkind <ro...@jhu.edu>]

Okay, it is all somewhat complicated, I understand better now, thanks.

I think it's probably a very common case to want to run warming queries 
only on commits (or other things that change the index; replication 
definitely counts!), and not on other newSearcher events that don't 
result in a new index requiring new warming.  Yes?

It seems there is really no way to configure that though. It would be 
awfully nice if there were, maybe a new event newSearcherForNewIndex or 
something.  But I realize nobody is probably going to write that patch 
for me, just putting it out there.

Markus Jelsma wrote:
>> Thanks for the XInclude suggestion -- that does require having my
>> warming query specified in an external file, right?
>>     
>
> Yes, just add it in the same solr_home/conf directory and you'll be fine. See 
> wiki for example and fallback scenario's.
>
>   
>> As far as when you might not want to have a firstSearcher warming query,
>> can you help me understand that better? Maybe I don't actually
>> understand when a new searcher ends up getting created after all. I
>> think that after a commit, that'll be newSearcher, not firstSearcher,
>> right?  And after a commit, since the index has changed, aren't you
>> going to need/want to run warming queries again?
>>     
>
> NewSearchers are created at various points. A commit/optimize can cause a new 
> searchers to be created (not first indeed), depending on their optional 
> attributes. Also reloading a core, replication etc will cause searcher to get 
> created, resp. first and new searchers to be created. 
> A new (or first) searcher is required to be able to search a new version of 
> your index. The old searcher will handle the current requests and the new will 
> search the new index version.
>
> After a commit/optimize or replication, you may want to have some warm up 
> queries standing by if your caches won't get properly populated by the 
> previous (and now old) searchers. Depends on your index, usage of it, 
> frequency of change and requests etc.
>
> Also see https://issues.apache.org/jira/browse/SOLR-1530 for an issue to allow 
> lazy loading of searchers in frequent commit environments.
>
>   
>> I guess, if I'm right about that, there are other cases where a new
>> searcher is created, that aren't the result of a commit?  Is there any
>> way to configure a warming query that will run after commit, but not
>> after these other events that create a new searcher?
>>     
>
> No. You'll only have one set of warm up queries for the newSearcher, Solr 
> won't make a distintion between the triggers of that newSearcher.
>
>   
>> Markus Jelsma wrote:
>>     
>>> Hi,
>>>
>>> Use XInclude to overcome the problem of maintaining two separate sets of
>>> warm up queries if they are the same for both searchers.
>>>
>>> A use-case for different sets of warm up queries is that a new searcher's
>>> cache can get populated by the cache of the old searcher, thus negating
>>> the need for warmup queries that are in the cache anyway (i.e. very
>>> common queries and filters). The first searcher's cache can't get
>>> populated so there is a need to warm up with common queries or filters.
>>>
>>> Cheers,
>>>
>>>       
>>>> I understand the difference between newSearcher and firstSearcher in
>>>> terms of when they are run.
>>>>
>>>> But thinking about warming queries, which is my use of new/firstSearcher
>>>> (and probably the most common use?), I can't think of any case but ones
>>>> where I'd want newSearcher and firstSearcher warming queries to be
>>>> identical.
>>>>
>>>> Is there something I'm missing, are there reasons/cases where you'd want
>>>> your newSearcher warming queries to be different than your firstSearcher
>>>> ones?
>>>>
>>>> If at least I'm write that _commonly_ you want them to be the same...
>>>> then it seems like it would be nice if there were a way to specify
>>>> warming queries that would be run for _both_ newSearcher and
>>>> firstSearcher, rather than have to duplicate the warming queries in both
>>>> places.  The duplication means every time I change a warming query, I
>>>> need to remember to change it in both places. Is there a way to avoid
>>>> this duplication that I'm missing?
>>>>
>>>> Jonathan
>>>>         

Re: newSearcher vs. firstSearcher

[Markus Jelsma <ma...@openindex.io>]

> Thanks for the XInclude suggestion -- that does require having my
> warming query specified in an external file, right?

Yes, just add it in the same solr_home/conf directory and you'll be fine. See 
wiki for example and fallback scenario's.

> 
> As far as when you might not want to have a firstSearcher warming query,
> can you help me understand that better? Maybe I don't actually
> understand when a new searcher ends up getting created after all. I
> think that after a commit, that'll be newSearcher, not firstSearcher,
> right?  And after a commit, since the index has changed, aren't you
> going to need/want to run warming queries again?

NewSearchers are created at various points. A commit/optimize can cause a new 
searchers to be created (not first indeed), depending on their optional 
attributes. Also reloading a core, replication etc will cause searcher to get 
created, resp. first and new searchers to be created. 
A new (or first) searcher is required to be able to search a new version of 
your index. The old searcher will handle the current requests and the new will 
search the new index version.

After a commit/optimize or replication, you may want to have some warm up 
queries standing by if your caches won't get properly populated by the 
previous (and now old) searchers. Depends on your index, usage of it, 
frequency of change and requests etc.

Also see https://issues.apache.org/jira/browse/SOLR-1530 for an issue to allow 
lazy loading of searchers in frequent commit environments.

> 
> I guess, if I'm right about that, there are other cases where a new
> searcher is created, that aren't the result of a commit?  Is there any
> way to configure a warming query that will run after commit, but not
> after these other events that create a new searcher?

No. You'll only have one set of warm up queries for the newSearcher, Solr 
won't make a distintion between the triggers of that newSearcher.

> 
> Markus Jelsma wrote:
> > Hi,
> > 
> > Use XInclude to overcome the problem of maintaining two separate sets of
> > warm up queries if they are the same for both searchers.
> > 
> > A use-case for different sets of warm up queries is that a new searcher's
> > cache can get populated by the cache of the old searcher, thus negating
> > the need for warmup queries that are in the cache anyway (i.e. very
> > common queries and filters). The first searcher's cache can't get
> > populated so there is a need to warm up with common queries or filters.
> > 
> > Cheers,
> > 
> >> I understand the difference between newSearcher and firstSearcher in
> >> terms of when they are run.
> >> 
> >> But thinking about warming queries, which is my use of new/firstSearcher
> >> (and probably the most common use?), I can't think of any case but ones
> >> where I'd want newSearcher and firstSearcher warming queries to be
> >> identical.
> >> 
> >> Is there something I'm missing, are there reasons/cases where you'd want
> >> your newSearcher warming queries to be different than your firstSearcher
> >> ones?
> >> 
> >> If at least I'm write that _commonly_ you want them to be the same...
> >> then it seems like it would be nice if there were a way to specify
> >> warming queries that would be run for _both_ newSearcher and
> >> firstSearcher, rather than have to duplicate the warming queries in both
> >> places.  The duplication means every time I change a warming query, I
> >> need to remember to change it in both places. Is there a way to avoid
> >> this duplication that I'm missing?
> >> 
> >> Jonathan

Re: newSearcher vs. firstSearcher

[Jonathan Rochkind <ro...@jhu.edu>]

Thanks for the XInclude suggestion -- that does require having my 
warming query specified in an external file, right?

As far as when you might not want to have a firstSearcher warming query, 
can you help me understand that better? Maybe I don't actually 
understand when a new searcher ends up getting created after all. I 
think that after a commit, that'll be newSearcher, not firstSearcher, 
right?  And after a commit, since the index has changed, aren't you 
going to need/want to run warming queries again?

I guess, if I'm right about that, there are other cases where a new 
searcher is created, that aren't the result of a commit?  Is there any 
way to configure a warming query that will run after commit, but not 
after these other events that create a new searcher?

Markus Jelsma wrote:
> Hi,
>
> Use XInclude to overcome the problem of maintaining two separate sets of warm 
> up queries if they are the same for both searchers.
>
> A use-case for different sets of warm up queries is that a new searcher's cache 
> can get populated by the cache of the old searcher, thus negating the need for 
> warmup queries that are in the cache anyway (i.e. very common queries and 
> filters). The first searcher's cache can't get populated so there is a need to 
> warm up with common queries or filters.
>
> Cheers,
>
>   
>> I understand the difference between newSearcher and firstSearcher in
>> terms of when they are run.
>>
>> But thinking about warming queries, which is my use of new/firstSearcher
>> (and probably the most common use?), I can't think of any case but ones
>> where I'd want newSearcher and firstSearcher warming queries to be
>> identical.
>>
>> Is there something I'm missing, are there reasons/cases where you'd want
>> your newSearcher warming queries to be different than your firstSearcher
>> ones?
>>
>> If at least I'm write that _commonly_ you want them to be the same...
>> then it seems like it would be nice if there were a way to specify
>> warming queries that would be run for _both_ newSearcher and
>> firstSearcher, rather than have to duplicate the warming queries in both
>> places.  The duplication means every time I change a warming query, I
>> need to remember to change it in both places. Is there a way to avoid
>> this duplication that I'm missing?
>>
>> Jonathan
>>     
>
>   

Re: newSearcher vs. firstSearcher

[Markus Jelsma <ma...@openindex.io>]

Hi,

Use XInclude to overcome the problem of maintaining two separate sets of warm 
up queries if they are the same for both searchers.

A use-case for different sets of warm up queries is that a new searcher's cache 
can get populated by the cache of the old searcher, thus negating the need for 
warmup queries that are in the cache anyway (i.e. very common queries and 
filters). The first searcher's cache can't get populated so there is a need to 
warm up with common queries or filters.

Cheers,

> I understand the difference between newSearcher and firstSearcher in
> terms of when they are run.
> 
> But thinking about warming queries, which is my use of new/firstSearcher
> (and probably the most common use?), I can't think of any case but ones
> where I'd want newSearcher and firstSearcher warming queries to be
> identical.
> 
> Is there something I'm missing, are there reasons/cases where you'd want
> your newSearcher warming queries to be different than your firstSearcher
> ones?
> 
> If at least I'm write that _commonly_ you want them to be the same...
> then it seems like it would be nice if there were a way to specify
> warming queries that would be run for _both_ newSearcher and
> firstSearcher, rather than have to duplicate the warming queries in both
> places.  The duplication means every time I change a warming query, I
> need to remember to change it in both places. Is there a way to avoid
> this duplication that I'm missing?
> 
> Jonathan