Searchers in single/multi-core environments

[Mark Ferguson <ma...@gmail.com>]

Hello,

My apologies if this topic has already been discussed but I haven't been
able to find a lot of information in the wiki or mailing lists.

I am looking for more information about how searchers work in different
environments. Correct me if I'm mistaken, but my understanding is that in a
single core environment, there is one searcher for the one index which
handles all queries. When a commit occurs, a new searcher is opened up on
the index during the commit. The old searcher is still available until the
commit finishes, at which point the active searcher becomes the new one and
the old searcher is destroyed. This is the purpose of the
maxWarmingSearchers argument -- it is the total number of searchers that can
be open in memory at any given point. What I'm not sure about is how this
number could ever be greater than 2 in a single core environment -- unless
another commit is sent before the new searcher finishes warming?

What I'm also curious about is how searchers are handled in a multi-core
environment. Does the maxWarmSearchers argument apply to the entire set of
cores, or to each individual core? If the latter, how is this handled if
each core uses a different solrconfig.xml and has a different value for
maxWarmSearchers?

Thanks for any information that you can provide.

Mark

Re: Searchers in single/multi-core environments

[Shalin Shekhar Mangar <sh...@gmail.com>]

On Sat, Feb 7, 2009 at 3:28 AM, Shalin Shekhar Mangar <
shalinmangar@gmail.com> wrote:

> If waitSearcher=false then the commit thread blocks until these operations
> finish.
>

I meant if waitSearcher=true then the commit thread blocks until these
operations finish.
-- 
Regards,
Shalin Shekhar Mangar.

Re: Searchers in single/multi-core environments

[Shalin Shekhar Mangar <sh...@gmail.com>]

On Sat, Feb 7, 2009 at 3:44 AM, Mark Ferguson <ma...@gmail.com>wrote:

>
> For example then, if I have 10 cores, and maxWarmSearchers is 2 for each
> core, if I send a commit to all of them at once this will not cause any
> exceptions, because each core handles its searchers separately?


Correct. Though it is a good idea to have a small gap between commits on
each core so that you don't run into resource issues (depending on how
intensive is your postCommit/auto-warming/newSearcher)

-- 
Regards,
Shalin Shekhar Mangar.

Re: Searchers in single/multi-core environments

[Mark Ferguson <ma...@gmail.com>]

>
> > What I'm also curious about is how searchers are handled in a multi-core
> > environment. Does the maxWarmSearchers argument apply to the entire set
> of
> > cores, or to each individual core?
>
>
> It applied to one core unless ofcourse, you are sharing the solrconfig.xml
> with multiple cores. Also, if you call core reload, a new core is created
> (with its own searcher) which replaces the old core.
>
>
Thanks very much for your time and explanation, it is a huge help. Just to
clarify that I am understanding correctly...

For example then, if I have 10 cores, and maxWarmSearchers is 2 for each
core, if I send a commit to all of them at once this will not cause any
exceptions, because each core handles its searchers separately?

Mark

Re: Searchers in single/multi-core environments

[Shalin Shekhar Mangar <sh...@gmail.com>]

On Sat, Feb 7, 2009 at 2:51 AM, Mark Ferguson <ma...@gmail.com>wrote:

>
> I am looking for more information about how searchers work in different
> environments. Correct me if I'm mistaken, but my understanding is that in a
> single core environment, there is one searcher for the one index which
> handles all queries. When a commit occurs, a new searcher is opened up on
> the index during the commit. The old searcher is still available until the
> commit finishes, at which point the active searcher becomes the new one and
> the old searcher is destroyed.


After commit is called, the postCommit/postOptimize hooks are executed (in
the same thread which called the commit). Then, in a new thread, a new
searcher is opened, auto-warming is performed, newSearcher event listeners
are executed and the new searcher is registered (i.e. it replaces the old
searcher). If useColdSearcher is true then the auto-warming is skipped. If
waitSearcher=false then the commit thread blocks until these operations
finish.

The old searcher is available until all it finishes all the requests it had
already received until the new searcher got registered.


> This is the purpose of the maxWarmingSearchers argument -- it is the total
> number of searchers that can
> be open in memory at any given point.


Not the total number of searchers but the total number of on-deck (warming)
searchers. So, total number of searchers will be maxWarmingSearchers + 1
(for the current registered searcher)


> What I'm not sure about is how this
> number could ever be greater than 2 in a single core environment -- unless
> another commit is sent before the new searcher finishes warming?


Correct. If warming or the newSearcher event listener takes a lot of time
and you call commit again, another searcher will be created.


> What I'm also curious about is how searchers are handled in a multi-core
> environment. Does the maxWarmSearchers argument apply to the entire set of
> cores, or to each individual core?


It applied to one core unless ofcourse, you are sharing the solrconfig.xml
with multiple cores. Also, if you call core reload, a new core is created
(with its own searcher) which replaces the old core.


> If the latter, how is this handled if
> each core uses a different solrconfig.xml and has a different value for
> maxWarmSearchers?
>

Each core maintains its configuration separately in memory. Configuration is
not shared between cores (except for the configuration in solr.xml)

Hope that helps.

-- 
Regards,
Shalin Shekhar Mangar.