You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by ade-b <ad...@gmail.com> on 2013/11/22 10:44:28 UTC
useColdSearcher in SolrCloud config
Hi
The definition of useColdSearcher config element in solrconfig.xml is
"If a search request comes in and there is no current registered searcher,
then immediately register the still warming searcher and use it. If "false"
then all requests will block until the first searcher is done warming".
By the term 'block', I assume SOLR returns a non 200 response to requests.
Does anybody know the exact response code returned when the server is
blocking requests?
If a new SOLR server is introduced into an existing array of SOLR servers
(in SOLR Cloud setup), it will sync it's index from the leader. To save you
having to specify warm-up queries in the solrconfig.xml file for first
searchers, would/could the new server not auto warm it's caches from the
caches of an existing server?
Thanks
Ade
--
View this message in context: http://lucene.472066.n3.nabble.com/useColdSearcher-in-SolrCloud-config-tp4102569.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: useColdSearcher in SolrCloud config
Posted by Bill Bell <bi...@gmail.com>.
Wouldn't that be true means use cold searcher? It seems backwards to me...
Sent from my iPad
> On Nov 22, 2013, at 2:44 AM, ade-b <ad...@gmail.com> wrote:
>
> Hi
>
> The definition of useColdSearcher config element in solrconfig.xml is
>
> "If a search request comes in and there is no current registered searcher,
> then immediately register the still warming searcher and use it. If "false"
> then all requests will block until the first searcher is done warming".
>
> By the term 'block', I assume SOLR returns a non 200 response to requests.
> Does anybody know the exact response code returned when the server is
> blocking requests?
>
> If a new SOLR server is introduced into an existing array of SOLR servers
> (in SOLR Cloud setup), it will sync it's index from the leader. To save you
> having to specify warm-up queries in the solrconfig.xml file for first
> searchers, would/could the new server not auto warm it's caches from the
> caches of an existing server?
>
> Thanks
> Ade
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/useColdSearcher-in-SolrCloud-config-tp4102569.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: useColdSearcher in SolrCloud config
Posted by Erick Erickson <er...@gmail.com>.
bq: For example, what if the leader could give a list of
queries/filters currently in the cache which could then be executed on
the replica?
How is the better than each replica firing off its own warming
queries for its caches etc? Each replica may well fire different
autowarm queries since there's no guarantee that their first
autowarmcount queries in the caches is the same, but that would
also be true of getting the autowarm queries from the leader.
Any firstSearcher and newSearcher queries will be identical
anyway since solrconfig.xml is identical, so that's not a problem.
So I remain to be convinced that this would buy us anything, but
I've been wrong more than once :)....
Best,
Erick
On Sat, Nov 23, 2013 at 1:31 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 11/23/2013 4:20 AM, Shalin Shekhar Mangar wrote:
> > As you said, loading caches from the caches of another server is not
> > feasible but there is some merit in warming with the queries of the
> > leader. For example, what if the leader could give a list of
> > queries/filters currently in the cache which could then be executed on
> > the replica? That'd be useful I think.
>
> That is an interesting idea.
>
> Thoughts, conjured with only surface understanding of the internals
> involved:
>
> One question is whether this would be done via the zookeeper queue or
> with direct inter-server communication. My only worry with doing it in
> zookeeper is the potential for it to put a major load on low-end
> zookeeper machines. We often tell people that their ZK nodes do not
> need much in the way of resources. If the user has high-end machines,
> even if they are doing double duty as SolrCloud and Zookeeper, that
> would not really be a worry.
>
> Would we want to control the max number of forwarded keys via the
> existing autowarmCount setting, or have a new per-cache setting with a
> relatively low default? If it's a new setting, I would recommend that
> it not be included in the example solrconfig.xml file, to discourage
> people from shooting themselves in the foot accidentally. It should be
> well documented in the wiki and ref guide as an expert setting.
>
> Thanks,
> Shawn
>
>
Re: useColdSearcher in SolrCloud config
Posted by Shawn Heisey <so...@elyograg.org>.
On 11/23/2013 4:20 AM, Shalin Shekhar Mangar wrote:
> As you said, loading caches from the caches of another server is not
> feasible but there is some merit in warming with the queries of the
> leader. For example, what if the leader could give a list of
> queries/filters currently in the cache which could then be executed on
> the replica? That'd be useful I think.
That is an interesting idea.
Thoughts, conjured with only surface understanding of the internals
involved:
One question is whether this would be done via the zookeeper queue or
with direct inter-server communication. My only worry with doing it in
zookeeper is the potential for it to put a major load on low-end
zookeeper machines. We often tell people that their ZK nodes do not
need much in the way of resources. If the user has high-end machines,
even if they are doing double duty as SolrCloud and Zookeeper, that
would not really be a worry.
Would we want to control the max number of forwarded keys via the
existing autowarmCount setting, or have a new per-cache setting with a
relatively low default? If it's a new setting, I would recommend that
it not be included in the example solrconfig.xml file, to discourage
people from shooting themselves in the foot accidentally. It should be
well documented in the wiki and ref guide as an expert setting.
Thanks,
Shawn
Re: useColdSearcher in SolrCloud config
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Fri, Nov 22, 2013 at 6:57 PM, Erick Erickson <er...@gmail.com> wrote:
> bq: If a new SOLR server....
>
> No. Apart from any ugly details about caches and internal
> doc IDs, you'd have to pull the caches over the wire to the
> new machine, and the caches could well be gigabytes in size.
> This is almost certainly much slower than just firing
> the warming queries locally. Seems like far too complex
> a functionality to put in place to save the effort of specifying a
> warmup query. Which you have to do anyway since you have to
> be ready to restart your cluster.
As you said, loading caches from the caches of another server is not
feasible but there is some merit in warming with the queries of the
leader. For example, what if the leader could give a list of
queries/filters currently in the cache which could then be executed on
the replica? That'd be useful I think.
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/useColdSearcher-in-SolrCloud-config-tp4102569.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
--
Regards,
Shalin Shekhar Mangar.
Re: useColdSearcher in SolrCloud config
Posted by Erick Erickson <er...@gmail.com>.
bq: By the term 'block', I assume SOLR returns a non 200
Pretty sure not. The query just waits around in the queue
in the server until the searcher is done warming, then the
search is executed and results returned.
bq: If a new SOLR server....
No. Apart from any ugly details about caches and internal
doc IDs, you'd have to pull the caches over the wire to the
new machine, and the caches could well be gigabytes in size.
This is almost certainly much slower than just firing
the warming queries locally. Seems like far too complex
a functionality to put in place to save the effort of specifying a
warmup query. Which you have to do anyway since you have to
be ready to restart your cluster.
Best,
Erick
On Fri, Nov 22, 2013 at 4:44 AM, ade-b <ad...@gmail.com> wrote:
> Hi
>
> The definition of useColdSearcher config element in solrconfig.xml is
>
> "If a search request comes in and there is no current registered searcher,
> then immediately register the still warming searcher and use it. If
> "false"
> then all requests will block until the first searcher is done warming".
>
> By the term 'block', I assume SOLR returns a non 200 response to requests.
> Does anybody know the exact response code returned when the server is
> blocking requests?
>
> If a new SOLR server is introduced into an existing array of SOLR servers
> (in SOLR Cloud setup), it will sync it's index from the leader. To save you
> having to specify warm-up queries in the solrconfig.xml file for first
> searchers, would/could the new server not auto warm it's caches from the
> caches of an existing server?
>
> Thanks
> Ade
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/useColdSearcher-in-SolrCloud-config-tp4102569.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>