Warming queries and Solr Cloud - just curious ...

[Timothy Potter <th...@gmail.com>]

When running in SolrCloud mode, does it make sense to disable distributed
mode for warming queries? i.e. distrib=false in my warming query config

I actually asked this on Erik's informative Webinar this morning but had to
drop off before I heard the answer ... so Erik might have answered this
already ;-)

My thinking here is that a hard commit gets sent around the cluster
automatically. Say I have 36 nodes (18 leaders and 18 replicas), on hard
commit, all 36 nodes will be warming up. If my warming queries are
distributed, then all nodes are going to be sending the same query
needlessly around the cluster 36 times - seems unnecessary.

Thoughts?

Cheers,
Tim

Re: Warming queries and Solr Cloud - just curious ...

[Joel Bernstein <jo...@gmail.com>]

That was a good fix Mark. I had this picture in my head of a large Solr
Cloud sending around thousands of simultaneous searches and crashing itself.


On Wed, Mar 27, 2013 at 6:47 PM, Timothy Potter <th...@gmail.com>wrote:

> lol - you know you're a bad ass when you've forgotten more about Solr cloud
> than the rest of us know ;-)
>
> On Wed, Mar 27, 2013 at 4:41 PM, Mark Miller <ma...@gmail.com>
> wrote:
>
> > Ah, interesting. Forgot about doing that issue entirely.
> >
> > - Mark
> >
> > On Mar 27, 2013, at 6:25 PM, Joel Bernstein <jo...@gmail.com> wrote:
> >
> > > I ran a quick test and distrib=false is being tacked on automatically.
> > Here
> > > is the log record:
> > >
> > > INFO: [collection1] webapp=null path=null
> > > params={sort=price+asc&event=newSearcher&q=solr&distrib=false} hits=1
> > > status=0 QTime=17
> > >
> > > So I think this is OK.
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Mar 27, 2013 at 6:02 PM, Joel Bernstein <jo...@gmail.com>
> > wrote:
> > >
> > >> This jira looks like it addresses this.
> > >>
> > >> https://issues.apache.org/jira/browse/SOLR-3081
> > >>
> > >> I'll run a quick test.
> > >>
> > >>
> > >> On Wed, Mar 27, 2013 at 5:41 PM, Timothy Potter <thelabdude@gmail.com
> > >wrote:
> > >>
> > >>> In our case, yes - same non-distrib query is warmed on each node.
> Seems
> > >>> like you'd need something a little more dynamic than statically
> > configured
> > >>> warming queries in solrconfig.xml for targeting specfic shards.
> > >>>
> > >>> Tim
> > >>>
> > >>> On Wed, Mar 27, 2013 at 2:04 PM, santoash <sa...@me.com> wrote:
> > >>>
> > >>>> This is interesting. I'm looking into doing something similar too.
> > >>>>
> > >>>> Quick question: Would you be targeting each of the shard with
> exactly
> > >>> the
> > >>>> same set of queries?
> > >>>>
> > >>>>
> > >>>> On Mar 27, 2013, at 12:59 PM, Mark Miller <ma...@gmail.com>
> > >>> wrote:
> > >>>>
> > >>>>> Yup. You only want to warm locally. We should add that to the wiki.
> > >>>>>
> > >>>>> - Mark
> > >>>>>
> > >>>>> On Mar 27, 2013, at 3:54 PM, Timothy Potter <th...@gmail.com>
> > >>>> wrote:
> > >>>>>
> > >>>>>> When running in SolrCloud mode, does it make sense to disable
> > >>>> distributed
> > >>>>>> mode for warming queries? i.e. distrib=false in my warming query
> > >>> config
> > >>>>>>
> > >>>>>> I actually asked this on Erik's informative Webinar this morning
> but
> > >>>> had to
> > >>>>>> drop off before I heard the answer ... so Erik might have answered
> > >>> this
> > >>>>>> already ;-)
> > >>>>>>
> > >>>>>> My thinking here is that a hard commit gets sent around the
> cluster
> > >>>>>> automatically. Say I have 36 nodes (18 leaders and 18 replicas),
> on
> > >>> hard
> > >>>>>> commit, all 36 nodes will be warming up. If my warming queries are
> > >>>>>> distributed, then all nodes are going to be sending the same query
> > >>>>>> needlessly around the cluster 36 times - seems unnecessary.
> > >>>>>>
> > >>>>>> Thoughts?
> > >>>>>>
> > >>>>>> Cheers,
> > >>>>>> Tim
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Joel Bernstein
> > >> Professional Services LucidWorks
> > >>
> > >
> > >
> > >
> > > --
> > > Joel Bernstein
> > > Professional Services LucidWorks
> >
> >
>



-- 
Joel Bernstein
Professional Services LucidWorks

Re: Warming queries and Solr Cloud - just curious ...

[Timothy Potter <th...@gmail.com>]

lol - you know you're a bad ass when you've forgotten more about Solr cloud
than the rest of us know ;-)

On Wed, Mar 27, 2013 at 4:41 PM, Mark Miller <ma...@gmail.com> wrote:

> Ah, interesting. Forgot about doing that issue entirely.
>
> - Mark
>
> On Mar 27, 2013, at 6:25 PM, Joel Bernstein <jo...@gmail.com> wrote:
>
> > I ran a quick test and distrib=false is being tacked on automatically.
> Here
> > is the log record:
> >
> > INFO: [collection1] webapp=null path=null
> > params={sort=price+asc&event=newSearcher&q=solr&distrib=false} hits=1
> > status=0 QTime=17
> >
> > So I think this is OK.
> >
> >
> >
> >
> >
> > On Wed, Mar 27, 2013 at 6:02 PM, Joel Bernstein <jo...@gmail.com>
> wrote:
> >
> >> This jira looks like it addresses this.
> >>
> >> https://issues.apache.org/jira/browse/SOLR-3081
> >>
> >> I'll run a quick test.
> >>
> >>
> >> On Wed, Mar 27, 2013 at 5:41 PM, Timothy Potter <thelabdude@gmail.com
> >wrote:
> >>
> >>> In our case, yes - same non-distrib query is warmed on each node. Seems
> >>> like you'd need something a little more dynamic than statically
> configured
> >>> warming queries in solrconfig.xml for targeting specfic shards.
> >>>
> >>> Tim
> >>>
> >>> On Wed, Mar 27, 2013 at 2:04 PM, santoash <sa...@me.com> wrote:
> >>>
> >>>> This is interesting. I'm looking into doing something similar too.
> >>>>
> >>>> Quick question: Would you be targeting each of the shard with exactly
> >>> the
> >>>> same set of queries?
> >>>>
> >>>>
> >>>> On Mar 27, 2013, at 12:59 PM, Mark Miller <ma...@gmail.com>
> >>> wrote:
> >>>>
> >>>>> Yup. You only want to warm locally. We should add that to the wiki.
> >>>>>
> >>>>> - Mark
> >>>>>
> >>>>> On Mar 27, 2013, at 3:54 PM, Timothy Potter <th...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>>> When running in SolrCloud mode, does it make sense to disable
> >>>> distributed
> >>>>>> mode for warming queries? i.e. distrib=false in my warming query
> >>> config
> >>>>>>
> >>>>>> I actually asked this on Erik's informative Webinar this morning but
> >>>> had to
> >>>>>> drop off before I heard the answer ... so Erik might have answered
> >>> this
> >>>>>> already ;-)
> >>>>>>
> >>>>>> My thinking here is that a hard commit gets sent around the cluster
> >>>>>> automatically. Say I have 36 nodes (18 leaders and 18 replicas), on
> >>> hard
> >>>>>> commit, all 36 nodes will be warming up. If my warming queries are
> >>>>>> distributed, then all nodes are going to be sending the same query
> >>>>>> needlessly around the cluster 36 times - seems unnecessary.
> >>>>>>
> >>>>>> Thoughts?
> >>>>>>
> >>>>>> Cheers,
> >>>>>> Tim
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >> --
> >> Joel Bernstein
> >> Professional Services LucidWorks
> >>
> >
> >
> >
> > --
> > Joel Bernstein
> > Professional Services LucidWorks
>
>

Re: Warming queries and Solr Cloud - just curious ...

[Mark Miller <ma...@gmail.com>]

Ah, interesting. Forgot about doing that issue entirely.

- Mark

On Mar 27, 2013, at 6:25 PM, Joel Bernstein <jo...@gmail.com> wrote:

> I ran a quick test and distrib=false is being tacked on automatically. Here
> is the log record:
> 
> INFO: [collection1] webapp=null path=null
> params={sort=price+asc&event=newSearcher&q=solr&distrib=false} hits=1
> status=0 QTime=17
> 
> So I think this is OK.
> 
> 
> 
> 
> 
> On Wed, Mar 27, 2013 at 6:02 PM, Joel Bernstein <jo...@gmail.com> wrote:
> 
>> This jira looks like it addresses this.
>> 
>> https://issues.apache.org/jira/browse/SOLR-3081
>> 
>> I'll run a quick test.
>> 
>> 
>> On Wed, Mar 27, 2013 at 5:41 PM, Timothy Potter <th...@gmail.com>wrote:
>> 
>>> In our case, yes - same non-distrib query is warmed on each node. Seems
>>> like you'd need something a little more dynamic than statically configured
>>> warming queries in solrconfig.xml for targeting specfic shards.
>>> 
>>> Tim
>>> 
>>> On Wed, Mar 27, 2013 at 2:04 PM, santoash <sa...@me.com> wrote:
>>> 
>>>> This is interesting. I'm looking into doing something similar too.
>>>> 
>>>> Quick question: Would you be targeting each of the shard with exactly
>>> the
>>>> same set of queries?
>>>> 
>>>> 
>>>> On Mar 27, 2013, at 12:59 PM, Mark Miller <ma...@gmail.com>
>>> wrote:
>>>> 
>>>>> Yup. You only want to warm locally. We should add that to the wiki.
>>>>> 
>>>>> - Mark
>>>>> 
>>>>> On Mar 27, 2013, at 3:54 PM, Timothy Potter <th...@gmail.com>
>>>> wrote:
>>>>> 
>>>>>> When running in SolrCloud mode, does it make sense to disable
>>>> distributed
>>>>>> mode for warming queries? i.e. distrib=false in my warming query
>>> config
>>>>>> 
>>>>>> I actually asked this on Erik's informative Webinar this morning but
>>>> had to
>>>>>> drop off before I heard the answer ... so Erik might have answered
>>> this
>>>>>> already ;-)
>>>>>> 
>>>>>> My thinking here is that a hard commit gets sent around the cluster
>>>>>> automatically. Say I have 36 nodes (18 leaders and 18 replicas), on
>>> hard
>>>>>> commit, all 36 nodes will be warming up. If my warming queries are
>>>>>> distributed, then all nodes are going to be sending the same query
>>>>>> needlessly around the cluster 36 times - seems unnecessary.
>>>>>> 
>>>>>> Thoughts?
>>>>>> 
>>>>>> Cheers,
>>>>>> Tim
>>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 
>> 
>> --
>> Joel Bernstein
>> Professional Services LucidWorks
>> 
> 
> 
> 
> -- 
> Joel Bernstein
> Professional Services LucidWorks

Re: Warming queries and Solr Cloud - just curious ...

[Joel Bernstein <jo...@gmail.com>]

I ran a quick test and distrib=false is being tacked on automatically. Here
is the log record:

INFO: [collection1] webapp=null path=null
params={sort=price+asc&event=newSearcher&q=solr&distrib=false} hits=1
status=0 QTime=17

So I think this is OK.





On Wed, Mar 27, 2013 at 6:02 PM, Joel Bernstein <jo...@gmail.com> wrote:

> This jira looks like it addresses this.
>
> https://issues.apache.org/jira/browse/SOLR-3081
>
> I'll run a quick test.
>
>
> On Wed, Mar 27, 2013 at 5:41 PM, Timothy Potter <th...@gmail.com>wrote:
>
>> In our case, yes - same non-distrib query is warmed on each node. Seems
>> like you'd need something a little more dynamic than statically configured
>> warming queries in solrconfig.xml for targeting specfic shards.
>>
>> Tim
>>
>> On Wed, Mar 27, 2013 at 2:04 PM, santoash <sa...@me.com> wrote:
>>
>> > This is interesting. I'm looking into doing something similar too.
>> >
>> > Quick question: Would you be targeting each of the shard with exactly
>> the
>> > same set of queries?
>> >
>> >
>> > On Mar 27, 2013, at 12:59 PM, Mark Miller <ma...@gmail.com>
>> wrote:
>> >
>> > > Yup. You only want to warm locally. We should add that to the wiki.
>> > >
>> > > - Mark
>> > >
>> > > On Mar 27, 2013, at 3:54 PM, Timothy Potter <th...@gmail.com>
>> > wrote:
>> > >
>> > >> When running in SolrCloud mode, does it make sense to disable
>> > distributed
>> > >> mode for warming queries? i.e. distrib=false in my warming query
>> config
>> > >>
>> > >> I actually asked this on Erik's informative Webinar this morning but
>> > had to
>> > >> drop off before I heard the answer ... so Erik might have answered
>> this
>> > >> already ;-)
>> > >>
>> > >> My thinking here is that a hard commit gets sent around the cluster
>> > >> automatically. Say I have 36 nodes (18 leaders and 18 replicas), on
>> hard
>> > >> commit, all 36 nodes will be warming up. If my warming queries are
>> > >> distributed, then all nodes are going to be sending the same query
>> > >> needlessly around the cluster 36 times - seems unnecessary.
>> > >>
>> > >> Thoughts?
>> > >>
>> > >> Cheers,
>> > >> Tim
>> > >
>> >
>> >
>>
>
>
>
> --
> Joel Bernstein
> Professional Services LucidWorks
>



-- 
Joel Bernstein
Professional Services LucidWorks

Re: Warming queries and Solr Cloud - just curious ...

[Joel Bernstein <jo...@gmail.com>]

This jira looks like it addresses this.

https://issues.apache.org/jira/browse/SOLR-3081

I'll run a quick test.


On Wed, Mar 27, 2013 at 5:41 PM, Timothy Potter <th...@gmail.com>wrote:

> In our case, yes - same non-distrib query is warmed on each node. Seems
> like you'd need something a little more dynamic than statically configured
> warming queries in solrconfig.xml for targeting specfic shards.
>
> Tim
>
> On Wed, Mar 27, 2013 at 2:04 PM, santoash <sa...@me.com> wrote:
>
> > This is interesting. I'm looking into doing something similar too.
> >
> > Quick question: Would you be targeting each of the shard with exactly the
> > same set of queries?
> >
> >
> > On Mar 27, 2013, at 12:59 PM, Mark Miller <ma...@gmail.com> wrote:
> >
> > > Yup. You only want to warm locally. We should add that to the wiki.
> > >
> > > - Mark
> > >
> > > On Mar 27, 2013, at 3:54 PM, Timothy Potter <th...@gmail.com>
> > wrote:
> > >
> > >> When running in SolrCloud mode, does it make sense to disable
> > distributed
> > >> mode for warming queries? i.e. distrib=false in my warming query
> config
> > >>
> > >> I actually asked this on Erik's informative Webinar this morning but
> > had to
> > >> drop off before I heard the answer ... so Erik might have answered
> this
> > >> already ;-)
> > >>
> > >> My thinking here is that a hard commit gets sent around the cluster
> > >> automatically. Say I have 36 nodes (18 leaders and 18 replicas), on
> hard
> > >> commit, all 36 nodes will be warming up. If my warming queries are
> > >> distributed, then all nodes are going to be sending the same query
> > >> needlessly around the cluster 36 times - seems unnecessary.
> > >>
> > >> Thoughts?
> > >>
> > >> Cheers,
> > >> Tim
> > >
> >
> >
>



-- 
Joel Bernstein
Professional Services LucidWorks

Re: Warming queries and Solr Cloud - just curious ...

[Timothy Potter <th...@gmail.com>]

In our case, yes - same non-distrib query is warmed on each node. Seems
like you'd need something a little more dynamic than statically configured
warming queries in solrconfig.xml for targeting specfic shards.

Tim

On Wed, Mar 27, 2013 at 2:04 PM, santoash <sa...@me.com> wrote:

> This is interesting. I'm looking into doing something similar too.
>
> Quick question: Would you be targeting each of the shard with exactly the
> same set of queries?
>
>
> On Mar 27, 2013, at 12:59 PM, Mark Miller <ma...@gmail.com> wrote:
>
> > Yup. You only want to warm locally. We should add that to the wiki.
> >
> > - Mark
> >
> > On Mar 27, 2013, at 3:54 PM, Timothy Potter <th...@gmail.com>
> wrote:
> >
> >> When running in SolrCloud mode, does it make sense to disable
> distributed
> >> mode for warming queries? i.e. distrib=false in my warming query config
> >>
> >> I actually asked this on Erik's informative Webinar this morning but
> had to
> >> drop off before I heard the answer ... so Erik might have answered this
> >> already ;-)
> >>
> >> My thinking here is that a hard commit gets sent around the cluster
> >> automatically. Say I have 36 nodes (18 leaders and 18 replicas), on hard
> >> commit, all 36 nodes will be warming up. If my warming queries are
> >> distributed, then all nodes are going to be sending the same query
> >> needlessly around the cluster 36 times - seems unnecessary.
> >>
> >> Thoughts?
> >>
> >> Cheers,
> >> Tim
> >
>
>

Re: Warming queries and Solr Cloud - just curious ...

[santoash <sa...@me.com>]

This is interesting. I'm looking into doing something similar too. 

Quick question: Would you be targeting each of the shard with exactly the same set of queries? 


On Mar 27, 2013, at 12:59 PM, Mark Miller <ma...@gmail.com> wrote:

> Yup. You only want to warm locally. We should add that to the wiki.
> 
> - Mark
> 
> On Mar 27, 2013, at 3:54 PM, Timothy Potter <th...@gmail.com> wrote:
> 
>> When running in SolrCloud mode, does it make sense to disable distributed
>> mode for warming queries? i.e. distrib=false in my warming query config
>> 
>> I actually asked this on Erik's informative Webinar this morning but had to
>> drop off before I heard the answer ... so Erik might have answered this
>> already ;-)
>> 
>> My thinking here is that a hard commit gets sent around the cluster
>> automatically. Say I have 36 nodes (18 leaders and 18 replicas), on hard
>> commit, all 36 nodes will be warming up. If my warming queries are
>> distributed, then all nodes are going to be sending the same query
>> needlessly around the cluster 36 times - seems unnecessary.
>> 
>> Thoughts?
>> 
>> Cheers,
>> Tim
> 

Re: Warming queries and Solr Cloud - just curious ...

[Erick Erickson <er...@gmail.com>]

Tim:

Unfortunately, due to the increase in spam pages from bots, we had to
lock down the Solr wiki. Post a request for us to add your Wiki ID
(and give us the ID!) to the list of authorized IDs and we'll get you
added (just takes a second). Or send me (or Steve Rowe) a private
e-mail if you'd prefer.

Best
Erick

On Wed, Mar 27, 2013 at 4:03 PM, Timothy Potter <th...@gmail.com> wrote:
> Ok - thanks for confirming Mark - I'll add that to the wiki.
>
> Cheers,
> Tim
>
> On Wed, Mar 27, 2013 at 1:59 PM, Mark Miller <ma...@gmail.com> wrote:
>
>> Yup. You only want to warm locally. We should add that to the wiki.
>>
>> - Mark
>>
>> On Mar 27, 2013, at 3:54 PM, Timothy Potter <th...@gmail.com> wrote:
>>
>> > When running in SolrCloud mode, does it make sense to disable distributed
>> > mode for warming queries? i.e. distrib=false in my warming query config
>> >
>> > I actually asked this on Erik's informative Webinar this morning but had
>> to
>> > drop off before I heard the answer ... so Erik might have answered this
>> > already ;-)
>> >
>> > My thinking here is that a hard commit gets sent around the cluster
>> > automatically. Say I have 36 nodes (18 leaders and 18 replicas), on hard
>> > commit, all 36 nodes will be warming up. If my warming queries are
>> > distributed, then all nodes are going to be sending the same query
>> > needlessly around the cluster 36 times - seems unnecessary.
>> >
>> > Thoughts?
>> >
>> > Cheers,
>> > Tim
>>
>>

Re: Warming queries and Solr Cloud - just curious ...

[Timothy Potter <th...@gmail.com>]

Ok - thanks for confirming Mark - I'll add that to the wiki.

Cheers,
Tim

On Wed, Mar 27, 2013 at 1:59 PM, Mark Miller <ma...@gmail.com> wrote:

> Yup. You only want to warm locally. We should add that to the wiki.
>
> - Mark
>
> On Mar 27, 2013, at 3:54 PM, Timothy Potter <th...@gmail.com> wrote:
>
> > When running in SolrCloud mode, does it make sense to disable distributed
> > mode for warming queries? i.e. distrib=false in my warming query config
> >
> > I actually asked this on Erik's informative Webinar this morning but had
> to
> > drop off before I heard the answer ... so Erik might have answered this
> > already ;-)
> >
> > My thinking here is that a hard commit gets sent around the cluster
> > automatically. Say I have 36 nodes (18 leaders and 18 replicas), on hard
> > commit, all 36 nodes will be warming up. If my warming queries are
> > distributed, then all nodes are going to be sending the same query
> > needlessly around the cluster 36 times - seems unnecessary.
> >
> > Thoughts?
> >
> > Cheers,
> > Tim
>
>

Re: Warming queries and Solr Cloud - just curious ...

[Mark Miller <ma...@gmail.com>]

Yup. You only want to warm locally. We should add that to the wiki.

- Mark

On Mar 27, 2013, at 3:54 PM, Timothy Potter <th...@gmail.com> wrote:

> When running in SolrCloud mode, does it make sense to disable distributed
> mode for warming queries? i.e. distrib=false in my warming query config
> 
> I actually asked this on Erik's informative Webinar this morning but had to
> drop off before I heard the answer ... so Erik might have answered this
> already ;-)
> 
> My thinking here is that a hard commit gets sent around the cluster
> automatically. Say I have 36 nodes (18 leaders and 18 replicas), on hard
> commit, all 36 nodes will be warming up. If my warming queries are
> distributed, then all nodes are going to be sending the same query
> needlessly around the cluster 36 times - seems unnecessary.
> 
> Thoughts?
> 
> Cheers,
> Tim