You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bill Au <bi...@gmail.com> on 2012/10/25 21:29:59 UTC
SolrCloud and distributed search
Is SolrCloud using distributed search behind the scene? Does it have the
same limitations (for example, doesn't support MoreLikeThis) distributed
search has?
Bill
Re: SolrCloud and distributed search
Posted by Yonik Seeley <yo...@lucidworks.com>.
On Fri, Oct 26, 2012 at 10:14 AM, Bill Au <bi...@gmail.com> wrote:
> I am currently using one master with multiple slaves so I do have high
> availability for searching now.
>
> My index does fit on a single machine and a single query does not take too
> long to execute. But I do want to take advantage of high availability of
> indexing and real time replication. So it looks like I can set up
> SolrCloud with only 1 shard (ie numShards=1).
>
> In this case is SolrCloud still using distributed search behind the
> screen? Will MoreLikeThis work?
>
> Does using SolrCloud with only 1 shard make any sense at all?
Yep.
Just pass distrib=false when querying.
At some point we need to optimize the case when a single node will
satisfy the query - and not just in the single shard case.
For upcoming custom hashing, if the requested hash range lies
completely in the shard, we shouldn't go distributed.
-Yonik
http://lucidworks.com
Re: SolrCloud and distributed search
Posted by Erick Erickson <er...@gmail.com>.
you can certainly save one hop by sending the update request to
the shard leader (how are you sending things BTW, SolrJ?).
But at this point that hasn't been done, see:
https://issues.apache.org/jira/browse/SOLR-3154
It's certainly on the roadmap, but for now it works functionally
so there are other first to fry....
FWIW,
Erick
On Mon, Oct 29, 2012 at 6:12 PM, Bill Au <bi...@gmail.com> wrote:
> Do updates always start at the shard leader first? If so one can save one
> internal request by only sending updates to the shard leader. I am
> assuming that when the shard leader is down, SolrJ's CloudSolrServer is
> smart enough to use the newly elected shard leader after a failover has
> occurred. Am I correct?
>
> Bill
>
> On Fri, Oct 26, 2012 at 11:42 AM, Tomás Fernández Löbbe <
> tomasflobbe@gmail.com> wrote:
>
>> If you are going to use SolrJ, CloudSolrServer is even better than a
>> round-robin load balancer for indexing, because it will send the documents
>> straight to the shard leader (you save one internal request). If not,
>> round-robin should be fine.
>>
>> Tomás
>>
>> On Fri, Oct 26, 2012 at 12:27 PM, Bill Au <bi...@gmail.com> wrote:
>>
>> > I am thinking of using a load balancer for both indexing and querying to
>> > spread both the indexing and querying load across all the machines.
>> >
>> > Bill
>> >
>> > On Fri, Oct 26, 2012 at 10:48 AM, Tomás Fernández Löbbe <
>> > tomasflobbe@gmail.com> wrote:
>> >
>> > > You should still use some kind of load balancer for searches, unless
>> you
>> > > use the CloudSolrServer (SolrJ) which includes the load balancing.
>> > > Tomás
>> > >
>> > > On Fri, Oct 26, 2012 at 11:46 AM, Erick Erickson <
>> > erickerickson@gmail.com
>> > > >wrote:
>> > >
>> > > > Yes, I think SolrCloud makes sense with a single shard for exactly
>> > > > this reason, NRT and multiple replicas. I don't know how you'd get
>> NRT
>> > > > on multiple machines without it.
>> > > >
>> > > > But do be aware of: https://issues.apache.org/jira/browse/SOLR-3971
>> > > > "A collection that is created with numShards=1 turns into a
>> > > > numShards=2 collection after starting up a second core and not
>> > > > specifying numShards."
>> > > >
>> > > > Erick
>> > > >
>> > > > On Fri, Oct 26, 2012 at 10:14 AM, Bill Au <bi...@gmail.com>
>> wrote:
>> > > > > I am currently using one master with multiple slaves so I do have
>> > high
>> > > > > availability for searching now.
>> > > > >
>> > > > > My index does fit on a single machine and a single query does not
>> > take
>> > > > too
>> > > > > long to execute. But I do want to take advantage of high
>> > availability
>> > > of
>> > > > > indexing and real time replication. So it looks like I can set up
>> > > > > SolrCloud with only 1 shard (ie numShards=1).
>> > > > >
>> > > > > In this case is SolrCloud still using distributed search behind the
>> > > > > screen? Will MoreLikeThis work?
>> > > > >
>> > > > > Does using SolrCloud with only 1 shard make any sense at all?
>> > > > >
>> > > > > Bill
>> > > > >
>> > > > > On Thu, Oct 25, 2012 at 4:29 PM, Tomás Fernández Löbbe <
>> > > > > tomasflobbe@gmail.com> wrote:
>> > > > >
>> > > > >> It also provides high availability for indexing and searching.
>> > > > >>
>> > > > >> On Thu, Oct 25, 2012 at 4:43 PM, Bill Au <bi...@gmail.com>
>> > wrote:
>> > > > >>
>> > > > >> > So I guess one would use SolrCloud for the same reasons as
>> > > distributed
>> > > > >> > search:
>> > > > >> >
>> > > > >> > When an index becomes too large to fit on a single system, or
>> > when a
>> > > > >> single
>> > > > >> > query takes too long to execute.
>> > > > >> >
>> > > > >> > Bill
>> > > > >> >
>> > > > >> > On Thu, Oct 25, 2012 at 3:38 PM, Shawn Heisey <
>> solr@elyograg.org>
>> > > > wrote:
>> > > > >> >
>> > > > >> > > On 10/25/2012 1:29 PM, Bill Au wrote:
>> > > > >> > >
>> > > > >> > >> Is SolrCloud using distributed search behind the scene? Does
>> > it
>> > > > have
>> > > > >> > the
>> > > > >> > >> same limitations (for example, doesn't support MoreLikeThis)
>> > > > >> distributed
>> > > > >> > >> search has?
>> > > > >> > >>
>> > > > >> > >
>> > > > >> > > Yes and yes.
>> > > > >> > >
>> > > > >> > >
>> > > > >> >
>> > > > >>
>> > > >
>> > >
>> >
>>
Re: SolrCloud and distributed search
Posted by Bill Au <bi...@gmail.com>.
Do updates always start at the shard leader first? If so one can save one
internal request by only sending updates to the shard leader. I am
assuming that when the shard leader is down, SolrJ's CloudSolrServer is
smart enough to use the newly elected shard leader after a failover has
occurred. Am I correct?
Bill
On Fri, Oct 26, 2012 at 11:42 AM, Tomás Fernández Löbbe <
tomasflobbe@gmail.com> wrote:
> If you are going to use SolrJ, CloudSolrServer is even better than a
> round-robin load balancer for indexing, because it will send the documents
> straight to the shard leader (you save one internal request). If not,
> round-robin should be fine.
>
> Tomás
>
> On Fri, Oct 26, 2012 at 12:27 PM, Bill Au <bi...@gmail.com> wrote:
>
> > I am thinking of using a load balancer for both indexing and querying to
> > spread both the indexing and querying load across all the machines.
> >
> > Bill
> >
> > On Fri, Oct 26, 2012 at 10:48 AM, Tomás Fernández Löbbe <
> > tomasflobbe@gmail.com> wrote:
> >
> > > You should still use some kind of load balancer for searches, unless
> you
> > > use the CloudSolrServer (SolrJ) which includes the load balancing.
> > > Tomás
> > >
> > > On Fri, Oct 26, 2012 at 11:46 AM, Erick Erickson <
> > erickerickson@gmail.com
> > > >wrote:
> > >
> > > > Yes, I think SolrCloud makes sense with a single shard for exactly
> > > > this reason, NRT and multiple replicas. I don't know how you'd get
> NRT
> > > > on multiple machines without it.
> > > >
> > > > But do be aware of: https://issues.apache.org/jira/browse/SOLR-3971
> > > > "A collection that is created with numShards=1 turns into a
> > > > numShards=2 collection after starting up a second core and not
> > > > specifying numShards."
> > > >
> > > > Erick
> > > >
> > > > On Fri, Oct 26, 2012 at 10:14 AM, Bill Au <bi...@gmail.com>
> wrote:
> > > > > I am currently using one master with multiple slaves so I do have
> > high
> > > > > availability for searching now.
> > > > >
> > > > > My index does fit on a single machine and a single query does not
> > take
> > > > too
> > > > > long to execute. But I do want to take advantage of high
> > availability
> > > of
> > > > > indexing and real time replication. So it looks like I can set up
> > > > > SolrCloud with only 1 shard (ie numShards=1).
> > > > >
> > > > > In this case is SolrCloud still using distributed search behind the
> > > > > screen? Will MoreLikeThis work?
> > > > >
> > > > > Does using SolrCloud with only 1 shard make any sense at all?
> > > > >
> > > > > Bill
> > > > >
> > > > > On Thu, Oct 25, 2012 at 4:29 PM, Tomás Fernández Löbbe <
> > > > > tomasflobbe@gmail.com> wrote:
> > > > >
> > > > >> It also provides high availability for indexing and searching.
> > > > >>
> > > > >> On Thu, Oct 25, 2012 at 4:43 PM, Bill Au <bi...@gmail.com>
> > wrote:
> > > > >>
> > > > >> > So I guess one would use SolrCloud for the same reasons as
> > > distributed
> > > > >> > search:
> > > > >> >
> > > > >> > When an index becomes too large to fit on a single system, or
> > when a
> > > > >> single
> > > > >> > query takes too long to execute.
> > > > >> >
> > > > >> > Bill
> > > > >> >
> > > > >> > On Thu, Oct 25, 2012 at 3:38 PM, Shawn Heisey <
> solr@elyograg.org>
> > > > wrote:
> > > > >> >
> > > > >> > > On 10/25/2012 1:29 PM, Bill Au wrote:
> > > > >> > >
> > > > >> > >> Is SolrCloud using distributed search behind the scene? Does
> > it
> > > > have
> > > > >> > the
> > > > >> > >> same limitations (for example, doesn't support MoreLikeThis)
> > > > >> distributed
> > > > >> > >> search has?
> > > > >> > >>
> > > > >> > >
> > > > >> > > Yes and yes.
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> >
>
Re: SolrCloud and distributed search
Posted by Tomás Fernández Löbbe <to...@gmail.com>.
If you are going to use SolrJ, CloudSolrServer is even better than a
round-robin load balancer for indexing, because it will send the documents
straight to the shard leader (you save one internal request). If not,
round-robin should be fine.
Tomás
On Fri, Oct 26, 2012 at 12:27 PM, Bill Au <bi...@gmail.com> wrote:
> I am thinking of using a load balancer for both indexing and querying to
> spread both the indexing and querying load across all the machines.
>
> Bill
>
> On Fri, Oct 26, 2012 at 10:48 AM, Tomás Fernández Löbbe <
> tomasflobbe@gmail.com> wrote:
>
> > You should still use some kind of load balancer for searches, unless you
> > use the CloudSolrServer (SolrJ) which includes the load balancing.
> > Tomás
> >
> > On Fri, Oct 26, 2012 at 11:46 AM, Erick Erickson <
> erickerickson@gmail.com
> > >wrote:
> >
> > > Yes, I think SolrCloud makes sense with a single shard for exactly
> > > this reason, NRT and multiple replicas. I don't know how you'd get NRT
> > > on multiple machines without it.
> > >
> > > But do be aware of: https://issues.apache.org/jira/browse/SOLR-3971
> > > "A collection that is created with numShards=1 turns into a
> > > numShards=2 collection after starting up a second core and not
> > > specifying numShards."
> > >
> > > Erick
> > >
> > > On Fri, Oct 26, 2012 at 10:14 AM, Bill Au <bi...@gmail.com> wrote:
> > > > I am currently using one master with multiple slaves so I do have
> high
> > > > availability for searching now.
> > > >
> > > > My index does fit on a single machine and a single query does not
> take
> > > too
> > > > long to execute. But I do want to take advantage of high
> availability
> > of
> > > > indexing and real time replication. So it looks like I can set up
> > > > SolrCloud with only 1 shard (ie numShards=1).
> > > >
> > > > In this case is SolrCloud still using distributed search behind the
> > > > screen? Will MoreLikeThis work?
> > > >
> > > > Does using SolrCloud with only 1 shard make any sense at all?
> > > >
> > > > Bill
> > > >
> > > > On Thu, Oct 25, 2012 at 4:29 PM, Tomás Fernández Löbbe <
> > > > tomasflobbe@gmail.com> wrote:
> > > >
> > > >> It also provides high availability for indexing and searching.
> > > >>
> > > >> On Thu, Oct 25, 2012 at 4:43 PM, Bill Au <bi...@gmail.com>
> wrote:
> > > >>
> > > >> > So I guess one would use SolrCloud for the same reasons as
> > distributed
> > > >> > search:
> > > >> >
> > > >> > When an index becomes too large to fit on a single system, or
> when a
> > > >> single
> > > >> > query takes too long to execute.
> > > >> >
> > > >> > Bill
> > > >> >
> > > >> > On Thu, Oct 25, 2012 at 3:38 PM, Shawn Heisey <so...@elyograg.org>
> > > wrote:
> > > >> >
> > > >> > > On 10/25/2012 1:29 PM, Bill Au wrote:
> > > >> > >
> > > >> > >> Is SolrCloud using distributed search behind the scene? Does
> it
> > > have
> > > >> > the
> > > >> > >> same limitations (for example, doesn't support MoreLikeThis)
> > > >> distributed
> > > >> > >> search has?
> > > >> > >>
> > > >> > >
> > > >> > > Yes and yes.
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
>
Re: SolrCloud and distributed search
Posted by Bill Au <bi...@gmail.com>.
I am thinking of using a load balancer for both indexing and querying to
spread both the indexing and querying load across all the machines.
Bill
On Fri, Oct 26, 2012 at 10:48 AM, Tomás Fernández Löbbe <
tomasflobbe@gmail.com> wrote:
> You should still use some kind of load balancer for searches, unless you
> use the CloudSolrServer (SolrJ) which includes the load balancing.
> Tomás
>
> On Fri, Oct 26, 2012 at 11:46 AM, Erick Erickson <erickerickson@gmail.com
> >wrote:
>
> > Yes, I think SolrCloud makes sense with a single shard for exactly
> > this reason, NRT and multiple replicas. I don't know how you'd get NRT
> > on multiple machines without it.
> >
> > But do be aware of: https://issues.apache.org/jira/browse/SOLR-3971
> > "A collection that is created with numShards=1 turns into a
> > numShards=2 collection after starting up a second core and not
> > specifying numShards."
> >
> > Erick
> >
> > On Fri, Oct 26, 2012 at 10:14 AM, Bill Au <bi...@gmail.com> wrote:
> > > I am currently using one master with multiple slaves so I do have high
> > > availability for searching now.
> > >
> > > My index does fit on a single machine and a single query does not take
> > too
> > > long to execute. But I do want to take advantage of high availability
> of
> > > indexing and real time replication. So it looks like I can set up
> > > SolrCloud with only 1 shard (ie numShards=1).
> > >
> > > In this case is SolrCloud still using distributed search behind the
> > > screen? Will MoreLikeThis work?
> > >
> > > Does using SolrCloud with only 1 shard make any sense at all?
> > >
> > > Bill
> > >
> > > On Thu, Oct 25, 2012 at 4:29 PM, Tomás Fernández Löbbe <
> > > tomasflobbe@gmail.com> wrote:
> > >
> > >> It also provides high availability for indexing and searching.
> > >>
> > >> On Thu, Oct 25, 2012 at 4:43 PM, Bill Au <bi...@gmail.com> wrote:
> > >>
> > >> > So I guess one would use SolrCloud for the same reasons as
> distributed
> > >> > search:
> > >> >
> > >> > When an index becomes too large to fit on a single system, or when a
> > >> single
> > >> > query takes too long to execute.
> > >> >
> > >> > Bill
> > >> >
> > >> > On Thu, Oct 25, 2012 at 3:38 PM, Shawn Heisey <so...@elyograg.org>
> > wrote:
> > >> >
> > >> > > On 10/25/2012 1:29 PM, Bill Au wrote:
> > >> > >
> > >> > >> Is SolrCloud using distributed search behind the scene? Does it
> > have
> > >> > the
> > >> > >> same limitations (for example, doesn't support MoreLikeThis)
> > >> distributed
> > >> > >> search has?
> > >> > >>
> > >> > >
> > >> > > Yes and yes.
> > >> > >
> > >> > >
> > >> >
> > >>
> >
>
Re: SolrCloud and distributed search
Posted by Tomás Fernández Löbbe <to...@gmail.com>.
You should still use some kind of load balancer for searches, unless you
use the CloudSolrServer (SolrJ) which includes the load balancing.
Tomás
On Fri, Oct 26, 2012 at 11:46 AM, Erick Erickson <er...@gmail.com>wrote:
> Yes, I think SolrCloud makes sense with a single shard for exactly
> this reason, NRT and multiple replicas. I don't know how you'd get NRT
> on multiple machines without it.
>
> But do be aware of: https://issues.apache.org/jira/browse/SOLR-3971
> "A collection that is created with numShards=1 turns into a
> numShards=2 collection after starting up a second core and not
> specifying numShards."
>
> Erick
>
> On Fri, Oct 26, 2012 at 10:14 AM, Bill Au <bi...@gmail.com> wrote:
> > I am currently using one master with multiple slaves so I do have high
> > availability for searching now.
> >
> > My index does fit on a single machine and a single query does not take
> too
> > long to execute. But I do want to take advantage of high availability of
> > indexing and real time replication. So it looks like I can set up
> > SolrCloud with only 1 shard (ie numShards=1).
> >
> > In this case is SolrCloud still using distributed search behind the
> > screen? Will MoreLikeThis work?
> >
> > Does using SolrCloud with only 1 shard make any sense at all?
> >
> > Bill
> >
> > On Thu, Oct 25, 2012 at 4:29 PM, Tomás Fernández Löbbe <
> > tomasflobbe@gmail.com> wrote:
> >
> >> It also provides high availability for indexing and searching.
> >>
> >> On Thu, Oct 25, 2012 at 4:43 PM, Bill Au <bi...@gmail.com> wrote:
> >>
> >> > So I guess one would use SolrCloud for the same reasons as distributed
> >> > search:
> >> >
> >> > When an index becomes too large to fit on a single system, or when a
> >> single
> >> > query takes too long to execute.
> >> >
> >> > Bill
> >> >
> >> > On Thu, Oct 25, 2012 at 3:38 PM, Shawn Heisey <so...@elyograg.org>
> wrote:
> >> >
> >> > > On 10/25/2012 1:29 PM, Bill Au wrote:
> >> > >
> >> > >> Is SolrCloud using distributed search behind the scene? Does it
> have
> >> > the
> >> > >> same limitations (for example, doesn't support MoreLikeThis)
> >> distributed
> >> > >> search has?
> >> > >>
> >> > >
> >> > > Yes and yes.
> >> > >
> >> > >
> >> >
> >>
>
Re: SolrCloud and distributed search
Posted by Erick Erickson <er...@gmail.com>.
Yes, I think SolrCloud makes sense with a single shard for exactly
this reason, NRT and multiple replicas. I don't know how you'd get NRT
on multiple machines without it.
But do be aware of: https://issues.apache.org/jira/browse/SOLR-3971
"A collection that is created with numShards=1 turns into a
numShards=2 collection after starting up a second core and not
specifying numShards."
Erick
On Fri, Oct 26, 2012 at 10:14 AM, Bill Au <bi...@gmail.com> wrote:
> I am currently using one master with multiple slaves so I do have high
> availability for searching now.
>
> My index does fit on a single machine and a single query does not take too
> long to execute. But I do want to take advantage of high availability of
> indexing and real time replication. So it looks like I can set up
> SolrCloud with only 1 shard (ie numShards=1).
>
> In this case is SolrCloud still using distributed search behind the
> screen? Will MoreLikeThis work?
>
> Does using SolrCloud with only 1 shard make any sense at all?
>
> Bill
>
> On Thu, Oct 25, 2012 at 4:29 PM, Tomás Fernández Löbbe <
> tomasflobbe@gmail.com> wrote:
>
>> It also provides high availability for indexing and searching.
>>
>> On Thu, Oct 25, 2012 at 4:43 PM, Bill Au <bi...@gmail.com> wrote:
>>
>> > So I guess one would use SolrCloud for the same reasons as distributed
>> > search:
>> >
>> > When an index becomes too large to fit on a single system, or when a
>> single
>> > query takes too long to execute.
>> >
>> > Bill
>> >
>> > On Thu, Oct 25, 2012 at 3:38 PM, Shawn Heisey <so...@elyograg.org> wrote:
>> >
>> > > On 10/25/2012 1:29 PM, Bill Au wrote:
>> > >
>> > >> Is SolrCloud using distributed search behind the scene? Does it have
>> > the
>> > >> same limitations (for example, doesn't support MoreLikeThis)
>> distributed
>> > >> search has?
>> > >>
>> > >
>> > > Yes and yes.
>> > >
>> > >
>> >
>>
Re: SolrCloud and distributed search
Posted by Bill Au <bi...@gmail.com>.
I am currently using one master with multiple slaves so I do have high
availability for searching now.
My index does fit on a single machine and a single query does not take too
long to execute. But I do want to take advantage of high availability of
indexing and real time replication. So it looks like I can set up
SolrCloud with only 1 shard (ie numShards=1).
In this case is SolrCloud still using distributed search behind the
screen? Will MoreLikeThis work?
Does using SolrCloud with only 1 shard make any sense at all?
Bill
On Thu, Oct 25, 2012 at 4:29 PM, Tomás Fernández Löbbe <
tomasflobbe@gmail.com> wrote:
> It also provides high availability for indexing and searching.
>
> On Thu, Oct 25, 2012 at 4:43 PM, Bill Au <bi...@gmail.com> wrote:
>
> > So I guess one would use SolrCloud for the same reasons as distributed
> > search:
> >
> > When an index becomes too large to fit on a single system, or when a
> single
> > query takes too long to execute.
> >
> > Bill
> >
> > On Thu, Oct 25, 2012 at 3:38 PM, Shawn Heisey <so...@elyograg.org> wrote:
> >
> > > On 10/25/2012 1:29 PM, Bill Au wrote:
> > >
> > >> Is SolrCloud using distributed search behind the scene? Does it have
> > the
> > >> same limitations (for example, doesn't support MoreLikeThis)
> distributed
> > >> search has?
> > >>
> > >
> > > Yes and yes.
> > >
> > >
> >
>
Re: SolrCloud and distributed search
Posted by Tomás Fernández Löbbe <to...@gmail.com>.
It also provides high availability for indexing and searching.
On Thu, Oct 25, 2012 at 4:43 PM, Bill Au <bi...@gmail.com> wrote:
> So I guess one would use SolrCloud for the same reasons as distributed
> search:
>
> When an index becomes too large to fit on a single system, or when a single
> query takes too long to execute.
>
> Bill
>
> On Thu, Oct 25, 2012 at 3:38 PM, Shawn Heisey <so...@elyograg.org> wrote:
>
> > On 10/25/2012 1:29 PM, Bill Au wrote:
> >
> >> Is SolrCloud using distributed search behind the scene? Does it have
> the
> >> same limitations (for example, doesn't support MoreLikeThis) distributed
> >> search has?
> >>
> >
> > Yes and yes.
> >
> >
>
Re: SolrCloud and distributed search
Posted by Bill Au <bi...@gmail.com>.
So I guess one would use SolrCloud for the same reasons as distributed
search:
When an index becomes too large to fit on a single system, or when a single
query takes too long to execute.
Bill
On Thu, Oct 25, 2012 at 3:38 PM, Shawn Heisey <so...@elyograg.org> wrote:
> On 10/25/2012 1:29 PM, Bill Au wrote:
>
>> Is SolrCloud using distributed search behind the scene? Does it have the
>> same limitations (for example, doesn't support MoreLikeThis) distributed
>> search has?
>>
>
> Yes and yes.
>
>
Re: SolrCloud and distributed search
Posted by Shawn Heisey <so...@elyograg.org>.
On 10/25/2012 1:29 PM, Bill Au wrote:
> Is SolrCloud using distributed search behind the scene? Does it have the
> same limitations (for example, doesn't support MoreLikeThis) distributed
> search has?
Yes and yes.