You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Bill Au <bi...@gmail.com> on 2012/11/14 23:47:14 UTC
consistency in SolrCloud replication
Will a newly indexed document included in search result in the shard leader
as soon as it has been indexed locally or is it included in search result
only after it has been forwarded to and indexed in all the replicas?
Bill
Re: consistency in SolrCloud replication
Posted by Mark Miller <ma...@gmail.com>.
> I want to know where SolrCloud stands in terms of CAP.
SolrCloud is a CP system. In the face of partitions, SolrCloud favors
consistency over availability (mostly concerning writes). The system
is eventually consistent, but should become consistent with a pretty
low latency, unlike many cases with an AP system (dynamo style
systems).
You can get things like read your own writes (simple out of the box
behavior) and monotonic reads with the SolrCloud architecture if your
application requires it.
Hopefully, we will be adding more knobs to control some of the
tradeoffs that the current architecture can make over time.
--
- Mark
RE: consistency in SolrCloud replication
Posted by Markus Jelsma <ma...@openindex.io>.
Solr is provides availability and it is tolerant to partioning so that leaves consistency. It is eventual consistent.
-----Original message-----
> From:Bill Au <bi...@gmail.com>
> Sent: Fri 16-Nov-2012 15:00
> To: solr-user@lucene.apache.org
> Subject: Re: consistency in SolrCloud replication
>
> Yes, my original question is about search. And Mark did answered is in his
> original reply. I am guessing that the replicas are updated sequentially
> so the newly added documents will be available in some replicas before
> other. I want to know where SolrCloud stands in terms of CAP.
>
> Bill
>
>
> On Thu, Nov 15, 2012 at 10:31 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> > I think Bill was asking about search....
> > I think the Q is whether the query hitting the shard where a doc was sent
> > for indexing would see that doc even before that doc has been copied to
> > replicas.
> >
> > I didn't test it, but I'd think the answer would be positive because of the
> > xa log.
> >
> > Otis
> > --
> > Performance Monitoring - http://sematext.com/spm
> > On Nov 15, 2012 11:30 AM, "Mark Miller" <ma...@gmail.com> wrote:
> >
> > > It depends - no commit necessary for realtime get. Otherwise, yes, you
> > > would need to do at least a soft commit. That works the same way though -
> > > so if you make your update, then do a soft commit, you can be sure your
> > > next search will see the update on all the replicas. And with realtime
> > get,
> > > of course no commit is necessary to see it.
> > >
> > > - Mark
> > >
> > > On Nov 15, 2012, at 10:40 AM, David Smiley (@MITRE.org) <
> > DSMILEY@mitre.org>
> > > wrote:
> > >
> > > > Mark Miller-3 wrote
> > > >> I'm talking about an update request. So if you make an update, when it
> > > >> returns, your next search will see the update, because it will be on
> > > >> all replicas.
> > > >
> > > > I presume this is only the case if (of course) the client also sent a
> > > > commit. So you're saying the commit call will not return unless all
> > > > replicas have completed their commits. Right?
> > > >
> > > > ~ David
> > > >
> > > >
> > > >
> > > > -----
> > > > Author:
> > > http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
> > > > --
> > > > View this message in context:
> > >
> > http://lucene.472066.n3.nabble.com/consistency-in-SolrCloud-replication-tp4020379p4020518.html
> > > > Sent from the Solr - User mailing list archive at Nabble.com.
> > >
> > >
> >
>
Re: consistency in SolrCloud replication
Posted by Bill Au <bi...@gmail.com>.
Yes, my original question is about search. And Mark did answered is in his
original reply. I am guessing that the replicas are updated sequentially
so the newly added documents will be available in some replicas before
other. I want to know where SolrCloud stands in terms of CAP.
Bill
On Thu, Nov 15, 2012 at 10:31 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> I think Bill was asking about search....
> I think the Q is whether the query hitting the shard where a doc was sent
> for indexing would see that doc even before that doc has been copied to
> replicas.
>
> I didn't test it, but I'd think the answer would be positive because of the
> xa log.
>
> Otis
> --
> Performance Monitoring - http://sematext.com/spm
> On Nov 15, 2012 11:30 AM, "Mark Miller" <ma...@gmail.com> wrote:
>
> > It depends - no commit necessary for realtime get. Otherwise, yes, you
> > would need to do at least a soft commit. That works the same way though -
> > so if you make your update, then do a soft commit, you can be sure your
> > next search will see the update on all the replicas. And with realtime
> get,
> > of course no commit is necessary to see it.
> >
> > - Mark
> >
> > On Nov 15, 2012, at 10:40 AM, David Smiley (@MITRE.org) <
> DSMILEY@mitre.org>
> > wrote:
> >
> > > Mark Miller-3 wrote
> > >> I'm talking about an update request. So if you make an update, when it
> > >> returns, your next search will see the update, because it will be on
> > >> all replicas.
> > >
> > > I presume this is only the case if (of course) the client also sent a
> > > commit. So you're saying the commit call will not return unless all
> > > replicas have completed their commits. Right?
> > >
> > > ~ David
> > >
> > >
> > >
> > > -----
> > > Author:
> > http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
> > > --
> > > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/consistency-in-SolrCloud-replication-tp4020379p4020518.html
> > > Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
>
Re: consistency in SolrCloud replication
Posted by Otis Gospodnetic <ot...@gmail.com>.
I think Bill was asking about search....
I think the Q is whether the query hitting the shard where a doc was sent
for indexing would see that doc even before that doc has been copied to
replicas.
I didn't test it, but I'd think the answer would be positive because of the
xa log.
Otis
--
Performance Monitoring - http://sematext.com/spm
On Nov 15, 2012 11:30 AM, "Mark Miller" <ma...@gmail.com> wrote:
> It depends - no commit necessary for realtime get. Otherwise, yes, you
> would need to do at least a soft commit. That works the same way though -
> so if you make your update, then do a soft commit, you can be sure your
> next search will see the update on all the replicas. And with realtime get,
> of course no commit is necessary to see it.
>
> - Mark
>
> On Nov 15, 2012, at 10:40 AM, David Smiley (@MITRE.org) <DS...@mitre.org>
> wrote:
>
> > Mark Miller-3 wrote
> >> I'm talking about an update request. So if you make an update, when it
> >> returns, your next search will see the update, because it will be on
> >> all replicas.
> >
> > I presume this is only the case if (of course) the client also sent a
> > commit. So you're saying the commit call will not return unless all
> > replicas have completed their commits. Right?
> >
> > ~ David
> >
> >
> >
> > -----
> > Author:
> http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
> > --
> > View this message in context:
> http://lucene.472066.n3.nabble.com/consistency-in-SolrCloud-replication-tp4020379p4020518.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
>
>
Re: consistency in SolrCloud replication
Posted by Mark Miller <ma...@gmail.com>.
It depends - no commit necessary for realtime get. Otherwise, yes, you would need to do at least a soft commit. That works the same way though - so if you make your update, then do a soft commit, you can be sure your next search will see the update on all the replicas. And with realtime get, of course no commit is necessary to see it.
- Mark
On Nov 15, 2012, at 10:40 AM, David Smiley (@MITRE.org) <DS...@mitre.org> wrote:
> Mark Miller-3 wrote
>> I'm talking about an update request. So if you make an update, when it
>> returns, your next search will see the update, because it will be on
>> all replicas.
>
> I presume this is only the case if (of course) the client also sent a
> commit. So you're saying the commit call will not return unless all
> replicas have completed their commits. Right?
>
> ~ David
>
>
>
> -----
> Author: http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
> --
> View this message in context: http://lucene.472066.n3.nabble.com/consistency-in-SolrCloud-replication-tp4020379p4020518.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Re: consistency in SolrCloud replication
Posted by "David Smiley (@MITRE.org)" <DS...@mitre.org>.
Mark Miller-3 wrote
> I'm talking about an update request. So if you make an update, when it
> returns, your next search will see the update, because it will be on
> all replicas.
I presume this is only the case if (of course) the client also sent a
commit. So you're saying the commit call will not return unless all
replicas have completed their commits. Right?
~ David
-----
Author: http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
--
View this message in context: http://lucene.472066.n3.nabble.com/consistency-in-SolrCloud-replication-tp4020379p4020518.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: consistency in SolrCloud replication
Posted by Mark Miller <ma...@gmail.com>.
I'm talking about an update request. So if you make an update, when it
returns, your next search will see the update, because it will be on
all replicas. Another process that is searching rapidly may see an
"eventually" consistent view though (very briefly). We have some ideas
to make that view "more consistent", for example by allowing for
searcher leases.
- Mark
On Thu, Nov 15, 2012 at 8:53 AM, Bill Au <bi...@gmail.com> wrote:
> Thanks for the info, Mark. By "a request won't return until it's affected
> all replicas", are you referring to the update request or the query?
>
> Bill
>
>
> On Wed, Nov 14, 2012 at 7:57 PM, Mark Miller <ma...@gmail.com> wrote:
>
>> It's included as soon as it has been indexed - though a request won't
>> return until it's affected all replicas. Low latency eventual consistency.
>>
>> - Mark
>>
>> On Nov 14, 2012, at 5:47 PM, Bill Au <bi...@gmail.com> wrote:
>>
>> > Will a newly indexed document included in search result in the shard
>> leader
>> > as soon as it has been indexed locally or is it included in search result
>> > only after it has been forwarded to and indexed in all the replicas?
>> >
>> > Bill
>>
>>
--
- Mark
Re: consistency in SolrCloud replication
Posted by Bill Au <bi...@gmail.com>.
Thanks for the info, Mark. By "a request won't return until it's affected
all replicas", are you referring to the update request or the query?
Bill
On Wed, Nov 14, 2012 at 7:57 PM, Mark Miller <ma...@gmail.com> wrote:
> It's included as soon as it has been indexed - though a request won't
> return until it's affected all replicas. Low latency eventual consistency.
>
> - Mark
>
> On Nov 14, 2012, at 5:47 PM, Bill Au <bi...@gmail.com> wrote:
>
> > Will a newly indexed document included in search result in the shard
> leader
> > as soon as it has been indexed locally or is it included in search result
> > only after it has been forwarded to and indexed in all the replicas?
> >
> > Bill
>
>
Re: consistency in SolrCloud replication
Posted by Mark Miller <ma...@gmail.com>.
It's included as soon as it has been indexed - though a request won't return until it's affected all replicas. Low latency eventual consistency.
- Mark
On Nov 14, 2012, at 5:47 PM, Bill Au <bi...@gmail.com> wrote:
> Will a newly indexed document included in search result in the shard leader
> as soon as it has been indexed locally or is it included in search result
> only after it has been forwarded to and indexed in all the replicas?
>
> Bill