You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by SuoNayi <su...@163.com> on 2013/04/16 07:36:40 UTC
Push/pull model between leader and replica in one shard
Hi, can someone explain more details about what model is used to sync docs between the lead and
replica in the shard?
The model can be push or pull.Supposing I have only one shard that has 1 leader and 2 replicas,
when the leader receives a update request, does it will scatter the request to each available and active
replica at first and then processes the request locally at last?In this case if the replicas are able to catch
up with the leader can I think this is a push model that the leader pushes updates to it's replicas?
What happens if a replica is behind the leader?Will the replica pull docs from the leader and keep
a track of the coming updates from the lead in a log(called tlog)?If so when it complete pulling docs
it will replay updates in the tlog at last?
regards
Re: Push/pull model between leader and replica in one shard
Posted by Mark Miller <ma...@gmail.com>.
Thanks, the earlier presentation is done with KeyNote and the later (more animation) is done with Tumult Hype.
- Mark
On Apr 17, 2013, at 3:43 AM, Furkan KAMACI <fu...@gmail.com> wrote:
> Hej Mark;
>
> What did you use to prepare your presentation, its really nice.
>
> 2013/4/17 Furkan KAMACI <fu...@gmail.com>
>
>> Really nice presentation.
>>
>>
>> 2013/4/17 Mark Miller <ma...@gmail.com>
>>
>>>
>>> On Apr 16, 2013, at 1:36 AM, SuoNayi <su...@163.com> wrote:
>>>
>>>> Hi, can someone explain more details about what model is used to sync
>>> docs between the lead and
>>>> replica in the shard?
>>>> The model can be push or pull.Supposing I have only one shard that has
>>> 1 leader and 2 replicas,
>>>> when the leader receives a update request, does it will scatter the
>>> request to each available and active
>>>> replica at first and then processes the request locally at last?In this
>>> case if the replicas are able to catch
>>>> up with the leader can I think this is a push model that the leader
>>> pushes updates to it's replicas?
>>>
>>> Currently, the leader adds the doc locally and then sends it to all
>>> replicas concurrently.
>>>
>>>>
>>>>
>>>> What happens if a replica is behind the leader?Will the replica pull
>>> docs from the leader and keep
>>>> a track of the coming updates from the lead in a log(called tlog)?If so
>>> when it complete pulling docs
>>>> it will replay updates in the tlog at last?
>>>
>>> If an update forwarded from a leader to a replica fails it's likely
>>> because that replica died. Just in case, the leader will ask that replica
>>> to enter "recovery".
>>>
>>> When a node comes up and is not a leader, it also enters "recovery".
>>>
>>> Recovery tries to peersync from the leader, and if that fails (works if
>>> off by about 100 updates), it replicates the entire index.
>>>
>>> If you are interested in more details on the SolrCloud architecture, I've
>>> given a few talks on it - two of them here:
>>>
>>> http://vimeo.com/43913870
>>> http://www.youtube.com/watch?v=eVK0wLkLw9w
>>>
>>> - Mark
>>>
>>>
>>
Re: Push/pull model between leader and replica in one shard
Posted by Furkan KAMACI <fu...@gmail.com>.
Hej Mark;
What did you use to prepare your presentation, its really nice.
2013/4/17 Furkan KAMACI <fu...@gmail.com>
> Really nice presentation.
>
>
> 2013/4/17 Mark Miller <ma...@gmail.com>
>
>>
>> On Apr 16, 2013, at 1:36 AM, SuoNayi <su...@163.com> wrote:
>>
>> > Hi, can someone explain more details about what model is used to sync
>> docs between the lead and
>> > replica in the shard?
>> > The model can be push or pull.Supposing I have only one shard that has
>> 1 leader and 2 replicas,
>> > when the leader receives a update request, does it will scatter the
>> request to each available and active
>> > replica at first and then processes the request locally at last?In this
>> case if the replicas are able to catch
>> > up with the leader can I think this is a push model that the leader
>> pushes updates to it's replicas?
>>
>> Currently, the leader adds the doc locally and then sends it to all
>> replicas concurrently.
>>
>> >
>> >
>> > What happens if a replica is behind the leader?Will the replica pull
>> docs from the leader and keep
>> > a track of the coming updates from the lead in a log(called tlog)?If so
>> when it complete pulling docs
>> > it will replay updates in the tlog at last?
>>
>> If an update forwarded from a leader to a replica fails it's likely
>> because that replica died. Just in case, the leader will ask that replica
>> to enter "recovery".
>>
>> When a node comes up and is not a leader, it also enters "recovery".
>>
>> Recovery tries to peersync from the leader, and if that fails (works if
>> off by about 100 updates), it replicates the entire index.
>>
>> If you are interested in more details on the SolrCloud architecture, I've
>> given a few talks on it - two of them here:
>>
>> http://vimeo.com/43913870
>> http://www.youtube.com/watch?v=eVK0wLkLw9w
>>
>> - Mark
>>
>>
>
Re: Push/pull model between leader and replica in one shard
Posted by Furkan KAMACI <fu...@gmail.com>.
Really nice presentation.
2013/4/17 Mark Miller <ma...@gmail.com>
>
> On Apr 16, 2013, at 1:36 AM, SuoNayi <su...@163.com> wrote:
>
> > Hi, can someone explain more details about what model is used to sync
> docs between the lead and
> > replica in the shard?
> > The model can be push or pull.Supposing I have only one shard that has 1
> leader and 2 replicas,
> > when the leader receives a update request, does it will scatter the
> request to each available and active
> > replica at first and then processes the request locally at last?In this
> case if the replicas are able to catch
> > up with the leader can I think this is a push model that the leader
> pushes updates to it's replicas?
>
> Currently, the leader adds the doc locally and then sends it to all
> replicas concurrently.
>
> >
> >
> > What happens if a replica is behind the leader?Will the replica pull
> docs from the leader and keep
> > a track of the coming updates from the lead in a log(called tlog)?If so
> when it complete pulling docs
> > it will replay updates in the tlog at last?
>
> If an update forwarded from a leader to a replica fails it's likely
> because that replica died. Just in case, the leader will ask that replica
> to enter "recovery".
>
> When a node comes up and is not a leader, it also enters "recovery".
>
> Recovery tries to peersync from the leader, and if that fails (works if
> off by about 100 updates), it replicates the entire index.
>
> If you are interested in more details on the SolrCloud architecture, I've
> given a few talks on it - two of them here:
>
> http://vimeo.com/43913870
> http://www.youtube.com/watch?v=eVK0wLkLw9w
>
> - Mark
>
>
Re: Push/pull model between leader and replica in one shard
Posted by Mark Miller <ma...@gmail.com>.
On Apr 16, 2013, at 1:36 AM, SuoNayi <su...@163.com> wrote:
> Hi, can someone explain more details about what model is used to sync docs between the lead and
> replica in the shard?
> The model can be push or pull.Supposing I have only one shard that has 1 leader and 2 replicas,
> when the leader receives a update request, does it will scatter the request to each available and active
> replica at first and then processes the request locally at last?In this case if the replicas are able to catch
> up with the leader can I think this is a push model that the leader pushes updates to it's replicas?
Currently, the leader adds the doc locally and then sends it to all replicas concurrently.
>
>
> What happens if a replica is behind the leader?Will the replica pull docs from the leader and keep
> a track of the coming updates from the lead in a log(called tlog)?If so when it complete pulling docs
> it will replay updates in the tlog at last?
If an update forwarded from a leader to a replica fails it's likely because that replica died. Just in case, the leader will ask that replica to enter "recovery".
When a node comes up and is not a leader, it also enters "recovery".
Recovery tries to peersync from the leader, and if that fails (works if off by about 100 updates), it replicates the entire index.
If you are interested in more details on the SolrCloud architecture, I've given a few talks on it - two of them here:
http://vimeo.com/43913870
http://www.youtube.com/watch?v=eVK0wLkLw9w
- Mark
Re: Push/pull model between leader and replica in one shard
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
Replication when everything is working well is push:
* request comes to any node, ideally leader
* doc is indexed on leader
* doc is copied to replicas
If replica falls too far behind (not exactly sure what the "too far"
threshold is), it uses pull to replicate the whole index from leader.
Mark can answer the part about where tlog gets replayed to catch up on
docs that were missed while big index replication pull was happening.
This is a good thread to read on this topic:
http://search-lucene.com/m/y1yj218J2v82
Otis
--
Solr & ElasticSearch Support
http://sematext.com/
On Tue, Apr 16, 2013 at 1:36 AM, SuoNayi <su...@163.com> wrote:
> Hi, can someone explain more details about what model is used to sync docs between the lead and
> replica in the shard?
> The model can be push or pull.Supposing I have only one shard that has 1 leader and 2 replicas,
> when the leader receives a update request, does it will scatter the request to each available and active
> replica at first and then processes the request locally at last?In this case if the replicas are able to catch
> up with the leader can I think this is a push model that the leader pushes updates to it's replicas?
>
>
> What happens if a replica is behind the leader?Will the replica pull docs from the leader and keep
> a track of the coming updates from the lead in a log(called tlog)?If so when it complete pulling docs
> it will replay updates in the tlog at last?
>
>
>
>
> regards
>
>
>
>
>
>
>