You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Taisuke Miyazaki <mi...@lifull.com> on 2020/04/06 05:53:53 UTC
If the leader dies, will the data be lost?
Hi,
Using solr 7.5.0 on solr cloud, and replica type is tlog.
If a leader dies, how is the re-election of the leader and the
synchronization of the replicas done?
In my opinion.
Leader dies→ New tlog replica tries to become Leader→ Replays tlogs not
reflected in the index→ Becomes Leader
Is this the right fit first?
Also, when another leader is elected, does it create a tlog that is only
available to the old leader? (I'm worried about data being lost if the
tlogs aren't synchronized.)
Re: If the leader dies, will the data be lost?
Posted by Erick Erickson <er...@gmail.com>.
bq. Does the replication by tlog always work like this?
Yes. Otherwise there’s no way to even attempt to guarantee data integrity. The raw updates _must_ be received by all non-leaders that are eligible (i.e. TLOG or NRT but not PULL replicas).
BTW, anecdotally I’ve seen indexing throughput take quite a hit because of this, like 30% or more when comparing leader-only with leader + NRT or TLOG replicas. This was an outlier because the records were very small and the communications overhead to forward the raw docs and wait for an ack back was large in comparison to the time it actually took to index the docs. Most installations have much less of a difference.
And, BTW, the docs are forwarded to individual replicas in parallel.
Best,
Erick
> On Apr 6, 2020, at 10:08 PM, Taisuke Miyazaki <mi...@lifull.com> wrote:
>
> Hi Erick,
>
>> Before the leader goes down, the sequence of an update is this.
>> - the doc comes in to the leader (TL)
>> - the doc is forwarded to all the other tlog replicas (TF) and written to
> _their_ tlogs
>> - all the TF replicas ack back to TL
>> - TL acks back to the client
>
> So the write request doesn't return a response until all the tlog replicas
> have been written!
> Thank you.
>
> BTW, does replication by tlog always work like this?
>
>
>
>
> 2020年4月6日(月) 20:42 Erick Erickson <er...@gmail.com>:
>
>> You’ve got the sequence, that’s it exactly.
>>
>> I don’t quite understand the second part of the question, but let me
>> address data loss.
>>
>> Before the leader goes down, the sequence of an update is this.
>> - the doc comes in to the leader (TL)
>> - the doc is forwarded to all the other tlog replicas (TF) and written to
>> _their_ tlogs
>> - all the TF replicas ack back to TL
>> - TL acks back to the client
>>
>> So, upon getting success back from the update request, all TLOG replicas
>> have the
>> docs in their local tlog files. So when the leadership changes, the new
>> leader
>> has all the docs to replay, thus no data loss.
>>
>> At that point, the old leader’s tlogs are irrelevant. When it comes back
>> online,
>> the sequence is:
>>
>> - synch from the new leader, including any tlogs. This effectively erases
>> the old tlogs
>> - start writing any new docs into the local tlog
>>
>> The old leader then remains a follower until some event changes things
>> again.
>>
>> Best,
>> Erick
>>
>>> On Apr 6, 2020, at 1:53 AM, Taisuke Miyazaki <mi...@lifull.com>
>> wrote:
>>>
>>> Hi,
>>> Using solr 7.5.0 on solr cloud, and replica type is tlog.
>>>
>>> If a leader dies, how is the re-election of the leader and the
>>> synchronization of the replicas done?
>>>
>>> In my opinion.
>>> Leader dies→ New tlog replica tries to become Leader→ Replays tlogs not
>>> reflected in the index→ Becomes Leader
>>> Is this the right fit first?
>>>
>>> Also, when another leader is elected, does it create a tlog that is only
>>> available to the old leader? (I'm worried about data being lost if the
>>> tlogs aren't synchronized.)
>>
>>
Re: If the leader dies, will the data be lost?
Posted by Taisuke Miyazaki <mi...@lifull.com>.
Hi Erick,
> Before the leader goes down, the sequence of an update is this.
> - the doc comes in to the leader (TL)
> - the doc is forwarded to all the other tlog replicas (TF) and written to
_their_ tlogs
> - all the TF replicas ack back to TL
> - TL acks back to the client
So the write request doesn't return a response until all the tlog replicas
have been written!
Thank you.
BTW, does replication by tlog always work like this?
2020年4月6日(月) 20:42 Erick Erickson <er...@gmail.com>:
> You’ve got the sequence, that’s it exactly.
>
> I don’t quite understand the second part of the question, but let me
> address data loss.
>
> Before the leader goes down, the sequence of an update is this.
> - the doc comes in to the leader (TL)
> - the doc is forwarded to all the other tlog replicas (TF) and written to
> _their_ tlogs
> - all the TF replicas ack back to TL
> - TL acks back to the client
>
> So, upon getting success back from the update request, all TLOG replicas
> have the
> docs in their local tlog files. So when the leadership changes, the new
> leader
> has all the docs to replay, thus no data loss.
>
> At that point, the old leader’s tlogs are irrelevant. When it comes back
> online,
> the sequence is:
>
> - synch from the new leader, including any tlogs. This effectively erases
> the old tlogs
> - start writing any new docs into the local tlog
>
> The old leader then remains a follower until some event changes things
> again.
>
> Best,
> Erick
>
> > On Apr 6, 2020, at 1:53 AM, Taisuke Miyazaki <mi...@lifull.com>
> wrote:
> >
> > Hi,
> > Using solr 7.5.0 on solr cloud, and replica type is tlog.
> >
> > If a leader dies, how is the re-election of the leader and the
> > synchronization of the replicas done?
> >
> > In my opinion.
> > Leader dies→ New tlog replica tries to become Leader→ Replays tlogs not
> > reflected in the index→ Becomes Leader
> > Is this the right fit first?
> >
> > Also, when another leader is elected, does it create a tlog that is only
> > available to the old leader? (I'm worried about data being lost if the
> > tlogs aren't synchronized.)
>
>
Re: If the leader dies, will the data be lost?
Posted by Erick Erickson <er...@gmail.com>.
You’ve got the sequence, that’s it exactly.
I don’t quite understand the second part of the question, but let me address data loss.
Before the leader goes down, the sequence of an update is this.
- the doc comes in to the leader (TL)
- the doc is forwarded to all the other tlog replicas (TF) and written to _their_ tlogs
- all the TF replicas ack back to TL
- TL acks back to the client
So, upon getting success back from the update request, all TLOG replicas have the
docs in their local tlog files. So when the leadership changes, the new leader
has all the docs to replay, thus no data loss.
At that point, the old leader’s tlogs are irrelevant. When it comes back online,
the sequence is:
- synch from the new leader, including any tlogs. This effectively erases the old tlogs
- start writing any new docs into the local tlog
The old leader then remains a follower until some event changes things again.
Best,
Erick
> On Apr 6, 2020, at 1:53 AM, Taisuke Miyazaki <mi...@lifull.com> wrote:
>
> Hi,
> Using solr 7.5.0 on solr cloud, and replica type is tlog.
>
> If a leader dies, how is the re-election of the leader and the
> synchronization of the replicas done?
>
> In my opinion.
> Leader dies→ New tlog replica tries to become Leader→ Replays tlogs not
> reflected in the index→ Becomes Leader
> Is this the right fit first?
>
> Also, when another leader is elected, does it create a tlog that is only
> available to the old leader? (I'm worried about data being lost if the
> tlogs aren't synchronized.)