You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Isuru Boyagane <is...@cse.mrt.ac.lk> on 2019/10/08 07:34:54 UTC
How Kafka leader replica decides to advance Highwater Mark (HW) based
on Kafka producer configurations.
Hi,
4
I read about the Kafka replication protocol. I found that Kafka maintains
LEO and HW. As I understood,
LEO: Offset of latest message a replica has seen.
HW: Offset of the latest message which is guaranteed that each replica has
persisted into their local log.
We can set the following acknowledgement methods in Kafka producers
configurations.
1. acks = 0 (Leader replica sends an acknowledgement to the producer
once it has seen the message)
2. acks = 1 (Leader replica sends an acknowledgement to the producer
once it has persisted the message to its local log)
3. acks = all (Leader replica sends an acknowledgement to the
producer once every in-sync replica has persisted the message to
its local
log)
So my question is how the leader advances the HW depending on the
acknowledgement method message producer uses.
What I think is,
1.
for acks = 0, Leader advances the HW when it sees a new message.
2.
for acks = 1, Leader advances the HW when it has written the new message
to its local log.
3.
for acks = all, Leader advances HW when each an every follower sent ack
that they persisted the message.
Is this correct? Can anyone clarify this?
Thanks
Re: How Kafka leader replica decides to advance Highwater Mark (HW)
based on Kafka producer configurations.
Posted by Isuru Boyagane <is...@cse.mrt.ac.lk>.
Hi Peter,
When acks = all and min.insync.replica= replication factor, HW advancement
is done synchronously with acks only because HW advancement is done after
all replicas have persisted the message. Other than that there is no clear
relation between ISR and HW advancement policy. right?
On Tue, 8 Oct 2019 at 21:25, Peter Bukowinski <pm...@gmail.com> wrote:
> The ack method only affects the producer. With acks=0 or 1, HW advancement
> is done asynchronously with the producer, so the leader will continue to
> accept writes regardless of follower status. Only if you have acks=all and
> min ISR set to the replication factor will your acks be synchronous with
> the HW.
>
> If you are producing to a topic partition that is under-replicated, your
> settings determine whether new data can be written to it or not.
>
> -- Peter (from phone)
>
> > On Oct 8, 2019, at 8:31 AM, Isuru Boyagane <
> isuruboyagane.16@cse.mrt.ac.lk> wrote:
> >
> > Hi Peter,
> >
> > Okay. Then, whatever the producer acks configuration is, Leader replica
> > waits for *all* other follower replicas to persist the message before
> > advancing HW(without considering the *current ISR* set)?
> >
> > Thanks
> >
> >> On Tue, 8 Oct 2019 at 14:54, Peter Bukowinski <pm...@gmail.com> wrote:
> >>
> >> The behavior of the high water mark is independent of producer ack
> >> settings. It is a property of topic partitions only. (Remember that
> >> multiple producers with different configurations can write to the same
> >> topic.) The high water mark advances to the latest offset that exists in
> >> all topic partition replicas. If your topic has a replication factor of
> 3,
> >> then a partition’s HW will only advance toward the LEO after both
> replicas
> >> are in sync with the leader.
> >>
> >> min.insync.replicas controls how many replicas must be in sync when your
> >> producer in using acks=all. This does not affect consumers.
> >>
> >> HW does affect consumers, since data between the HW and LEO cannot be
> >> consumed yet.
> >>
> >> --
> >> Peter
> >>
> >>> On Oct 8, 2019, at 12:35 AM, Isuru Boyagane <
> >> isuruboyagane.16@cse.mrt.ac.lk> wrote:
> >>>
> >>> Hi,
> >>>
> >>> 4
> >>>
> >>> I read about the Kafka replication protocol. I found that Kafka
> maintains
> >>> LEO and HW. As I understood,
> >>>
> >>> LEO: Offset of latest message a replica has seen.
> >>>
> >>> HW: Offset of the latest message which is guaranteed that each replica
> >> has
> >>> persisted into their local log.
> >>>
> >>> We can set the following acknowledgement methods in Kafka producers
> >>> configurations.
> >>>
> >>>
> >>> 1. acks = 0 (Leader replica sends an acknowledgement to the producer
> >>> once it has seen the message)
> >>> 2. acks = 1 (Leader replica sends an acknowledgement to the
> producer
> >>> once it has persisted the message to its local log)
> >>> 3. acks = all (Leader replica sends an acknowledgement to the
> >>> producer once every in-sync replica has persisted the message to
> >>> its local
> >>> log)
> >>>
> >>> So my question is how the leader advances the HW depending on the
> >>> acknowledgement method message producer uses.
> >>>
> >>> What I think is,
> >>>
> >>> 1.
> >>>
> >>> for acks = 0, Leader advances the HW when it sees a new message.
> >>> 2.
> >>>
> >>> for acks = 1, Leader advances the HW when it has written the new
> >> message
> >>> to its local log.
> >>> 3.
> >>>
> >>> for acks = all, Leader advances HW when each an every follower sent
> ack
> >>> that they persisted the message.
> >>>
> >>> Is this correct? Can anyone clarify this?
> >>>
> >>> Thanks
> >>
> >
> >
> > --
> > isuru
>
--
Isuru
Re: How Kafka leader replica decides to advance Highwater Mark (HW) based on Kafka producer configurations.
Posted by Peter Bukowinski <pm...@gmail.com>.
The ack method only affects the producer. With acks=0 or 1, HW advancement is done asynchronously with the producer, so the leader will continue to accept writes regardless of follower status. Only if you have acks=all and min ISR set to the replication factor will your acks be synchronous with the HW.
If you are producing to a topic partition that is under-replicated, your settings determine whether new data can be written to it or not.
-- Peter (from phone)
> On Oct 8, 2019, at 8:31 AM, Isuru Boyagane <is...@cse.mrt.ac.lk> wrote:
>
> Hi Peter,
>
> Okay. Then, whatever the producer acks configuration is, Leader replica
> waits for *all* other follower replicas to persist the message before
> advancing HW(without considering the *current ISR* set)?
>
> Thanks
>
>> On Tue, 8 Oct 2019 at 14:54, Peter Bukowinski <pm...@gmail.com> wrote:
>>
>> The behavior of the high water mark is independent of producer ack
>> settings. It is a property of topic partitions only. (Remember that
>> multiple producers with different configurations can write to the same
>> topic.) The high water mark advances to the latest offset that exists in
>> all topic partition replicas. If your topic has a replication factor of 3,
>> then a partition’s HW will only advance toward the LEO after both replicas
>> are in sync with the leader.
>>
>> min.insync.replicas controls how many replicas must be in sync when your
>> producer in using acks=all. This does not affect consumers.
>>
>> HW does affect consumers, since data between the HW and LEO cannot be
>> consumed yet.
>>
>> --
>> Peter
>>
>>> On Oct 8, 2019, at 12:35 AM, Isuru Boyagane <
>> isuruboyagane.16@cse.mrt.ac.lk> wrote:
>>>
>>> Hi,
>>>
>>> 4
>>>
>>> I read about the Kafka replication protocol. I found that Kafka maintains
>>> LEO and HW. As I understood,
>>>
>>> LEO: Offset of latest message a replica has seen.
>>>
>>> HW: Offset of the latest message which is guaranteed that each replica
>> has
>>> persisted into their local log.
>>>
>>> We can set the following acknowledgement methods in Kafka producers
>>> configurations.
>>>
>>>
>>> 1. acks = 0 (Leader replica sends an acknowledgement to the producer
>>> once it has seen the message)
>>> 2. acks = 1 (Leader replica sends an acknowledgement to the producer
>>> once it has persisted the message to its local log)
>>> 3. acks = all (Leader replica sends an acknowledgement to the
>>> producer once every in-sync replica has persisted the message to
>>> its local
>>> log)
>>>
>>> So my question is how the leader advances the HW depending on the
>>> acknowledgement method message producer uses.
>>>
>>> What I think is,
>>>
>>> 1.
>>>
>>> for acks = 0, Leader advances the HW when it sees a new message.
>>> 2.
>>>
>>> for acks = 1, Leader advances the HW when it has written the new
>> message
>>> to its local log.
>>> 3.
>>>
>>> for acks = all, Leader advances HW when each an every follower sent ack
>>> that they persisted the message.
>>>
>>> Is this correct? Can anyone clarify this?
>>>
>>> Thanks
>>
>
>
> --
> isuru
Re: How Kafka leader replica decides to advance Highwater Mark (HW)
based on Kafka producer configurations.
Posted by Isuru Boyagane <is...@cse.mrt.ac.lk>.
Hi Peter,
Okay. Then, whatever the producer acks configuration is, Leader replica
waits for *all* other follower replicas to persist the message before
advancing HW(without considering the *current ISR* set)?
Thanks
On Tue, 8 Oct 2019 at 14:54, Peter Bukowinski <pm...@gmail.com> wrote:
> The behavior of the high water mark is independent of producer ack
> settings. It is a property of topic partitions only. (Remember that
> multiple producers with different configurations can write to the same
> topic.) The high water mark advances to the latest offset that exists in
> all topic partition replicas. If your topic has a replication factor of 3,
> then a partition’s HW will only advance toward the LEO after both replicas
> are in sync with the leader.
>
> min.insync.replicas controls how many replicas must be in sync when your
> producer in using acks=all. This does not affect consumers.
>
> HW does affect consumers, since data between the HW and LEO cannot be
> consumed yet.
>
> --
> Peter
>
> > On Oct 8, 2019, at 12:35 AM, Isuru Boyagane <
> isuruboyagane.16@cse.mrt.ac.lk> wrote:
> >
> > Hi,
> >
> > 4
> >
> > I read about the Kafka replication protocol. I found that Kafka maintains
> > LEO and HW. As I understood,
> >
> > LEO: Offset of latest message a replica has seen.
> >
> > HW: Offset of the latest message which is guaranteed that each replica
> has
> > persisted into their local log.
> >
> > We can set the following acknowledgement methods in Kafka producers
> > configurations.
> >
> >
> > 1. acks = 0 (Leader replica sends an acknowledgement to the producer
> > once it has seen the message)
> > 2. acks = 1 (Leader replica sends an acknowledgement to the producer
> > once it has persisted the message to its local log)
> > 3. acks = all (Leader replica sends an acknowledgement to the
> > producer once every in-sync replica has persisted the message to
> > its local
> > log)
> >
> > So my question is how the leader advances the HW depending on the
> > acknowledgement method message producer uses.
> >
> > What I think is,
> >
> > 1.
> >
> > for acks = 0, Leader advances the HW when it sees a new message.
> > 2.
> >
> > for acks = 1, Leader advances the HW when it has written the new
> message
> > to its local log.
> > 3.
> >
> > for acks = all, Leader advances HW when each an every follower sent ack
> > that they persisted the message.
> >
> > Is this correct? Can anyone clarify this?
> >
> > Thanks
>
--
isuru
Re: How Kafka leader replica decides to advance Highwater Mark (HW) based on Kafka producer configurations.
Posted by Peter Bukowinski <pm...@gmail.com>.
The behavior of the high water mark is independent of producer ack settings. It is a property of topic partitions only. (Remember that multiple producers with different configurations can write to the same topic.) The high water mark advances to the latest offset that exists in all topic partition replicas. If your topic has a replication factor of 3, then a partition’s HW will only advance toward the LEO after both replicas are in sync with the leader.
min.insync.replicas controls how many replicas must be in sync when your producer in using acks=all. This does not affect consumers.
HW does affect consumers, since data between the HW and LEO cannot be consumed yet.
--
Peter
> On Oct 8, 2019, at 12:35 AM, Isuru Boyagane <is...@cse.mrt.ac.lk> wrote:
>
> Hi,
>
> 4
>
> I read about the Kafka replication protocol. I found that Kafka maintains
> LEO and HW. As I understood,
>
> LEO: Offset of latest message a replica has seen.
>
> HW: Offset of the latest message which is guaranteed that each replica has
> persisted into their local log.
>
> We can set the following acknowledgement methods in Kafka producers
> configurations.
>
>
> 1. acks = 0 (Leader replica sends an acknowledgement to the producer
> once it has seen the message)
> 2. acks = 1 (Leader replica sends an acknowledgement to the producer
> once it has persisted the message to its local log)
> 3. acks = all (Leader replica sends an acknowledgement to the
> producer once every in-sync replica has persisted the message to
> its local
> log)
>
> So my question is how the leader advances the HW depending on the
> acknowledgement method message producer uses.
>
> What I think is,
>
> 1.
>
> for acks = 0, Leader advances the HW when it sees a new message.
> 2.
>
> for acks = 1, Leader advances the HW when it has written the new message
> to its local log.
> 3.
>
> for acks = all, Leader advances HW when each an every follower sent ack
> that they persisted the message.
>
> Is this correct? Can anyone clarify this?
>
> Thanks