You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Nag Y <an...@gmail.com> on 2020/06/20 20:04:59 UTC
Highwater mark interpretation
As I understand it, the consumer can only read "committed" messages - which
I believe, if we look at internals of it, committed messages are nothing
but messages which are upto the high watermark.
*The high watermark is the offset of the last message that was successfully
copied to all of the log’s replicas. *
*Having said that, if one of the replica is down, will high water mark be*
*advanced?*
*If replica can't come forever, can we consider this message cant be
consumed by the consumer since it is never committed *
Re: Highwater mark interpretation
Posted by D C <dr...@gmail.com>.
Hey Nag Y,
I’m not exactly sure if reducing the replication factor while a broker is
down would release the messages to be consumed (or at least not on all
partitions) for the simple fact that it might just remove the last replica
in the list which might not mach your unreachable broker.
Personally i would go and do a manual reassignment of partitions (kafka
manager allows you to do that in an easy visual environment) and move the
replicas out of the broken broker to a working one and once that’s done and
the data copied to the new broker the high watermark should go up as all
the replicas will be in sync.
Cheers,
D
On Sunday, June 21, 2020, Nag Y <an...@gmail.com> wrote:
> Thanks D C. Thanks a lot . That is quite a detailed explanation.
> If I understand correctly, ( ignoring the case where producers
> create transactions) - since the replica is down and never comes , the high
> watermark CANNOT advance and the consumer CAN NOT read the messages which
> were sent after the replica is down as the message is NOT committed - Hope
> this is correct ?
——————————
Indeed, this is correct.
——————————
>
> To address this situation, either we should make sure the replica is up or
> reduce the replication factor so that the message will be committed and
> consumer can start reading the messages ...
>
> Regards,
> Nag
>
>
> On Sun, Jun 21, 2020 at 3:25 AM D C <dr...@gmail.com> wrote:
>
> > The short answer is : yes, a consumer can only consume messages up to the
> > High Watermark.
> >
> > The long answer is not exactly, for the following reasons:
> >
> > At the partition level you have 3 major offsets that are important to the
> > health of the partition and accessibility from the consumer pov:
> > LeO (log end offset) - which represents the highest offset in the highest
> > segment
> > High Watermark - which represents the latest offset that has been
> > replicated to all the followers
> > LSO (Last stable offset) - which is important when you use producers that
> > create transactions - which represents the the highest offset that has
> been
> > committed by a transaction and that is allowed to be read with isolation
> > level = read_commited.
> >
> > The LeO can only be higher or equal to the High Watermark (for obvious
> > reasons)
> > The High Watermark can only be higher or equal to the LSO (the messages
> up
> > to this point may have been committed to all the followers but the
> > transaction isn't yet finished)
> > And coming to your question, in case the transaction hasn't finished, the
> > LSO may be lower than the High Watermark so if your consumer is accessing
> > the data in Read_Committed, it won't be able to surpass the LSO.
> >
> > Cheers,
> > D
> >
> > On Sat, Jun 20, 2020 at 9:05 PM Nag Y <an...@gmail.com>
> wrote:
> >
> > > As I understand it, the consumer can only read "committed" messages -
> > which
> > > I believe, if we look at internals of it, committed messages are
> nothing
> > > but messages which are upto the high watermark.
> > > *The high watermark is the offset of the last message that was
> > successfully
> > > copied to all of the log’s replicas. *
> > >
> > > *Having said that, if one of the replica is down, will high water mark
> > be*
> > > *advanced?*
> > >
> > > *If replica can't come forever, can we consider this message cant be
> > > consumed by the consumer since it is never committed *
> > >
> >
>
--
Re: Highwater mark interpretation
Posted by Nag Y <an...@gmail.com>.
Thanks Liam Clarke-Hutchinson for inputs.
Yes, the case that i described is very rare as it is expected *like you
mentioned already *the replica failures are transient and eventually caught
up once they are back . And the kind of infrastructure we have today, this
could eventually be supported by monitoring tools or by using auto heal
architectures like kubernetes.
If it ever happens, that is where *as D C suggested *to re assign
partitions to start the consuming the messages/
Regards,
Nag
On Sun, Jun 21, 2020 at 12:11 PM Liam Clarke-Hutchinson <
liam.clarke@adscale.co.nz> wrote:
> Hi Nag,
>
> In my experience running Kafka in production for 6 years, so long as the
> number of replicas (and the leader is one of those replicas) in the insync
> replica set (ISR) is greater than the min.insync.replica setting, the
> partition leader will accept writes, and customers can read those writes,
> even if the topic is underreplicated.
>
> Being able to do so is sort of one of the key features of Kafka.
>
> An out of sync partition replica will catch up when it comes back online,
> but until it's a member of the ISR it won't be eligible to become the
> partition leader, or for rack aware consumption (a rack aware consumer in
> Kafka 2.4.5+ (IIRC) can consume from a replica in the same rack instead of
> having to consume from the leader, which can improve latency and in cloud
> environments reduce data ingress/egress costs).
>
> But, I've never encountered a situation where an underreplicated partition
> prevents consuming.
>
> Cheers,
>
> Liam Clarke-Hutchinson
>
> On Sun, 21 Jun. 2020, 2:08 pm Nag Y, <an...@gmail.com> wrote:
>
> > Thanks D C. Thanks a lot . That is quite a detailed explanation.
> > If I understand correctly, ( ignoring the case where producers
> > create transactions) - since the replica is down and never comes , the
> high
> > watermark CANNOT advance and the consumer CAN NOT read the messages which
> > were sent after the replica is down as the message is NOT committed -
> Hope
> > this is correct ?
> >
> > To address this situation, either we should make sure the replica is up
> or
> > reduce the replication factor so that the message will be committed and
> > consumer can start reading the messages ...
> >
> > Regards,
> > Nag
> >
> >
> > On Sun, Jun 21, 2020 at 3:25 AM D C <dr...@gmail.com> wrote:
> >
> > > The short answer is : yes, a consumer can only consume messages up to
> the
> > > High Watermark.
> > >
> > > The long answer is not exactly, for the following reasons:
> > >
> > > At the partition level you have 3 major offsets that are important to
> the
> > > health of the partition and accessibility from the consumer pov:
> > > LeO (log end offset) - which represents the highest offset in the
> highest
> > > segment
> > > High Watermark - which represents the latest offset that has been
> > > replicated to all the followers
> > > LSO (Last stable offset) - which is important when you use producers
> that
> > > create transactions - which represents the the highest offset that has
> > been
> > > committed by a transaction and that is allowed to be read with
> isolation
> > > level = read_commited.
> > >
> > > The LeO can only be higher or equal to the High Watermark (for obvious
> > > reasons)
> > > The High Watermark can only be higher or equal to the LSO (the messages
> > up
> > > to this point may have been committed to all the followers but the
> > > transaction isn't yet finished)
> > > And coming to your question, in case the transaction hasn't finished,
> the
> > > LSO may be lower than the High Watermark so if your consumer is
> accessing
> > > the data in Read_Committed, it won't be able to surpass the LSO.
> > >
> > > Cheers,
> > > D
> > >
> > > On Sat, Jun 20, 2020 at 9:05 PM Nag Y <an...@gmail.com>
> > wrote:
> > >
> > > > As I understand it, the consumer can only read "committed" messages -
> > > which
> > > > I believe, if we look at internals of it, committed messages are
> > nothing
> > > > but messages which are upto the high watermark.
> > > > *The high watermark is the offset of the last message that was
> > > successfully
> > > > copied to all of the log’s replicas. *
> > > >
> > > > *Having said that, if one of the replica is down, will high water
> mark
> > > be*
> > > > *advanced?*
> > > >
> > > > *If replica can't come forever, can we consider this message cant be
> > > > consumed by the consumer since it is never committed *
> > > >
> > >
> >
>
Re: Highwater mark interpretation
Posted by Liam Clarke-Hutchinson <li...@adscale.co.nz>.
Hi Nag,
In my experience running Kafka in production for 6 years, so long as the
number of replicas (and the leader is one of those replicas) in the insync
replica set (ISR) is greater than the min.insync.replica setting, the
partition leader will accept writes, and customers can read those writes,
even if the topic is underreplicated.
Being able to do so is sort of one of the key features of Kafka.
An out of sync partition replica will catch up when it comes back online,
but until it's a member of the ISR it won't be eligible to become the
partition leader, or for rack aware consumption (a rack aware consumer in
Kafka 2.4.5+ (IIRC) can consume from a replica in the same rack instead of
having to consume from the leader, which can improve latency and in cloud
environments reduce data ingress/egress costs).
But, I've never encountered a situation where an underreplicated partition
prevents consuming.
Cheers,
Liam Clarke-Hutchinson
On Sun, 21 Jun. 2020, 2:08 pm Nag Y, <an...@gmail.com> wrote:
> Thanks D C. Thanks a lot . That is quite a detailed explanation.
> If I understand correctly, ( ignoring the case where producers
> create transactions) - since the replica is down and never comes , the high
> watermark CANNOT advance and the consumer CAN NOT read the messages which
> were sent after the replica is down as the message is NOT committed - Hope
> this is correct ?
>
> To address this situation, either we should make sure the replica is up or
> reduce the replication factor so that the message will be committed and
> consumer can start reading the messages ...
>
> Regards,
> Nag
>
>
> On Sun, Jun 21, 2020 at 3:25 AM D C <dr...@gmail.com> wrote:
>
> > The short answer is : yes, a consumer can only consume messages up to the
> > High Watermark.
> >
> > The long answer is not exactly, for the following reasons:
> >
> > At the partition level you have 3 major offsets that are important to the
> > health of the partition and accessibility from the consumer pov:
> > LeO (log end offset) - which represents the highest offset in the highest
> > segment
> > High Watermark - which represents the latest offset that has been
> > replicated to all the followers
> > LSO (Last stable offset) - which is important when you use producers that
> > create transactions - which represents the the highest offset that has
> been
> > committed by a transaction and that is allowed to be read with isolation
> > level = read_commited.
> >
> > The LeO can only be higher or equal to the High Watermark (for obvious
> > reasons)
> > The High Watermark can only be higher or equal to the LSO (the messages
> up
> > to this point may have been committed to all the followers but the
> > transaction isn't yet finished)
> > And coming to your question, in case the transaction hasn't finished, the
> > LSO may be lower than the High Watermark so if your consumer is accessing
> > the data in Read_Committed, it won't be able to surpass the LSO.
> >
> > Cheers,
> > D
> >
> > On Sat, Jun 20, 2020 at 9:05 PM Nag Y <an...@gmail.com>
> wrote:
> >
> > > As I understand it, the consumer can only read "committed" messages -
> > which
> > > I believe, if we look at internals of it, committed messages are
> nothing
> > > but messages which are upto the high watermark.
> > > *The high watermark is the offset of the last message that was
> > successfully
> > > copied to all of the log’s replicas. *
> > >
> > > *Having said that, if one of the replica is down, will high water mark
> > be*
> > > *advanced?*
> > >
> > > *If replica can't come forever, can we consider this message cant be
> > > consumed by the consumer since it is never committed *
> > >
> >
>
Re: Highwater mark interpretation
Posted by D C <dr...@gmail.com>.
Hey Nag Y,
I’m not exactly sure if reducing the replication factor while a broker is
down would release the messages to be consumed (or at least not on all
partitions) for the simple fact that it might just remove the last replica
in the list which might not mach your unreachable broker.
Personally i would go and do a manual reassignment of partitions (kafka
manager allows you to do that in an easy visual environment) and move the
replicas out of the broken broker to a working one and once that’s done and
the data copied to the new broker the high watermark should go up as all
the replicas will be in sync.
Cheers,
D
On Sunday, June 21, 2020, Nag Y <an...@gmail.com> wrote:
> Thanks D C. Thanks a lot . That is quite a detailed explanation.
> If I understand correctly, ( ignoring the case where producers
> create transactions) - since the replica is down and never comes , the high
> watermark CANNOT advance and the consumer CAN NOT read the messages which
> were sent after the replica is down as the message is NOT committed - Hope
> this is correct ?
——————————
Indeed, this is correct.
——————————
>
> To address this situation, either we should make sure the replica is up or
> reduce the replication factor so that the message will be committed and
> consumer can start reading the messages ...
>
> Regards,
> Nag
>
>
> On Sun, Jun 21, 2020 at 3:25 AM D C <dr...@gmail.com> wrote:
>
> > The short answer is : yes, a consumer can only consume messages up to the
> > High Watermark.
> >
> > The long answer is not exactly, for the following reasons:
> >
> > At the partition level you have 3 major offsets that are important to the
> > health of the partition and accessibility from the consumer pov:
> > LeO (log end offset) - which represents the highest offset in the highest
> > segment
> > High Watermark - which represents the latest offset that has been
> > replicated to all the followers
> > LSO (Last stable offset) - which is important when you use producers that
> > create transactions - which represents the the highest offset that has
> been
> > committed by a transaction and that is allowed to be read with isolation
> > level = read_commited.
> >
> > The LeO can only be higher or equal to the High Watermark (for obvious
> > reasons)
> > The High Watermark can only be higher or equal to the LSO (the messages
> up
> > to this point may have been committed to all the followers but the
> > transaction isn't yet finished)
> > And coming to your question, in case the transaction hasn't finished, the
> > LSO may be lower than the High Watermark so if your consumer is accessing
> > the data in Read_Committed, it won't be able to surpass the LSO.
> >
> > Cheers,
> > D
> >
> > On Sat, Jun 20, 2020 at 9:05 PM Nag Y <an...@gmail.com>
> wrote:
> >
> > > As I understand it, the consumer can only read "committed" messages -
> > which
> > > I believe, if we look at internals of it, committed messages are
> nothing
> > > but messages which are upto the high watermark.
> > > *The high watermark is the offset of the last message that was
> > successfully
> > > copied to all of the log’s replicas. *
> > >
> > > *Having said that, if one of the replica is down, will high water mark
> > be*
> > > *advanced?*
> > >
> > > *If replica can't come forever, can we consider this message cant be
> > > consumed by the consumer since it is never committed *
> > >
> >
>
--
Re: Highwater mark interpretation
Posted by Nag Y <an...@gmail.com>.
Thanks D C. Thanks a lot . That is quite a detailed explanation.
If I understand correctly, ( ignoring the case where producers
create transactions) - since the replica is down and never comes , the high
watermark CANNOT advance and the consumer CAN NOT read the messages which
were sent after the replica is down as the message is NOT committed - Hope
this is correct ?
To address this situation, either we should make sure the replica is up or
reduce the replication factor so that the message will be committed and
consumer can start reading the messages ...
Regards,
Nag
On Sun, Jun 21, 2020 at 3:25 AM D C <dr...@gmail.com> wrote:
> The short answer is : yes, a consumer can only consume messages up to the
> High Watermark.
>
> The long answer is not exactly, for the following reasons:
>
> At the partition level you have 3 major offsets that are important to the
> health of the partition and accessibility from the consumer pov:
> LeO (log end offset) - which represents the highest offset in the highest
> segment
> High Watermark - which represents the latest offset that has been
> replicated to all the followers
> LSO (Last stable offset) - which is important when you use producers that
> create transactions - which represents the the highest offset that has been
> committed by a transaction and that is allowed to be read with isolation
> level = read_commited.
>
> The LeO can only be higher or equal to the High Watermark (for obvious
> reasons)
> The High Watermark can only be higher or equal to the LSO (the messages up
> to this point may have been committed to all the followers but the
> transaction isn't yet finished)
> And coming to your question, in case the transaction hasn't finished, the
> LSO may be lower than the High Watermark so if your consumer is accessing
> the data in Read_Committed, it won't be able to surpass the LSO.
>
> Cheers,
> D
>
> On Sat, Jun 20, 2020 at 9:05 PM Nag Y <an...@gmail.com> wrote:
>
> > As I understand it, the consumer can only read "committed" messages -
> which
> > I believe, if we look at internals of it, committed messages are nothing
> > but messages which are upto the high watermark.
> > *The high watermark is the offset of the last message that was
> successfully
> > copied to all of the log’s replicas. *
> >
> > *Having said that, if one of the replica is down, will high water mark
> be*
> > *advanced?*
> >
> > *If replica can't come forever, can we consider this message cant be
> > consumed by the consumer since it is never committed *
> >
>
Re: Highwater mark interpretation
Posted by Nag Y <an...@gmail.com>.
Thanks D C. Thanks a lot . That is quite a detailed explanation.
If I understand correctly, ( ignoring the case where producers
create transactions) - since the replica is down and never comes , the high
watermark CANNOT advance and the consumer CAN NOT read the messages which
were sent after the replica is down as the message is NOT committed - Hope
this is correct ?
To address this situation, either we should make sure the replica is up or
reduce the replication factor so that the message will be committed and
consumer can start reading the messages ...
Regards,
Nag
On Sun, Jun 21, 2020 at 3:25 AM D C <dr...@gmail.com> wrote:
> The short answer is : yes, a consumer can only consume messages up to the
> High Watermark.
>
> The long answer is not exactly, for the following reasons:
>
> At the partition level you have 3 major offsets that are important to the
> health of the partition and accessibility from the consumer pov:
> LeO (log end offset) - which represents the highest offset in the highest
> segment
> High Watermark - which represents the latest offset that has been
> replicated to all the followers
> LSO (Last stable offset) - which is important when you use producers that
> create transactions - which represents the the highest offset that has been
> committed by a transaction and that is allowed to be read with isolation
> level = read_commited.
>
> The LeO can only be higher or equal to the High Watermark (for obvious
> reasons)
> The High Watermark can only be higher or equal to the LSO (the messages up
> to this point may have been committed to all the followers but the
> transaction isn't yet finished)
> And coming to your question, in case the transaction hasn't finished, the
> LSO may be lower than the High Watermark so if your consumer is accessing
> the data in Read_Committed, it won't be able to surpass the LSO.
>
> Cheers,
> D
>
> On Sat, Jun 20, 2020 at 9:05 PM Nag Y <an...@gmail.com> wrote:
>
> > As I understand it, the consumer can only read "committed" messages -
> which
> > I believe, if we look at internals of it, committed messages are nothing
> > but messages which are upto the high watermark.
> > *The high watermark is the offset of the last message that was
> successfully
> > copied to all of the log’s replicas. *
> >
> > *Having said that, if one of the replica is down, will high water mark
> be*
> > *advanced?*
> >
> > *If replica can't come forever, can we consider this message cant be
> > consumed by the consumer since it is never committed *
> >
>
Re: Highwater mark interpretation
Posted by D C <dr...@gmail.com>.
The short answer is : yes, a consumer can only consume messages up to the
High Watermark.
The long answer is not exactly, for the following reasons:
At the partition level you have 3 major offsets that are important to the
health of the partition and accessibility from the consumer pov:
LeO (log end offset) - which represents the highest offset in the highest
segment
High Watermark - which represents the latest offset that has been
replicated to all the followers
LSO (Last stable offset) - which is important when you use producers that
create transactions - which represents the the highest offset that has been
committed by a transaction and that is allowed to be read with isolation
level = read_commited.
The LeO can only be higher or equal to the High Watermark (for obvious
reasons)
The High Watermark can only be higher or equal to the LSO (the messages up
to this point may have been committed to all the followers but the
transaction isn't yet finished)
And coming to your question, in case the transaction hasn't finished, the
LSO may be lower than the High Watermark so if your consumer is accessing
the data in Read_Committed, it won't be able to surpass the LSO.
Cheers,
D
On Sat, Jun 20, 2020 at 9:05 PM Nag Y <an...@gmail.com> wrote:
> As I understand it, the consumer can only read "committed" messages - which
> I believe, if we look at internals of it, committed messages are nothing
> but messages which are upto the high watermark.
> *The high watermark is the offset of the last message that was successfully
> copied to all of the log’s replicas. *
>
> *Having said that, if one of the replica is down, will high water mark be*
> *advanced?*
>
> *If replica can't come forever, can we consider this message cant be
> consumed by the consumer since it is never committed *
>
Re: Highwater mark interpretation
Posted by D C <dr...@gmail.com>.
The short answer is : yes, a consumer can only consume messages up to the
High Watermark.
The long answer is not exactly, for the following reasons:
At the partition level you have 3 major offsets that are important to the
health of the partition and accessibility from the consumer pov:
LeO (log end offset) - which represents the highest offset in the highest
segment
High Watermark - which represents the latest offset that has been
replicated to all the followers
LSO (Last stable offset) - which is important when you use producers that
create transactions - which represents the the highest offset that has been
committed by a transaction and that is allowed to be read with isolation
level = read_commited.
The LeO can only be higher or equal to the High Watermark (for obvious
reasons)
The High Watermark can only be higher or equal to the LSO (the messages up
to this point may have been committed to all the followers but the
transaction isn't yet finished)
And coming to your question, in case the transaction hasn't finished, the
LSO may be lower than the High Watermark so if your consumer is accessing
the data in Read_Committed, it won't be able to surpass the LSO.
Cheers,
D
On Sat, Jun 20, 2020 at 9:05 PM Nag Y <an...@gmail.com> wrote:
> As I understand it, the consumer can only read "committed" messages - which
> I believe, if we look at internals of it, committed messages are nothing
> but messages which are upto the high watermark.
> *The high watermark is the offset of the last message that was successfully
> copied to all of the log’s replicas. *
>
> *Having said that, if one of the replica is down, will high water mark be*
> *advanced?*
>
> *If replica can't come forever, can we consider this message cant be
> consumed by the consumer since it is never committed *
>