Re: [DISCUSS] KIP-92 - Add per partition lag metrics to KafkaConsumer

[Joel Koshy <jj...@gmail.com>]

LGTM. However, can you comment on the effect of releasing ownership of
partitions after a rebalance? For e.g., should it reset itself to (say) -1?
or removed? This really applies to any per-partition metrics that we intend
to maintain in the consumer.

On Mon, Nov 14, 2016 at 9:35 AM, Becket Qin <be...@gmail.com> wrote:

> Hey Michael,
>
> Thanks for the comments. Exposing the lag on the client side may serve some
> cases a little different from monitoring. For example, one of the use case
> we have was that an application has some high priority and low priority
> topics to consume. They want to switch between consuming from high priority
> and low priority topics alternately based on the lag on the partition. i.e.
> if the consume has already caught up with the log end offset of a high
> priority topic, the application will switch to consume from the low
> priority topics. Otherwise it will continue consuming from the high
> priority topics. We have seen a few other similar use cases that require a
> programmatic access to the lag. Although people can always use
> offsetsForTimes() to get the LEO, but it is more expensive call involving
> an RPC and is a blocking call.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
> On Mon, Nov 14, 2016 at 9:18 AM, Michael Pearce <Mi...@ig.com>
> wrote:
>
> > Should state I have no objections adding this client side, just more a
> > question to why we don't look and propose to add this broker side also.
> >
> > Sent using OWA for iPhone
> > ________________________________________
> > From: Michael Pearce <Mi...@ig.com>
> > Sent: Monday, November 14, 2016 4:58:45 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-92 - Add per partition lag metrics to
> > KafkaConsumer
> >
> > Why do we not look to expose the lag broker side centrally?
> >
> > Eg like burrow.
> >
> > From an operations point it's a lot easier to monitor lag centrally than
> > per application. Also then you'd be able to see lag of consumers not
> alive
> > or stalled.
> >
> > The information if the consumer uses Kafka based or zookeeper offsets is
> > available to the broker.
> > ________________________________________
> > From: Becket Qin <be...@gmail.com>
> > Sent: Sunday, November 13, 2016 4:13:01 AM
> > To: dev@kafka.apache.org
> > Subject: [DISCUSS] KIP-92 - Add per partition lag metrics to
> KafkaConsumer
> >
> > Hi,
> >
> > We created KIP-92 to propose adding per partition lag metrics to
> > KafkaConsumer.
> >
> > The KIP wiki link is the following:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 92+-+Add+per+partition+lag+metrics+to+KafkaConsumer
> >
> > Comments are welcome.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> > The information contained in this email is strictly confidential and for
> > the use of the addressee only, unless otherwise indicated. If you are not
> > the intended recipient, please do not read, copy, use or disclose to
> others
> > this message or any attachment. Please also notify the sender by replying
> > to this email or by telephone (+44(020 7896 0011) and then delete the
> email
> > and any copies of it. Opinions, conclusion (etc) that do not relate to
> the
> > official business of this company shall be understood as neither given
> nor
> > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > registered in England and Wales, company number 04008957) and IG Index
> > Limited (a company registered in England and Wales, company number
> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> > Index Limited (register number 114059) are authorised and regulated by
> the
> > Financial Conduct Authority.
> >
>

Re: [DISCUSS] KIP-92 - Add per partition lag metrics to KafkaConsumer

[Becket Qin <be...@gmail.com>]

Good point. I think it makes sense to remove the corresponding metric if
the consumer does not own a partition anymore. I have updated the KIP.

On Wed, Dec 21, 2016 at 10:24 AM, Joel Koshy <jj...@gmail.com> wrote:

> LGTM. However, can you comment on the effect of releasing ownership of
> partitions after a rebalance? For e.g., should it reset itself to (say) -1?
> or removed? This really applies to any per-partition metrics that we intend
> to maintain in the consumer.
>
> On Mon, Nov 14, 2016 at 9:35 AM, Becket Qin <be...@gmail.com> wrote:
>
> > Hey Michael,
> >
> > Thanks for the comments. Exposing the lag on the client side may serve
> some
> > cases a little different from monitoring. For example, one of the use
> case
> > we have was that an application has some high priority and low priority
> > topics to consume. They want to switch between consuming from high
> priority
> > and low priority topics alternately based on the lag on the partition.
> i.e.
> > if the consume has already caught up with the log end offset of a high
> > priority topic, the application will switch to consume from the low
> > priority topics. Otherwise it will continue consuming from the high
> > priority topics. We have seen a few other similar use cases that require
> a
> > programmatic access to the lag. Although people can always use
> > offsetsForTimes() to get the LEO, but it is more expensive call involving
> > an RPC and is a blocking call.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> >
> > On Mon, Nov 14, 2016 at 9:18 AM, Michael Pearce <Mi...@ig.com>
> > wrote:
> >
> > > Should state I have no objections adding this client side, just more a
> > > question to why we don't look and propose to add this broker side also.
> > >
> > > Sent using OWA for iPhone
> > > ________________________________________
> > > From: Michael Pearce <Mi...@ig.com>
> > > Sent: Monday, November 14, 2016 4:58:45 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-92 - Add per partition lag metrics to
> > > KafkaConsumer
> > >
> > > Why do we not look to expose the lag broker side centrally?
> > >
> > > Eg like burrow.
> > >
> > > From an operations point it's a lot easier to monitor lag centrally
> than
> > > per application. Also then you'd be able to see lag of consumers not
> > alive
> > > or stalled.
> > >
> > > The information if the consumer uses Kafka based or zookeeper offsets
> is
> > > available to the broker.
> > > ________________________________________
> > > From: Becket Qin <be...@gmail.com>
> > > Sent: Sunday, November 13, 2016 4:13:01 AM
> > > To: dev@kafka.apache.org
> > > Subject: [DISCUSS] KIP-92 - Add per partition lag metrics to
> > KafkaConsumer
> > >
> > > Hi,
> > >
> > > We created KIP-92 to propose adding per partition lag metrics to
> > > KafkaConsumer.
> > >
> > > The KIP wiki link is the following:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 92+-+Add+per+partition+lag+metrics+to+KafkaConsumer
> > >
> > > Comments are welcome.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > > The information contained in this email is strictly confidential and
> for
> > > the use of the addressee only, unless otherwise indicated. If you are
> not
> > > the intended recipient, please do not read, copy, use or disclose to
> > others
> > > this message or any attachment. Please also notify the sender by
> replying
> > > to this email or by telephone (+44(020 7896 0011) and then delete the
> > email
> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
> > the
> > > official business of this company shall be understood as neither given
> > nor
> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
> > > registered in England and Wales, company number 04008957) and IG Index
> > > Limited (a company registered in England and Wales, company number
> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
> IG
> > > Index Limited (register number 114059) are authorised and regulated by
> > the
> > > Financial Conduct Authority.
> > >
> >
>