You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Robert Metzger <rm...@apache.org> on 2016/01/20 12:09:09 UTC
SimpleConsumer.getOffsetsBefore() in 0.9 KafkaConsumer
Hi,
I'm currently looking into implementing a load shedding strategy into
Flink's Kafka consumer.
Therefore, I would like to allow users to request the latest offset of the
subscribed TopicPartitions, so that they can
a) determine the lag
b) maybe set the next fetch offset to the latest offset (or anything in
between)
With the SimpleConsumer, there is the getOffsetsBefore() method, which
allows to request the largest offset.
For the new KafkaConsumer, I only found the seekToEnd() and
seekToBeginning() methods, but they don't allow to get the actual end or
beginning.
Are there plans to add such an API to the KafkaConsumer again?
Can you suggest any workarounds?
Regards,
Robert
Re: SimpleConsumer.getOffsetsBefore() in 0.9 KafkaConsumer
Posted by Robert Metzger <rm...@apache.org>.
Thank you for your replies Gwen and Jason.
Lets continue the discussion in the JIRA.
On Fri, Jan 22, 2016 at 8:37 PM, Jason Gustafson <ja...@confluent.io> wrote:
> The offset API is one of the known gaps in the new consumer. There is a
> JIRA (KAFKA-1332), but we might need a KIP to make that change now that the
> API is released. For now, Gwen's suggestion is the only way to do it.
>
> -Jason
>
> On Thu, Jan 21, 2016 at 8:22 PM, Gwen Shapira <gw...@confluent.io> wrote:
>
> > Hi Robert!
> >
> > Jason is the expert, and I hope he'll respond soon.
> >
> > Meanwhile: I think that you can do what you are trying to do by:
> > 1. call position() to get the current position you are consuming
> > 2. call seekToEnd() and then position(), which will give you the last
> > position at the point in which you called seekToEnd()
> > 3. Use another seek() call to go to the offset you really want to get to.
> >
> > I don't think there are current plans to add getOffsetsBefore, but maybe
> we
> > need it for the use-case you specified.
> > I think the developer mailing list (or a JIRA) will be a better place for
> > an API discussion.
> >
> > Gwen
> >
> > On Wed, Jan 20, 2016 at 3:09 AM, Robert Metzger <rm...@apache.org>
> > wrote:
> >
> > > Hi,
> > >
> > > I'm currently looking into implementing a load shedding strategy into
> > > Flink's Kafka consumer.
> > >
> > > Therefore, I would like to allow users to request the latest offset of
> > the
> > > subscribed TopicPartitions, so that they can
> > > a) determine the lag
> > > b) maybe set the next fetch offset to the latest offset (or anything in
> > > between)
> > >
> > > With the SimpleConsumer, there is the getOffsetsBefore() method, which
> > > allows to request the largest offset.
> > > For the new KafkaConsumer, I only found the seekToEnd() and
> > > seekToBeginning() methods, but they don't allow to get the actual end
> or
> > > beginning.
> > >
> > >
> > > Are there plans to add such an API to the KafkaConsumer again?
> > > Can you suggest any workarounds?
> > >
> > > Regards,
> > > Robert
> > >
> >
>
Re: SimpleConsumer.getOffsetsBefore() in 0.9 KafkaConsumer
Posted by Jason Gustafson <ja...@confluent.io>.
The offset API is one of the known gaps in the new consumer. There is a
JIRA (KAFKA-1332), but we might need a KIP to make that change now that the
API is released. For now, Gwen's suggestion is the only way to do it.
-Jason
On Thu, Jan 21, 2016 at 8:22 PM, Gwen Shapira <gw...@confluent.io> wrote:
> Hi Robert!
>
> Jason is the expert, and I hope he'll respond soon.
>
> Meanwhile: I think that you can do what you are trying to do by:
> 1. call position() to get the current position you are consuming
> 2. call seekToEnd() and then position(), which will give you the last
> position at the point in which you called seekToEnd()
> 3. Use another seek() call to go to the offset you really want to get to.
>
> I don't think there are current plans to add getOffsetsBefore, but maybe we
> need it for the use-case you specified.
> I think the developer mailing list (or a JIRA) will be a better place for
> an API discussion.
>
> Gwen
>
> On Wed, Jan 20, 2016 at 3:09 AM, Robert Metzger <rm...@apache.org>
> wrote:
>
> > Hi,
> >
> > I'm currently looking into implementing a load shedding strategy into
> > Flink's Kafka consumer.
> >
> > Therefore, I would like to allow users to request the latest offset of
> the
> > subscribed TopicPartitions, so that they can
> > a) determine the lag
> > b) maybe set the next fetch offset to the latest offset (or anything in
> > between)
> >
> > With the SimpleConsumer, there is the getOffsetsBefore() method, which
> > allows to request the largest offset.
> > For the new KafkaConsumer, I only found the seekToEnd() and
> > seekToBeginning() methods, but they don't allow to get the actual end or
> > beginning.
> >
> >
> > Are there plans to add such an API to the KafkaConsumer again?
> > Can you suggest any workarounds?
> >
> > Regards,
> > Robert
> >
>
Re: SimpleConsumer.getOffsetsBefore() in 0.9 KafkaConsumer
Posted by Gwen Shapira <gw...@confluent.io>.
Hi Robert!
Jason is the expert, and I hope he'll respond soon.
Meanwhile: I think that you can do what you are trying to do by:
1. call position() to get the current position you are consuming
2. call seekToEnd() and then position(), which will give you the last
position at the point in which you called seekToEnd()
3. Use another seek() call to go to the offset you really want to get to.
I don't think there are current plans to add getOffsetsBefore, but maybe we
need it for the use-case you specified.
I think the developer mailing list (or a JIRA) will be a better place for
an API discussion.
Gwen
On Wed, Jan 20, 2016 at 3:09 AM, Robert Metzger <rm...@apache.org> wrote:
> Hi,
>
> I'm currently looking into implementing a load shedding strategy into
> Flink's Kafka consumer.
>
> Therefore, I would like to allow users to request the latest offset of the
> subscribed TopicPartitions, so that they can
> a) determine the lag
> b) maybe set the next fetch offset to the latest offset (or anything in
> between)
>
> With the SimpleConsumer, there is the getOffsetsBefore() method, which
> allows to request the largest offset.
> For the new KafkaConsumer, I only found the seekToEnd() and
> seekToBeginning() methods, but they don't allow to get the actual end or
> beginning.
>
>
> Are there plans to add such an API to the KafkaConsumer again?
> Can you suggest any workarounds?
>
> Regards,
> Robert
>