You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Yifan Ying <na...@gmail.com> on 2016/10/12 21:26:05 UTC
Manually update consumer offset stored in Kafka
Hi,
In old consumers, we use the following command line tool to manually update
offsets stored in zk:
*./kafka-run-class.sh kafka.tools.UpdateOffsetsInZK [latest | earliest]
[consumer.properties file path] [topic]*
But it doesn't work with offsets stored in Kafka. How can I update the
Kafka offsets to latest?
Yifan
--
Yifan
Re: Manually update consumer offset stored in Kafka
Posted by Yifan Ying <na...@gmail.com>.
Yeah, this works! The consumer API is able to update the consumer offset.
The only downside is to stop the real consumers.
On Fri, Oct 14, 2016 at 2:47 PM, Kevin A <k4...@gmail.com> wrote:
> I didn't find an off-the-shelf tool to do this when I needed to a few weeks
> ago (which was kind of surprising).
>
> I used the kafka-python library (my company's wrappers around it, actually)
> to pretend I was in the consumer group I wanted to update and called commit
> with the offsets I wanted.
>
> First I stopped the real consumers. Then I ran my little script. Then
> re-started the real consumers.
>
> Hope that helps. (Or elicits a more effective response. ;))
>
> On Fri, Oct 14, 2016 at 10:53 AM, Yifan Ying <na...@gmail.com> wrote:
>
> > Hi Jeff,
> >
> > Could you explain how you send messages to __consumer_offsets to
> overwrite
> > offsets? Thanks!
> >
> > Yifan
> >
> > On Fri, Oct 14, 2016 at 9:55 AM, Jeff Widman <je...@netskope.com> wrote:
> >
> > > I also would like to know this.
> > >
> > > Is the solution to just use a console producer against the internal
> > topics
> > > that store the offsets?
> > >
> > > On Wed, Oct 12, 2016 at 2:26 PM, Yifan Ying <na...@gmail.com>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > In old consumers, we use the following command line tool to manually
> > > update
> > > > offsets stored in zk:
> > > >
> > > > *./kafka-run-class.sh kafka.tools.UpdateOffsetsInZK [latest |
> earliest]
> > > > [consumer.properties file path] [topic]*
> > > >
> > > > But it doesn't work with offsets stored in Kafka. How can I update
> the
> > > > Kafka offsets to latest?
> > > >
> > > > Yifan
> > > >
> > > > --
> > > > Yifan
> > > >
> > >
> >
> >
> >
> > --
> > Yifan
> >
>
--
Yifan
Re: Manually update consumer offset stored in Kafka
Posted by Kevin A <k4...@gmail.com>.
I didn't find an off-the-shelf tool to do this when I needed to a few weeks
ago (which was kind of surprising).
I used the kafka-python library (my company's wrappers around it, actually)
to pretend I was in the consumer group I wanted to update and called commit
with the offsets I wanted.
First I stopped the real consumers. Then I ran my little script. Then
re-started the real consumers.
Hope that helps. (Or elicits a more effective response. ;))
On Fri, Oct 14, 2016 at 10:53 AM, Yifan Ying <na...@gmail.com> wrote:
> Hi Jeff,
>
> Could you explain how you send messages to __consumer_offsets to overwrite
> offsets? Thanks!
>
> Yifan
>
> On Fri, Oct 14, 2016 at 9:55 AM, Jeff Widman <je...@netskope.com> wrote:
>
> > I also would like to know this.
> >
> > Is the solution to just use a console producer against the internal
> topics
> > that store the offsets?
> >
> > On Wed, Oct 12, 2016 at 2:26 PM, Yifan Ying <na...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > In old consumers, we use the following command line tool to manually
> > update
> > > offsets stored in zk:
> > >
> > > *./kafka-run-class.sh kafka.tools.UpdateOffsetsInZK [latest | earliest]
> > > [consumer.properties file path] [topic]*
> > >
> > > But it doesn't work with offsets stored in Kafka. How can I update the
> > > Kafka offsets to latest?
> > >
> > > Yifan
> > >
> > > --
> > > Yifan
> > >
> >
>
>
>
> --
> Yifan
>
Re: Re: Manually update consumer offset stored in Kafka
Posted by Yuanjia <yu...@163.com>.
Hi Yifan，
You can try this procedure with kafka0.10, stop consumer group before do that.
consumer.subscribe(Arrays.asList(topic));//consumer "enable.auto.commit" set false
consumer.poll(1000);
consumer.commitSync(offsets);//the offsets is to be updated consumer offset
Yuanjia Li
From: Yifan Ying
Date: 2016-10-15 01:53
To: users
Subject: Re: Manually update consumer offset stored in Kafka
Hi Jeff,
Could you explain how you send messages to __consumer_offsets to overwrite
offsets? Thanks!
Yifan
On Fri, Oct 14, 2016 at 9:55 AM, Jeff Widman <je...@netskope.com> wrote:
> I also would like to know this.
>
> Is the solution to just use a console producer against the internal topics
> that store the offsets?
>
> On Wed, Oct 12, 2016 at 2:26 PM, Yifan Ying <na...@gmail.com> wrote:
>
> > Hi,
> >
> > In old consumers, we use the following command line tool to manually
> update
> > offsets stored in zk:
> >
> > *./kafka-run-class.sh kafka.tools.UpdateOffsetsInZK [latest | earliest]
> > [consumer.properties file path] [topic]*
> >
> > But it doesn't work with offsets stored in Kafka. How can I update the
> > Kafka offsets to latest?
> >
> > Yifan
> >
> > --
> > Yifan
> >
>
--
Yifan
Re: Manually update consumer offset stored in Kafka
Posted by Yifan Ying <na...@gmail.com>.
Hi Jeff,
Could you explain how you send messages to __consumer_offsets to overwrite
offsets? Thanks!
Yifan
On Fri, Oct 14, 2016 at 9:55 AM, Jeff Widman <je...@netskope.com> wrote:
> I also would like to know this.
>
> Is the solution to just use a console producer against the internal topics
> that store the offsets?
>
> On Wed, Oct 12, 2016 at 2:26 PM, Yifan Ying <na...@gmail.com> wrote:
>
> > Hi,
> >
> > In old consumers, we use the following command line tool to manually
> update
> > offsets stored in zk:
> >
> > *./kafka-run-class.sh kafka.tools.UpdateOffsetsInZK [latest | earliest]
> > [consumer.properties file path] [topic]*
> >
> > But it doesn't work with offsets stored in Kafka. How can I update the
> > Kafka offsets to latest?
> >
> > Yifan
> >
> > --
> > Yifan
> >
>
--
Yifan
Re: Manually update consumer offset stored in Kafka
Posted by Jeff Widman <je...@netskope.com>.
I also would like to know this.
Is the solution to just use a console producer against the internal topics
that store the offsets?
On Wed, Oct 12, 2016 at 2:26 PM, Yifan Ying <na...@gmail.com> wrote:
> Hi,
>
> In old consumers, we use the following command line tool to manually update
> offsets stored in zk:
>
> *./kafka-run-class.sh kafka.tools.UpdateOffsetsInZK [latest | earliest]
> [consumer.properties file path] [topic]*
>
> But it doesn't work with offsets stored in Kafka. How can I update the
> Kafka offsets to latest?
>
> Yifan
>
> --
> Yifan
>