You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ali Nazemian <al...@gmail.com> on 2017/11/23 08:02:59 UTC
Kafka 0.10.0.2 reset offset in the new-consumer mode
Hi All,
I am using Kafka 0.10.0.2 and I am not able to upgrade my Kafka version. I
have a situation that after removing Kafka topic, I am getting the
following error in Kafka-Storm Spout client because the offset hasn't been
reset properly. I was wondering how I can reset the offset in the
new-consumer mode with this version of Kafka. Since I am using
new-consumer, Kafka does not maintain offset info in Zookeeper.
2017-11-23 07:55:52.870 o.a.s.k.s.i.OffsetManager [WARN]
topic-partition [indexing-1] has unexpected offset [7259]. Current
committed Offset [8855387]
Regards,
Ali
Re: Kafka 0.10.0.2 reset offset in the new-consumer mode
Posted by Ali Nazemian <al...@gmail.com>.
Thanks, Brett. We will work on writing a client to do this for us, then.
Regards,
Ali
On Thu, Nov 23, 2017 at 11:02 PM, Brett Rann <br...@zendesk.com.invalid>
wrote:
> Ah apologies.
>
> Found the KIP where the one I suggested was added:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 122%3A+Add+Reset+Consumer+Group+Offsets+tooling
>
> And from the discussion thread link, a couple in to the thread:
>
> Currently the only way for an admin to successfully override offsets is if
> either:
> 1. they send an OffsetCommitRequest with generationId -1 and if the group
> is in the Empty state
> 2. they send an OffsetCommitRequest impersonating a member of the group
> with accurate generationId and memberId
>
> You can actually get away with overriding while members of the group are
> live with method 2 by using group information from DescribeGroupsRequest.
>
>
>
> On Thu, Nov 23, 2017 at 9:20 PM, Ali Nazemian <al...@gmail.com>
> wrote:
>
> > Unfortunately, it doesn't have that option in this version of Kafka!
> >
> > On Thu, Nov 23, 2017 at 9:02 PM, Brett Rann <br...@zendesk.com.invalid>
> > wrote:
> >
> > > I don't know about kafka-storm spout, but you could try using
> > > the kafka-consumer-groups.sh cli to reset the offset. It has
> > > a --reset-offsets option.
> > >
> > > On Thu, Nov 23, 2017 at 7:02 PM, Ali Nazemian <al...@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I am using Kafka 0.10.0.2 and I am not able to upgrade my Kafka
> > version.
> > > I
> > > > have a situation that after removing Kafka topic, I am getting the
> > > > following error in Kafka-Storm Spout client because the offset hasn't
> > > been
> > > > reset properly. I was wondering how I can reset the offset in the
> > > > new-consumer mode with this version of Kafka. Since I am using
> > > > new-consumer, Kafka does not maintain offset info in Zookeeper.
> > > >
> > > > 2017-11-23 07:55:52.870 o.a.s.k.s.i.OffsetManager [WARN]
> > > > topic-partition [indexing-1] has unexpected offset [7259]. Current
> > > > committed Offset [8855387]
> > > >
> > > >
> > > > Regards,
> > > > Ali
> > > >
> > >
> >
> >
> >
> > --
> > A.Nazemian
> >
>
>
>
> --
>
> Brett Rann
>
> Senior DevOps Engineer
>
>
> Zendesk International Ltd
>
> 395 Collins Street, Melbourne VIC 3000 Australia
>
> Mobile: +61 (0) 418 826 017
>
--
A.Nazemian
Re: Kafka 0.10.0.2 reset offset in the new-consumer mode
Posted by Brett Rann <br...@zendesk.com.INVALID>.
Ah apologies.
Found the KIP where the one I suggested was added:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling
And from the discussion thread link, a couple in to the thread:
Currently the only way for an admin to successfully override offsets is if
either:
1. they send an OffsetCommitRequest with generationId -1 and if the group
is in the Empty state
2. they send an OffsetCommitRequest impersonating a member of the group
with accurate generationId and memberId
You can actually get away with overriding while members of the group are
live with method 2 by using group information from DescribeGroupsRequest.
On Thu, Nov 23, 2017 at 9:20 PM, Ali Nazemian <al...@gmail.com> wrote:
> Unfortunately, it doesn't have that option in this version of Kafka!
>
> On Thu, Nov 23, 2017 at 9:02 PM, Brett Rann <br...@zendesk.com.invalid>
> wrote:
>
> > I don't know about kafka-storm spout, but you could try using
> > the kafka-consumer-groups.sh cli to reset the offset. It has
> > a --reset-offsets option.
> >
> > On Thu, Nov 23, 2017 at 7:02 PM, Ali Nazemian <al...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I am using Kafka 0.10.0.2 and I am not able to upgrade my Kafka
> version.
> > I
> > > have a situation that after removing Kafka topic, I am getting the
> > > following error in Kafka-Storm Spout client because the offset hasn't
> > been
> > > reset properly. I was wondering how I can reset the offset in the
> > > new-consumer mode with this version of Kafka. Since I am using
> > > new-consumer, Kafka does not maintain offset info in Zookeeper.
> > >
> > > 2017-11-23 07:55:52.870 o.a.s.k.s.i.OffsetManager [WARN]
> > > topic-partition [indexing-1] has unexpected offset [7259]. Current
> > > committed Offset [8855387]
> > >
> > >
> > > Regards,
> > > Ali
> > >
> >
>
>
>
> --
> A.Nazemian
>
--
Brett Rann
Senior DevOps Engineer
Zendesk International Ltd
395 Collins Street, Melbourne VIC 3000 Australia
Mobile: +61 (0) 418 826 017
Re: Kafka 0.10.0.2 reset offset in the new-consumer mode
Posted by Ali Nazemian <al...@gmail.com>.
Unfortunately, it doesn't have that option in this version of Kafka!
On Thu, Nov 23, 2017 at 9:02 PM, Brett Rann <br...@zendesk.com.invalid>
wrote:
> I don't know about kafka-storm spout, but you could try using
> the kafka-consumer-groups.sh cli to reset the offset. It has
> a --reset-offsets option.
>
> On Thu, Nov 23, 2017 at 7:02 PM, Ali Nazemian <al...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > I am using Kafka 0.10.0.2 and I am not able to upgrade my Kafka version.
> I
> > have a situation that after removing Kafka topic, I am getting the
> > following error in Kafka-Storm Spout client because the offset hasn't
> been
> > reset properly. I was wondering how I can reset the offset in the
> > new-consumer mode with this version of Kafka. Since I am using
> > new-consumer, Kafka does not maintain offset info in Zookeeper.
> >
> > 2017-11-23 07:55:52.870 o.a.s.k.s.i.OffsetManager [WARN]
> > topic-partition [indexing-1] has unexpected offset [7259]. Current
> > committed Offset [8855387]
> >
> >
> > Regards,
> > Ali
> >
>
--
A.Nazemian
Re: Kafka 0.10.0.2 reset offset in the new-consumer mode
Posted by Brett Rann <br...@zendesk.com.INVALID>.
I don't know about kafka-storm spout, but you could try using
the kafka-consumer-groups.sh cli to reset the offset. It has
a --reset-offsets option.
On Thu, Nov 23, 2017 at 7:02 PM, Ali Nazemian <al...@gmail.com> wrote:
> Hi All,
>
> I am using Kafka 0.10.0.2 and I am not able to upgrade my Kafka version. I
> have a situation that after removing Kafka topic, I am getting the
> following error in Kafka-Storm Spout client because the offset hasn't been
> reset properly. I was wondering how I can reset the offset in the
> new-consumer mode with this version of Kafka. Since I am using
> new-consumer, Kafka does not maintain offset info in Zookeeper.
>
> 2017-11-23 07:55:52.870 o.a.s.k.s.i.OffsetManager [WARN]
> topic-partition [indexing-1] has unexpected offset [7259]. Current
> committed Offset [8855387]
>
>
> Regards,
> Ali
>