You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Glen Ogilvie <gl...@oss.co.nz> on 2017/04/10 04:44:49 UTC
Re: How to set offset for a consumer in Kafka 0.10.0.X
Hi Sudev,
My version is pretty much the same as https://github.com/goibibo/woof/tree/master/scripts/offset_commit_manual.
So, little point posting code which does the same thing, also it python, although with a different method of parsing the params to it.
Sorry about the late reply, didn't remember to check this mailing list for a while.
Regards
--
Glen Ogilvie
Open Systems Specialists
Level 1, 162 Grafton Road
http://www.oss.co.nz/
Ph: +64 9 984 3000
Mobile: +64 21 684 146
GPG Key: ACED9C17
From: "Sudev A C" <su...@goibibo.com>
To: "users" <us...@kafka.apache.org>
Sent: Thursday, 9 March, 2017 7:25:21 PM
Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
Hi Glen,
Maybe have a look at this, slightly different version. Not doing anything
more than what you have seen in gist.
You should pass broker url as first parameter and and a json containing
desired offsets as second parameter.
https://github.com/goibibo/woof/tree/master/scripts/offset_commit_manual
As suggested in previous mail it will be good to shut all consumers before
reseting the offset.
Thanks
Sudev
On Thu, 9 Mar 2017 at 11:41 AM, Manikumar <ma...@gmail.com> wrote:
> A tool for reseting consumer group offsets is proposed here:
> KIP-122:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling
>
> KIP is currently in the voting stage.
>
> On Thu, Mar 9, 2017 at 7:35 AM, Jeff Widman <je...@netskope.com> wrote:
>
> > Did you have to do anything different beyond what was already in the
> Gist?
> >
> > I'd be curious to see the code as I've considered putting together a
> small
> > repo of various python scripts I've found useful when working with Kafka
> >
> > On Wed, Mar 8, 2017 at 1:11 PM, Glen Ogilvie <gl...@oss.co.nz>
> > wrote:
> >
> > > Thank you Jeff and Robert.
> > >
> > > I've had success in getting the offset position to seek to a position
> in
> > > the queue, when all other consumers in the group are off.
> > >
> > > Would anyone like me to tidy the tool up enough that it could be
> included
> > > in Kafka or the docs?
> > >
> > > Regards
> > > --
> > > Glen Ogilvie
> > > Open Systems Specialists
> > > Level 1, 162 Grafton Road
> > > http://www.oss.co.nz/
> > >
> > > Ph: +64 9 984 3000
> > > Mobile: +64 21 684 146
> > > GPG Key: ACED9C17
> > >
> > >
> > > From: "Robert Quinlivan" <rq...@signal.co>
> > > To: "users" <us...@kafka.apache.org>
> > > Sent: Thursday, 9 March, 2017 6:42:15 AM
> > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> > >
> > > The best approach would be:
> > > - Have all consumers in your group shut down
> > > - Have an offset reset tool join with the same group name as above
> > > - Offset tool subscribes to all topic-partitions, seeks to the desired
> > > offset, and commits.
> > > - Offset tool shuts down
> > > - Consumers then restart and re-join the consumer group, resuming at
> the
> > > offsets that were last committed for each topic-partition
> > >
> > > On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman <je...@netskope.com>
> wrote:
> > >
> > > > Yeah, that gist looks like it *should* work. I haven't tested it so
> > can't
> > > > guarantee.
> > > >
> > > > On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie <glen.ogilvie@oss.co.nz
> >
> > > > wrote:
> > > >
> > > > > Hi Jeff,
> > > > >
> > > > > Yes, the work I am doing is ops work. Logstash is consuming from
> the
> > > > topic
> > > > > + consumer group, and I don't want it to start at the beginning,
> but
> > > > rather
> > > > > at a specific offset,
> > > > > so setting the offset for the consumer group externally, then
> > starting
> > > up
> > > > > logstash is my goal.
> > > > >
> > > > > I'm still a little unclear as to how to do this.
> > > > >
> > > > > Is this python script: http://pastebin.com/tvxj1wTX
> > > > >
> > > > > The right way to go about getting the offset set to a specific
> value
> > > > > (12345678 in this example) for a specific consumer group?
> > > > >
> > > > > Regards
> > > > > --
> > > > > Glen Ogilvie
> > > > > Open Systems Specialists
> > > > > Level 1, 162 Grafton Road
> > > > > http://www.oss.co.nz/
> > > > >
> > > > > Ph: +64 9 984 3000
> > > > > Mobile: +64 21 684 146
> > > > > GPG Key: ACED9C17
> > > > >
> > > > >
> > > > > From: "Jeff Widman" <je...@netskope.com>
> > > > > To: "users" <us...@kafka.apache.org>
> > > > > Sent: Wednesday, 8 March, 2017 1:41:17 PM
> > > > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> > > > >
> > > > > Offsets for modern kafka consumers are stored in an internal Kafka
> > > topic,
> > > > > so they aren't as easy to change as zookeeper.
> > > > >
> > > > > To set a consumer offset, you need a consumer within a consumer
> group
> > > to
> > > > > call commit() with your explicit offset. If needed, you can create
> a
> > > > dummy
> > > > > consumer and tell it to join an existing consumer group.
> > > > >
> > > > > Take a look at this migration script, especially the part where it
> > > > commits
> > > > > the offset to see how it can work in Scala:
> > > > > https://github.com/apache/kafka/pull/2615/files
> > > > >
> > > > > It's fairly straightforward to do this within most other clients as
> > > well.
> > > > > If you're doing some quick ops work where you don't want to spin up
> > the
> > > > > JVM, then it's fairly easy to do this using kafka-python.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <
> glen.ogilvie@oss.co.nz
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to
> figure
> > > out
> > > > > if
> > > > > > I can manually
> > > > > > set a consumer offset, for a specific consumer when that consumer
> > is
> > > > > > stopped.
> > > > > >
> > > > > > It looks like it used to be done using:
> kafka.tools.ExportZkOffsets
> > > and
> > > > > > kafka.tools.ImportZkOffsets
> > > > > > ( https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > System+Tools#SystemTools-ConsumerOffsetChecker )
> > > > > >
> > > > > > However, if my version they don't work, because they try and read
> > > from
> > > > > > zookeeper /consumers which is empty.. I think they are old tools.
> > > > > >
> > > > > > Does anyone know where in zookeeper, where the current kafka
> keeps
> > > > > > consumer offsets?
> > > > > >
> > > > > > Regards
> > > > > > --
> > > > > > Glen Ogilvie
> > > > > > Open Systems Specialists
> > > > > > Level 1, 162 Grafton Road
> > > > > > http://www.oss.co.nz/
> > > > > >
> > > > > > Ph: +64 9 984 3000
> > > > > > Mobile: +64 21 684 146
> > > > > > GPG Key: ACED9C17
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Robert Quinlivan
> > > Software Engineer, Signal
> > >
> >
>