You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Glen Ogilvie <gl...@oss.co.nz> on 2017/03/08 00:08:57 UTC
How to set offset for a consumer in Kafka 0.10.0.X
Hi,
We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure out if I can manually
set a consumer offset, for a specific consumer when that consumer is stopped.
It looks like it used to be done using: kafka.tools.ExportZkOffsets and kafka.tools.ImportZkOffsets
( https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-ConsumerOffsetChecker )
However, if my version they don't work, because they try and read from zookeeper /consumers which is empty.. I think they are old tools.
Does anyone know where in zookeeper, where the current kafka keeps consumer offsets?
Regards
--
Glen Ogilvie
Open Systems Specialists
Level 1, 162 Grafton Road
http://www.oss.co.nz/
Ph: +64 9 984 3000
Mobile: +64 21 684 146
GPG Key: ACED9C17
Re: How to set offset for a consumer in Kafka 0.10.0.X
Posted by Glen Ogilvie <gl...@oss.co.nz>.
Hi Sudev,
My version is pretty much the same as https://github.com/goibibo/woof/tree/master/scripts/offset_commit_manual.
So, little point posting code which does the same thing, also it python, although with a different method of parsing the params to it.
Sorry about the late reply, didn't remember to check this mailing list for a while.
Regards
--
Glen Ogilvie
Open Systems Specialists
Level 1, 162 Grafton Road
http://www.oss.co.nz/
Ph: +64 9 984 3000
Mobile: +64 21 684 146
GPG Key: ACED9C17
From: "Sudev A C" <su...@goibibo.com>
To: "users" <us...@kafka.apache.org>
Sent: Thursday, 9 March, 2017 7:25:21 PM
Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
Hi Glen,
Maybe have a look at this, slightly different version. Not doing anything
more than what you have seen in gist.
You should pass broker url as first parameter and and a json containing
desired offsets as second parameter.
https://github.com/goibibo/woof/tree/master/scripts/offset_commit_manual
As suggested in previous mail it will be good to shut all consumers before
reseting the offset.
Thanks
Sudev
On Thu, 9 Mar 2017 at 11:41 AM, Manikumar <ma...@gmail.com> wrote:
> A tool for reseting consumer group offsets is proposed here:
> KIP-122:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling
>
> KIP is currently in the voting stage.
>
> On Thu, Mar 9, 2017 at 7:35 AM, Jeff Widman <je...@netskope.com> wrote:
>
> > Did you have to do anything different beyond what was already in the
> Gist?
> >
> > I'd be curious to see the code as I've considered putting together a
> small
> > repo of various python scripts I've found useful when working with Kafka
> >
> > On Wed, Mar 8, 2017 at 1:11 PM, Glen Ogilvie <gl...@oss.co.nz>
> > wrote:
> >
> > > Thank you Jeff and Robert.
> > >
> > > I've had success in getting the offset position to seek to a position
> in
> > > the queue, when all other consumers in the group are off.
> > >
> > > Would anyone like me to tidy the tool up enough that it could be
> included
> > > in Kafka or the docs?
> > >
> > > Regards
> > > --
> > > Glen Ogilvie
> > > Open Systems Specialists
> > > Level 1, 162 Grafton Road
> > > http://www.oss.co.nz/
> > >
> > > Ph: +64 9 984 3000
> > > Mobile: +64 21 684 146
> > > GPG Key: ACED9C17
> > >
> > >
> > > From: "Robert Quinlivan" <rq...@signal.co>
> > > To: "users" <us...@kafka.apache.org>
> > > Sent: Thursday, 9 March, 2017 6:42:15 AM
> > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> > >
> > > The best approach would be:
> > > - Have all consumers in your group shut down
> > > - Have an offset reset tool join with the same group name as above
> > > - Offset tool subscribes to all topic-partitions, seeks to the desired
> > > offset, and commits.
> > > - Offset tool shuts down
> > > - Consumers then restart and re-join the consumer group, resuming at
> the
> > > offsets that were last committed for each topic-partition
> > >
> > > On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman <je...@netskope.com>
> wrote:
> > >
> > > > Yeah, that gist looks like it *should* work. I haven't tested it so
> > can't
> > > > guarantee.
> > > >
> > > > On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie <glen.ogilvie@oss.co.nz
> >
> > > > wrote:
> > > >
> > > > > Hi Jeff,
> > > > >
> > > > > Yes, the work I am doing is ops work. Logstash is consuming from
> the
> > > > topic
> > > > > + consumer group, and I don't want it to start at the beginning,
> but
> > > > rather
> > > > > at a specific offset,
> > > > > so setting the offset for the consumer group externally, then
> > starting
> > > up
> > > > > logstash is my goal.
> > > > >
> > > > > I'm still a little unclear as to how to do this.
> > > > >
> > > > > Is this python script: http://pastebin.com/tvxj1wTX
> > > > >
> > > > > The right way to go about getting the offset set to a specific
> value
> > > > > (12345678 in this example) for a specific consumer group?
> > > > >
> > > > > Regards
> > > > > --
> > > > > Glen Ogilvie
> > > > > Open Systems Specialists
> > > > > Level 1, 162 Grafton Road
> > > > > http://www.oss.co.nz/
> > > > >
> > > > > Ph: +64 9 984 3000
> > > > > Mobile: +64 21 684 146
> > > > > GPG Key: ACED9C17
> > > > >
> > > > >
> > > > > From: "Jeff Widman" <je...@netskope.com>
> > > > > To: "users" <us...@kafka.apache.org>
> > > > > Sent: Wednesday, 8 March, 2017 1:41:17 PM
> > > > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> > > > >
> > > > > Offsets for modern kafka consumers are stored in an internal Kafka
> > > topic,
> > > > > so they aren't as easy to change as zookeeper.
> > > > >
> > > > > To set a consumer offset, you need a consumer within a consumer
> group
> > > to
> > > > > call commit() with your explicit offset. If needed, you can create
> a
> > > > dummy
> > > > > consumer and tell it to join an existing consumer group.
> > > > >
> > > > > Take a look at this migration script, especially the part where it
> > > > commits
> > > > > the offset to see how it can work in Scala:
> > > > > https://github.com/apache/kafka/pull/2615/files
> > > > >
> > > > > It's fairly straightforward to do this within most other clients as
> > > well.
> > > > > If you're doing some quick ops work where you don't want to spin up
> > the
> > > > > JVM, then it's fairly easy to do this using kafka-python.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <
> glen.ogilvie@oss.co.nz
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to
> figure
> > > out
> > > > > if
> > > > > > I can manually
> > > > > > set a consumer offset, for a specific consumer when that consumer
> > is
> > > > > > stopped.
> > > > > >
> > > > > > It looks like it used to be done using:
> kafka.tools.ExportZkOffsets
> > > and
> > > > > > kafka.tools.ImportZkOffsets
> > > > > > ( https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > System+Tools#SystemTools-ConsumerOffsetChecker )
> > > > > >
> > > > > > However, if my version they don't work, because they try and read
> > > from
> > > > > > zookeeper /consumers which is empty.. I think they are old tools.
> > > > > >
> > > > > > Does anyone know where in zookeeper, where the current kafka
> keeps
> > > > > > consumer offsets?
> > > > > >
> > > > > > Regards
> > > > > > --
> > > > > > Glen Ogilvie
> > > > > > Open Systems Specialists
> > > > > > Level 1, 162 Grafton Road
> > > > > > http://www.oss.co.nz/
> > > > > >
> > > > > > Ph: +64 9 984 3000
> > > > > > Mobile: +64 21 684 146
> > > > > > GPG Key: ACED9C17
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Robert Quinlivan
> > > Software Engineer, Signal
> > >
> >
>
Re: How to set offset for a consumer in Kafka 0.10.0.X
Posted by Sudev A C <su...@goibibo.com>.
Hi Glen,
Maybe have a look at this, slightly different version. Not doing anything
more than what you have seen in gist.
You should pass broker url as first parameter and and a json containing
desired offsets as second parameter.
https://github.com/goibibo/woof/tree/master/scripts/offset_commit_manual
As suggested in previous mail it will be good to shut all consumers before
reseting the offset.
Thanks
Sudev
On Thu, 9 Mar 2017 at 11:41 AM, Manikumar <ma...@gmail.com> wrote:
> A tool for reseting consumer group offsets is proposed here:
> KIP-122:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling
>
> KIP is currently in the voting stage.
>
> On Thu, Mar 9, 2017 at 7:35 AM, Jeff Widman <je...@netskope.com> wrote:
>
> > Did you have to do anything different beyond what was already in the
> Gist?
> >
> > I'd be curious to see the code as I've considered putting together a
> small
> > repo of various python scripts I've found useful when working with Kafka
> >
> > On Wed, Mar 8, 2017 at 1:11 PM, Glen Ogilvie <gl...@oss.co.nz>
> > wrote:
> >
> > > Thank you Jeff and Robert.
> > >
> > > I've had success in getting the offset position to seek to a position
> in
> > > the queue, when all other consumers in the group are off.
> > >
> > > Would anyone like me to tidy the tool up enough that it could be
> included
> > > in Kafka or the docs?
> > >
> > > Regards
> > > --
> > > Glen Ogilvie
> > > Open Systems Specialists
> > > Level 1, 162 Grafton Road
> > > http://www.oss.co.nz/
> > >
> > > Ph: +64 9 984 3000
> > > Mobile: +64 21 684 146
> > > GPG Key: ACED9C17
> > >
> > >
> > > From: "Robert Quinlivan" <rq...@signal.co>
> > > To: "users" <us...@kafka.apache.org>
> > > Sent: Thursday, 9 March, 2017 6:42:15 AM
> > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> > >
> > > The best approach would be:
> > > - Have all consumers in your group shut down
> > > - Have an offset reset tool join with the same group name as above
> > > - Offset tool subscribes to all topic-partitions, seeks to the desired
> > > offset, and commits.
> > > - Offset tool shuts down
> > > - Consumers then restart and re-join the consumer group, resuming at
> the
> > > offsets that were last committed for each topic-partition
> > >
> > > On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman <je...@netskope.com>
> wrote:
> > >
> > > > Yeah, that gist looks like it *should* work. I haven't tested it so
> > can't
> > > > guarantee.
> > > >
> > > > On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie <glen.ogilvie@oss.co.nz
> >
> > > > wrote:
> > > >
> > > > > Hi Jeff,
> > > > >
> > > > > Yes, the work I am doing is ops work. Logstash is consuming from
> the
> > > > topic
> > > > > + consumer group, and I don't want it to start at the beginning,
> but
> > > > rather
> > > > > at a specific offset,
> > > > > so setting the offset for the consumer group externally, then
> > starting
> > > up
> > > > > logstash is my goal.
> > > > >
> > > > > I'm still a little unclear as to how to do this.
> > > > >
> > > > > Is this python script: http://pastebin.com/tvxj1wTX
> > > > >
> > > > > The right way to go about getting the offset set to a specific
> value
> > > > > (12345678 in this example) for a specific consumer group?
> > > > >
> > > > > Regards
> > > > > --
> > > > > Glen Ogilvie
> > > > > Open Systems Specialists
> > > > > Level 1, 162 Grafton Road
> > > > > http://www.oss.co.nz/
> > > > >
> > > > > Ph: +64 9 984 3000
> > > > > Mobile: +64 21 684 146
> > > > > GPG Key: ACED9C17
> > > > >
> > > > >
> > > > > From: "Jeff Widman" <je...@netskope.com>
> > > > > To: "users" <us...@kafka.apache.org>
> > > > > Sent: Wednesday, 8 March, 2017 1:41:17 PM
> > > > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> > > > >
> > > > > Offsets for modern kafka consumers are stored in an internal Kafka
> > > topic,
> > > > > so they aren't as easy to change as zookeeper.
> > > > >
> > > > > To set a consumer offset, you need a consumer within a consumer
> group
> > > to
> > > > > call commit() with your explicit offset. If needed, you can create
> a
> > > > dummy
> > > > > consumer and tell it to join an existing consumer group.
> > > > >
> > > > > Take a look at this migration script, especially the part where it
> > > > commits
> > > > > the offset to see how it can work in Scala:
> > > > > https://github.com/apache/kafka/pull/2615/files
> > > > >
> > > > > It's fairly straightforward to do this within most other clients as
> > > well.
> > > > > If you're doing some quick ops work where you don't want to spin up
> > the
> > > > > JVM, then it's fairly easy to do this using kafka-python.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <
> glen.ogilvie@oss.co.nz
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to
> figure
> > > out
> > > > > if
> > > > > > I can manually
> > > > > > set a consumer offset, for a specific consumer when that consumer
> > is
> > > > > > stopped.
> > > > > >
> > > > > > It looks like it used to be done using:
> kafka.tools.ExportZkOffsets
> > > and
> > > > > > kafka.tools.ImportZkOffsets
> > > > > > ( https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > System+Tools#SystemTools-ConsumerOffsetChecker )
> > > > > >
> > > > > > However, if my version they don't work, because they try and read
> > > from
> > > > > > zookeeper /consumers which is empty.. I think they are old tools.
> > > > > >
> > > > > > Does anyone know where in zookeeper, where the current kafka
> keeps
> > > > > > consumer offsets?
> > > > > >
> > > > > > Regards
> > > > > > --
> > > > > > Glen Ogilvie
> > > > > > Open Systems Specialists
> > > > > > Level 1, 162 Grafton Road
> > > > > > http://www.oss.co.nz/
> > > > > >
> > > > > > Ph: +64 9 984 3000
> > > > > > Mobile: +64 21 684 146
> > > > > > GPG Key: ACED9C17
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Robert Quinlivan
> > > Software Engineer, Signal
> > >
> >
>
Re: How to set offset for a consumer in Kafka 0.10.0.X
Posted by Manikumar <ma...@gmail.com>.
A tool for reseting consumer group offsets is proposed here:
KIP-122:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling
KIP is currently in the voting stage.
On Thu, Mar 9, 2017 at 7:35 AM, Jeff Widman <je...@netskope.com> wrote:
> Did you have to do anything different beyond what was already in the Gist?
>
> I'd be curious to see the code as I've considered putting together a small
> repo of various python scripts I've found useful when working with Kafka
>
> On Wed, Mar 8, 2017 at 1:11 PM, Glen Ogilvie <gl...@oss.co.nz>
> wrote:
>
> > Thank you Jeff and Robert.
> >
> > I've had success in getting the offset position to seek to a position in
> > the queue, when all other consumers in the group are off.
> >
> > Would anyone like me to tidy the tool up enough that it could be included
> > in Kafka or the docs?
> >
> > Regards
> > --
> > Glen Ogilvie
> > Open Systems Specialists
> > Level 1, 162 Grafton Road
> > http://www.oss.co.nz/
> >
> > Ph: +64 9 984 3000
> > Mobile: +64 21 684 146
> > GPG Key: ACED9C17
> >
> >
> > From: "Robert Quinlivan" <rq...@signal.co>
> > To: "users" <us...@kafka.apache.org>
> > Sent: Thursday, 9 March, 2017 6:42:15 AM
> > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> >
> > The best approach would be:
> > - Have all consumers in your group shut down
> > - Have an offset reset tool join with the same group name as above
> > - Offset tool subscribes to all topic-partitions, seeks to the desired
> > offset, and commits.
> > - Offset tool shuts down
> > - Consumers then restart and re-join the consumer group, resuming at the
> > offsets that were last committed for each topic-partition
> >
> > On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman <je...@netskope.com> wrote:
> >
> > > Yeah, that gist looks like it *should* work. I haven't tested it so
> can't
> > > guarantee.
> > >
> > > On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie <gl...@oss.co.nz>
> > > wrote:
> > >
> > > > Hi Jeff,
> > > >
> > > > Yes, the work I am doing is ops work. Logstash is consuming from the
> > > topic
> > > > + consumer group, and I don't want it to start at the beginning, but
> > > rather
> > > > at a specific offset,
> > > > so setting the offset for the consumer group externally, then
> starting
> > up
> > > > logstash is my goal.
> > > >
> > > > I'm still a little unclear as to how to do this.
> > > >
> > > > Is this python script: http://pastebin.com/tvxj1wTX
> > > >
> > > > The right way to go about getting the offset set to a specific value
> > > > (12345678 in this example) for a specific consumer group?
> > > >
> > > > Regards
> > > > --
> > > > Glen Ogilvie
> > > > Open Systems Specialists
> > > > Level 1, 162 Grafton Road
> > > > http://www.oss.co.nz/
> > > >
> > > > Ph: +64 9 984 3000
> > > > Mobile: +64 21 684 146
> > > > GPG Key: ACED9C17
> > > >
> > > >
> > > > From: "Jeff Widman" <je...@netskope.com>
> > > > To: "users" <us...@kafka.apache.org>
> > > > Sent: Wednesday, 8 March, 2017 1:41:17 PM
> > > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> > > >
> > > > Offsets for modern kafka consumers are stored in an internal Kafka
> > topic,
> > > > so they aren't as easy to change as zookeeper.
> > > >
> > > > To set a consumer offset, you need a consumer within a consumer group
> > to
> > > > call commit() with your explicit offset. If needed, you can create a
> > > dummy
> > > > consumer and tell it to join an existing consumer group.
> > > >
> > > > Take a look at this migration script, especially the part where it
> > > commits
> > > > the offset to see how it can work in Scala:
> > > > https://github.com/apache/kafka/pull/2615/files
> > > >
> > > > It's fairly straightforward to do this within most other clients as
> > well.
> > > > If you're doing some quick ops work where you don't want to spin up
> the
> > > > JVM, then it's fairly easy to do this using kafka-python.
> > > >
> > > >
> > > >
> > > > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <glen.ogilvie@oss.co.nz
> >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure
> > out
> > > > if
> > > > > I can manually
> > > > > set a consumer offset, for a specific consumer when that consumer
> is
> > > > > stopped.
> > > > >
> > > > > It looks like it used to be done using: kafka.tools.ExportZkOffsets
> > and
> > > > > kafka.tools.ImportZkOffsets
> > > > > ( https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > System+Tools#SystemTools-ConsumerOffsetChecker )
> > > > >
> > > > > However, if my version they don't work, because they try and read
> > from
> > > > > zookeeper /consumers which is empty.. I think they are old tools.
> > > > >
> > > > > Does anyone know where in zookeeper, where the current kafka keeps
> > > > > consumer offsets?
> > > > >
> > > > > Regards
> > > > > --
> > > > > Glen Ogilvie
> > > > > Open Systems Specialists
> > > > > Level 1, 162 Grafton Road
> > > > > http://www.oss.co.nz/
> > > > >
> > > > > Ph: +64 9 984 3000
> > > > > Mobile: +64 21 684 146
> > > > > GPG Key: ACED9C17
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Robert Quinlivan
> > Software Engineer, Signal
> >
>
Re: How to set offset for a consumer in Kafka 0.10.0.X
Posted by Jeff Widman <je...@netskope.com>.
Did you have to do anything different beyond what was already in the Gist?
I'd be curious to see the code as I've considered putting together a small
repo of various python scripts I've found useful when working with Kafka
On Wed, Mar 8, 2017 at 1:11 PM, Glen Ogilvie <gl...@oss.co.nz> wrote:
> Thank you Jeff and Robert.
>
> I've had success in getting the offset position to seek to a position in
> the queue, when all other consumers in the group are off.
>
> Would anyone like me to tidy the tool up enough that it could be included
> in Kafka or the docs?
>
> Regards
> --
> Glen Ogilvie
> Open Systems Specialists
> Level 1, 162 Grafton Road
> http://www.oss.co.nz/
>
> Ph: +64 9 984 3000
> Mobile: +64 21 684 146
> GPG Key: ACED9C17
>
>
> From: "Robert Quinlivan" <rq...@signal.co>
> To: "users" <us...@kafka.apache.org>
> Sent: Thursday, 9 March, 2017 6:42:15 AM
> Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
>
> The best approach would be:
> - Have all consumers in your group shut down
> - Have an offset reset tool join with the same group name as above
> - Offset tool subscribes to all topic-partitions, seeks to the desired
> offset, and commits.
> - Offset tool shuts down
> - Consumers then restart and re-join the consumer group, resuming at the
> offsets that were last committed for each topic-partition
>
> On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman <je...@netskope.com> wrote:
>
> > Yeah, that gist looks like it *should* work. I haven't tested it so can't
> > guarantee.
> >
> > On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie <gl...@oss.co.nz>
> > wrote:
> >
> > > Hi Jeff,
> > >
> > > Yes, the work I am doing is ops work. Logstash is consuming from the
> > topic
> > > + consumer group, and I don't want it to start at the beginning, but
> > rather
> > > at a specific offset,
> > > so setting the offset for the consumer group externally, then starting
> up
> > > logstash is my goal.
> > >
> > > I'm still a little unclear as to how to do this.
> > >
> > > Is this python script: http://pastebin.com/tvxj1wTX
> > >
> > > The right way to go about getting the offset set to a specific value
> > > (12345678 in this example) for a specific consumer group?
> > >
> > > Regards
> > > --
> > > Glen Ogilvie
> > > Open Systems Specialists
> > > Level 1, 162 Grafton Road
> > > http://www.oss.co.nz/
> > >
> > > Ph: +64 9 984 3000
> > > Mobile: +64 21 684 146
> > > GPG Key: ACED9C17
> > >
> > >
> > > From: "Jeff Widman" <je...@netskope.com>
> > > To: "users" <us...@kafka.apache.org>
> > > Sent: Wednesday, 8 March, 2017 1:41:17 PM
> > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> > >
> > > Offsets for modern kafka consumers are stored in an internal Kafka
> topic,
> > > so they aren't as easy to change as zookeeper.
> > >
> > > To set a consumer offset, you need a consumer within a consumer group
> to
> > > call commit() with your explicit offset. If needed, you can create a
> > dummy
> > > consumer and tell it to join an existing consumer group.
> > >
> > > Take a look at this migration script, especially the part where it
> > commits
> > > the offset to see how it can work in Scala:
> > > https://github.com/apache/kafka/pull/2615/files
> > >
> > > It's fairly straightforward to do this within most other clients as
> well.
> > > If you're doing some quick ops work where you don't want to spin up the
> > > JVM, then it's fairly easy to do this using kafka-python.
> > >
> > >
> > >
> > > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <gl...@oss.co.nz>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure
> out
> > > if
> > > > I can manually
> > > > set a consumer offset, for a specific consumer when that consumer is
> > > > stopped.
> > > >
> > > > It looks like it used to be done using: kafka.tools.ExportZkOffsets
> and
> > > > kafka.tools.ImportZkOffsets
> > > > ( https://cwiki.apache.org/confluence/display/KAFKA/
> > > > System+Tools#SystemTools-ConsumerOffsetChecker )
> > > >
> > > > However, if my version they don't work, because they try and read
> from
> > > > zookeeper /consumers which is empty.. I think they are old tools.
> > > >
> > > > Does anyone know where in zookeeper, where the current kafka keeps
> > > > consumer offsets?
> > > >
> > > > Regards
> > > > --
> > > > Glen Ogilvie
> > > > Open Systems Specialists
> > > > Level 1, 162 Grafton Road
> > > > http://www.oss.co.nz/
> > > >
> > > > Ph: +64 9 984 3000
> > > > Mobile: +64 21 684 146
> > > > GPG Key: ACED9C17
> > > >
> > >
> >
>
>
>
> --
> Robert Quinlivan
> Software Engineer, Signal
>
Re: How to set offset for a consumer in Kafka 0.10.0.X
Posted by Glen Ogilvie <gl...@oss.co.nz>.
Thank you Jeff and Robert.
I've had success in getting the offset position to seek to a position in the queue, when all other consumers in the group are off.
Would anyone like me to tidy the tool up enough that it could be included in Kafka or the docs?
Regards
--
Glen Ogilvie
Open Systems Specialists
Level 1, 162 Grafton Road
http://www.oss.co.nz/
Ph: +64 9 984 3000
Mobile: +64 21 684 146
GPG Key: ACED9C17
From: "Robert Quinlivan" <rq...@signal.co>
To: "users" <us...@kafka.apache.org>
Sent: Thursday, 9 March, 2017 6:42:15 AM
Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
The best approach would be:
- Have all consumers in your group shut down
- Have an offset reset tool join with the same group name as above
- Offset tool subscribes to all topic-partitions, seeks to the desired
offset, and commits.
- Offset tool shuts down
- Consumers then restart and re-join the consumer group, resuming at the
offsets that were last committed for each topic-partition
On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman <je...@netskope.com> wrote:
> Yeah, that gist looks like it *should* work. I haven't tested it so can't
> guarantee.
>
> On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie <gl...@oss.co.nz>
> wrote:
>
> > Hi Jeff,
> >
> > Yes, the work I am doing is ops work. Logstash is consuming from the
> topic
> > + consumer group, and I don't want it to start at the beginning, but
> rather
> > at a specific offset,
> > so setting the offset for the consumer group externally, then starting up
> > logstash is my goal.
> >
> > I'm still a little unclear as to how to do this.
> >
> > Is this python script: http://pastebin.com/tvxj1wTX
> >
> > The right way to go about getting the offset set to a specific value
> > (12345678 in this example) for a specific consumer group?
> >
> > Regards
> > --
> > Glen Ogilvie
> > Open Systems Specialists
> > Level 1, 162 Grafton Road
> > http://www.oss.co.nz/
> >
> > Ph: +64 9 984 3000
> > Mobile: +64 21 684 146
> > GPG Key: ACED9C17
> >
> >
> > From: "Jeff Widman" <je...@netskope.com>
> > To: "users" <us...@kafka.apache.org>
> > Sent: Wednesday, 8 March, 2017 1:41:17 PM
> > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> >
> > Offsets for modern kafka consumers are stored in an internal Kafka topic,
> > so they aren't as easy to change as zookeeper.
> >
> > To set a consumer offset, you need a consumer within a consumer group to
> > call commit() with your explicit offset. If needed, you can create a
> dummy
> > consumer and tell it to join an existing consumer group.
> >
> > Take a look at this migration script, especially the part where it
> commits
> > the offset to see how it can work in Scala:
> > https://github.com/apache/kafka/pull/2615/files
> >
> > It's fairly straightforward to do this within most other clients as well.
> > If you're doing some quick ops work where you don't want to spin up the
> > JVM, then it's fairly easy to do this using kafka-python.
> >
> >
> >
> > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <gl...@oss.co.nz>
> > wrote:
> >
> > > Hi,
> > >
> > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure out
> > if
> > > I can manually
> > > set a consumer offset, for a specific consumer when that consumer is
> > > stopped.
> > >
> > > It looks like it used to be done using: kafka.tools.ExportZkOffsets and
> > > kafka.tools.ImportZkOffsets
> > > ( https://cwiki.apache.org/confluence/display/KAFKA/
> > > System+Tools#SystemTools-ConsumerOffsetChecker )
> > >
> > > However, if my version they don't work, because they try and read from
> > > zookeeper /consumers which is empty.. I think they are old tools.
> > >
> > > Does anyone know where in zookeeper, where the current kafka keeps
> > > consumer offsets?
> > >
> > > Regards
> > > --
> > > Glen Ogilvie
> > > Open Systems Specialists
> > > Level 1, 162 Grafton Road
> > > http://www.oss.co.nz/
> > >
> > > Ph: +64 9 984 3000
> > > Mobile: +64 21 684 146
> > > GPG Key: ACED9C17
> > >
> >
>
--
Robert Quinlivan
Software Engineer, Signal
Re: How to set offset for a consumer in Kafka 0.10.0.X
Posted by Robert Quinlivan <rq...@signal.co>.
The best approach would be:
- Have all consumers in your group shut down
- Have an offset reset tool join with the same group name as above
- Offset tool subscribes to all topic-partitions, seeks to the desired
offset, and commits.
- Offset tool shuts down
- Consumers then restart and re-join the consumer group, resuming at the
offsets that were last committed for each topic-partition
On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman <je...@netskope.com> wrote:
> Yeah, that gist looks like it *should* work. I haven't tested it so can't
> guarantee.
>
> On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie <gl...@oss.co.nz>
> wrote:
>
> > Hi Jeff,
> >
> > Yes, the work I am doing is ops work. Logstash is consuming from the
> topic
> > + consumer group, and I don't want it to start at the beginning, but
> rather
> > at a specific offset,
> > so setting the offset for the consumer group externally, then starting up
> > logstash is my goal.
> >
> > I'm still a little unclear as to how to do this.
> >
> > Is this python script: http://pastebin.com/tvxj1wTX
> >
> > The right way to go about getting the offset set to a specific value
> > (12345678 in this example) for a specific consumer group?
> >
> > Regards
> > --
> > Glen Ogilvie
> > Open Systems Specialists
> > Level 1, 162 Grafton Road
> > http://www.oss.co.nz/
> >
> > Ph: +64 9 984 3000
> > Mobile: +64 21 684 146
> > GPG Key: ACED9C17
> >
> >
> > From: "Jeff Widman" <je...@netskope.com>
> > To: "users" <us...@kafka.apache.org>
> > Sent: Wednesday, 8 March, 2017 1:41:17 PM
> > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
> >
> > Offsets for modern kafka consumers are stored in an internal Kafka topic,
> > so they aren't as easy to change as zookeeper.
> >
> > To set a consumer offset, you need a consumer within a consumer group to
> > call commit() with your explicit offset. If needed, you can create a
> dummy
> > consumer and tell it to join an existing consumer group.
> >
> > Take a look at this migration script, especially the part where it
> commits
> > the offset to see how it can work in Scala:
> > https://github.com/apache/kafka/pull/2615/files
> >
> > It's fairly straightforward to do this within most other clients as well.
> > If you're doing some quick ops work where you don't want to spin up the
> > JVM, then it's fairly easy to do this using kafka-python.
> >
> >
> >
> > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <gl...@oss.co.nz>
> > wrote:
> >
> > > Hi,
> > >
> > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure out
> > if
> > > I can manually
> > > set a consumer offset, for a specific consumer when that consumer is
> > > stopped.
> > >
> > > It looks like it used to be done using: kafka.tools.ExportZkOffsets and
> > > kafka.tools.ImportZkOffsets
> > > ( https://cwiki.apache.org/confluence/display/KAFKA/
> > > System+Tools#SystemTools-ConsumerOffsetChecker )
> > >
> > > However, if my version they don't work, because they try and read from
> > > zookeeper /consumers which is empty.. I think they are old tools.
> > >
> > > Does anyone know where in zookeeper, where the current kafka keeps
> > > consumer offsets?
> > >
> > > Regards
> > > --
> > > Glen Ogilvie
> > > Open Systems Specialists
> > > Level 1, 162 Grafton Road
> > > http://www.oss.co.nz/
> > >
> > > Ph: +64 9 984 3000
> > > Mobile: +64 21 684 146
> > > GPG Key: ACED9C17
> > >
> >
>
--
Robert Quinlivan
Software Engineer, Signal
Re: How to set offset for a consumer in Kafka 0.10.0.X
Posted by Jeff Widman <je...@netskope.com>.
Yeah, that gist looks like it *should* work. I haven't tested it so can't
guarantee.
On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie <gl...@oss.co.nz> wrote:
> Hi Jeff,
>
> Yes, the work I am doing is ops work. Logstash is consuming from the topic
> + consumer group, and I don't want it to start at the beginning, but rather
> at a specific offset,
> so setting the offset for the consumer group externally, then starting up
> logstash is my goal.
>
> I'm still a little unclear as to how to do this.
>
> Is this python script: http://pastebin.com/tvxj1wTX
>
> The right way to go about getting the offset set to a specific value
> (12345678 in this example) for a specific consumer group?
>
> Regards
> --
> Glen Ogilvie
> Open Systems Specialists
> Level 1, 162 Grafton Road
> http://www.oss.co.nz/
>
> Ph: +64 9 984 3000
> Mobile: +64 21 684 146
> GPG Key: ACED9C17
>
>
> From: "Jeff Widman" <je...@netskope.com>
> To: "users" <us...@kafka.apache.org>
> Sent: Wednesday, 8 March, 2017 1:41:17 PM
> Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
>
> Offsets for modern kafka consumers are stored in an internal Kafka topic,
> so they aren't as easy to change as zookeeper.
>
> To set a consumer offset, you need a consumer within a consumer group to
> call commit() with your explicit offset. If needed, you can create a dummy
> consumer and tell it to join an existing consumer group.
>
> Take a look at this migration script, especially the part where it commits
> the offset to see how it can work in Scala:
> https://github.com/apache/kafka/pull/2615/files
>
> It's fairly straightforward to do this within most other clients as well.
> If you're doing some quick ops work where you don't want to spin up the
> JVM, then it's fairly easy to do this using kafka-python.
>
>
>
> On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <gl...@oss.co.nz>
> wrote:
>
> > Hi,
> >
> > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure out
> if
> > I can manually
> > set a consumer offset, for a specific consumer when that consumer is
> > stopped.
> >
> > It looks like it used to be done using: kafka.tools.ExportZkOffsets and
> > kafka.tools.ImportZkOffsets
> > ( https://cwiki.apache.org/confluence/display/KAFKA/
> > System+Tools#SystemTools-ConsumerOffsetChecker )
> >
> > However, if my version they don't work, because they try and read from
> > zookeeper /consumers which is empty.. I think they are old tools.
> >
> > Does anyone know where in zookeeper, where the current kafka keeps
> > consumer offsets?
> >
> > Regards
> > --
> > Glen Ogilvie
> > Open Systems Specialists
> > Level 1, 162 Grafton Road
> > http://www.oss.co.nz/
> >
> > Ph: +64 9 984 3000
> > Mobile: +64 21 684 146
> > GPG Key: ACED9C17
> >
>
Re: How to set offset for a consumer in Kafka 0.10.0.X
Posted by Glen Ogilvie <gl...@oss.co.nz>.
Hi Jeff,
Yes, the work I am doing is ops work. Logstash is consuming from the topic + consumer group, and I don't want it to start at the beginning, but rather at a specific offset,
so setting the offset for the consumer group externally, then starting up logstash is my goal.
I'm still a little unclear as to how to do this.
Is this python script: http://pastebin.com/tvxj1wTX
The right way to go about getting the offset set to a specific value (12345678 in this example) for a specific consumer group?
Regards
--
Glen Ogilvie
Open Systems Specialists
Level 1, 162 Grafton Road
http://www.oss.co.nz/
Ph: +64 9 984 3000
Mobile: +64 21 684 146
GPG Key: ACED9C17
From: "Jeff Widman" <je...@netskope.com>
To: "users" <us...@kafka.apache.org>
Sent: Wednesday, 8 March, 2017 1:41:17 PM
Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X
Offsets for modern kafka consumers are stored in an internal Kafka topic,
so they aren't as easy to change as zookeeper.
To set a consumer offset, you need a consumer within a consumer group to
call commit() with your explicit offset. If needed, you can create a dummy
consumer and tell it to join an existing consumer group.
Take a look at this migration script, especially the part where it commits
the offset to see how it can work in Scala:
https://github.com/apache/kafka/pull/2615/files
It's fairly straightforward to do this within most other clients as well.
If you're doing some quick ops work where you don't want to spin up the
JVM, then it's fairly easy to do this using kafka-python.
On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <gl...@oss.co.nz> wrote:
> Hi,
>
> We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure out if
> I can manually
> set a consumer offset, for a specific consumer when that consumer is
> stopped.
>
> It looks like it used to be done using: kafka.tools.ExportZkOffsets and
> kafka.tools.ImportZkOffsets
> ( https://cwiki.apache.org/confluence/display/KAFKA/
> System+Tools#SystemTools-ConsumerOffsetChecker )
>
> However, if my version they don't work, because they try and read from
> zookeeper /consumers which is empty.. I think they are old tools.
>
> Does anyone know where in zookeeper, where the current kafka keeps
> consumer offsets?
>
> Regards
> --
> Glen Ogilvie
> Open Systems Specialists
> Level 1, 162 Grafton Road
> http://www.oss.co.nz/
>
> Ph: +64 9 984 3000
> Mobile: +64 21 684 146
> GPG Key: ACED9C17
>
Re: How to set offset for a consumer in Kafka 0.10.0.X
Posted by Jeff Widman <je...@netskope.com>.
Offsets for modern kafka consumers are stored in an internal Kafka topic,
so they aren't as easy to change as zookeeper.
To set a consumer offset, you need a consumer within a consumer group to
call commit() with your explicit offset. If needed, you can create a dummy
consumer and tell it to join an existing consumer group.
Take a look at this migration script, especially the part where it commits
the offset to see how it can work in Scala:
https://github.com/apache/kafka/pull/2615/files
It's fairly straightforward to do this within most other clients as well.
If you're doing some quick ops work where you don't want to spin up the
JVM, then it's fairly easy to do this using kafka-python.
On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie <gl...@oss.co.nz> wrote:
> Hi,
>
> We are running Kafka 0.10.0.X, with zookeeper. I'm trying to figure out if
> I can manually
> set a consumer offset, for a specific consumer when that consumer is
> stopped.
>
> It looks like it used to be done using: kafka.tools.ExportZkOffsets and
> kafka.tools.ImportZkOffsets
> ( https://cwiki.apache.org/confluence/display/KAFKA/
> System+Tools#SystemTools-ConsumerOffsetChecker )
>
> However, if my version they don't work, because they try and read from
> zookeeper /consumers which is empty.. I think they are old tools.
>
> Does anyone know where in zookeeper, where the current kafka keeps
> consumer offsets?
>
> Regards
> --
> Glen Ogilvie
> Open Systems Specialists
> Level 1, 162 Grafton Road
> http://www.oss.co.nz/
>
> Ph: +64 9 984 3000
> Mobile: +64 21 684 146
> GPG Key: ACED9C17
>