You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Han JU <ju...@gmail.com> on 2015/12/29 16:11:03 UTC
How to reset a consumer-group's offset in kafka 0.9?
Hello,
For local test purpose I need to frequently reset offset for a consumer
group. In 0.8 I just delete the consumer group's zk node under
`/consumers`. But with the redesign of the 0.9, how could I achieve the
same thing?
Thanks!
--
*JU Han*
Software Engineer @ Teads.tv
+33 0619608888
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Franco Giacosa <fg...@gmail.com>.
Hi,
If you want to reset the consumer offset (I am not sure about the group's
offset) you can use this property in 0.9.0
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
2015-12-29 16:35 GMT+01:00 Han JU <ju...@gmail.com>:
> Hi Stevo,
>
> But by deleting and recreating the topic, do I remove also the messages
> ingested?
> My use case is that I ingest prepared messages once and run consumer tests
> multiple times, between each test run I reset the consumer group's offset
> so that each run starts from the beginning and consumers all the messages.
>
> 2015-12-29 16:19 GMT+01:00 Stevo Slavić <ss...@gmail.com>:
>
> > Have you considered deleting and recreating topic used in test?
> > Once topic is clean, read/poll once - any committed offset should be
> > outside of the range, and consumer should reset offset.
> >
> > On Tue, Dec 29, 2015 at 4:11 PM, Han JU <ju...@gmail.com> wrote:
> >
> > > Hello,
> > >
> > > For local test purpose I need to frequently reset offset for a consumer
> > > group. In 0.8 I just delete the consumer group's zk node under
> > > `/consumers`. But with the redesign of the 0.9, how could I achieve the
> > > same thing?
> > >
> > > Thanks!
> > >
> > > --
> > > *JU Han*
> > >
> > > Software Engineer @ Teads.tv
> > >
> > > +33 0619608888
> > >
> >
>
>
>
> --
> *JU Han*
>
> Software Engineer @ Teads.tv
>
> +33 0619608888
>
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Gwen Shapira <gw...@confluent.io>.
Created this to follow up: https://issues.apache.org/jira/browse/KAFKA-3059
On Mon, Jan 4, 2016 at 9:21 AM, Stevo Slavić <ss...@gmail.com> wrote:
> IMO just command/tool to commit offset for given consumer group and topic
> should work (maybe partition can be optional parameter, if partition is not
> provided commit same offset for all partitions of given topic). Then one
> can commit offset -1 for given consumer group and given topic, effectively
> making offset invalid (outside of range).
>
> On Mon, Jan 4, 2016 at 6:00 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > What you really want is to remove the committed offsets in Kafka servers,
> > not the consumer group registry metadata (e.g. such as partition
> assignment
> > information, etc), which will not be done immediately after the group is
> > removed, but only after the offsets expiration has elapsed.
> >
> > We can add this feature as an admin command in future versions.
> >
> > Guozhang
> >
> >
> > On Mon, Jan 4, 2016 at 2:12 AM, Stevo Slavić <ss...@gmail.com> wrote:
> >
> > > Created related ticket
> https://issues.apache.org/jira/browse/KAFKA-3057
> > > Maybe request for extra operations support/tools could be a separate
> > > ticket.
> > >
> > > On Mon, Jan 4, 2016 at 10:48 AM, Han JU <ju...@gmail.com>
> wrote:
> > >
> > > > Thanks a lot Guozhang!
> > > > So currently there's no way to delete a consumer group in with the
> new
> > > > consumer API? Do you plan to add it in the next versions?
> > > >
> > > >
> > > > 2015-12-31 20:09 GMT+01:00 Guozhang Wang <wa...@gmail.com>:
> > > >
> > > > > Hello Han,
> > > > >
> > > > > 1. As Marko mentioned you can use "seek" in the 0.9 Java consumer
> to
> > > > reset
> > > > > your consuming offsets. Or if you are stopping the consumer between
> > > your
> > > > > test runs you can also commit() with offset 0 before you closing
> your
> > > > > consumer at the end of each test.
> > > > >
> > > > > 2. In the 0.9 Java consumer both the consumer registry information
> > and
> > > > the
> > > > > offsets are stored in Kafka servers instead of on the ZK, you can
> > find
> > > a
> > > > > more detailed description and motivation of this change here:
> > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > > > >
> > > > > So the "AdminUtils.deleteConsumerGroupInZK" will not help in
> removing
> > > the
> > > > > consumer registry information.
> > > > >
> > > > > 3. We have deprecated kafka.tools.ConsumerOffsetChecker in 0.9, see
> > > > > "deprecations in 0.9.0" here:
> > > > >
> > > > > http://kafka.apache.org/documentation.html#upgrade_9_breaking
> > > > >
> > > > > Instead you can try to use bin/kafka-consumer-groups.sh
> > > > > (kafka.admin.ConsumerGroupCommand).
> > > > >
> > > > > Guozhang
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Dec 30, 2015 at 9:10 AM, Han JU <ju...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Marko,
> > > > > >
> > > > > > Yes we're currently using this on our production kafka 0.8. But
> it
> > > does
> > > > > not
> > > > > > seem to work with the new consumer API in 0.9.
> > > > > > To answer my own question about deleting consumer group in new
> > > consumer
> > > > > > API, it seems that it's currently not possible with the new
> > consumer
> > > > API
> > > > > > (there's no delete related method in `AdminClient` of the new
> > > consumer
> > > > > > API).
> > > > > >
> > > > > >
> > > > > > 2015-12-30 17:02 GMT+01:00 Marko Bonaći <
> marko.bonaci@sematext.com
> > >:
> > > > > >
> > > > > > > If you want to monitor offset (ZK or Kafka based), try with
> > > > QuantFind's
> > > > > > > Kafka Offset Monitor.
> > > > > > > If you use Docker, it's easy as:
> > > > > > >
> > > > > > > docker run -p 8080:8080 -e ZK=zk_hostname:2181
> > > > > > > jpodeszwik/kafka-offset-monitor
> > > > > > > and then opening browser to dockerhost:8080.
> > > > > > >
> > > > > > > If not in the Docker mood, use instructions here:
> > > > > > > https://github.com/quantifind/KafkaOffsetMonitor
> > > > > > >
> > > > > > > Marko Bonaći
> > > > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> > > > Management
> > > > > > > Solr & Elasticsearch Support
> > > > > > > Sematext <http://sematext.com/> | Contact
> > > > > > > <http://sematext.com/about/contact.html>
> > > > > > >
> > > > > > > On Wed, Dec 30, 2015 at 12:54 PM, Han JU <
> ju.han.felix@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Thanks guys. The `seek` seems a solution. But it's more
> > > cumbersome
> > > > > than
> > > > > > > in
> > > > > > > > 0.8 because I have to plug in some extra code in my consumer
> > > > > > abstractions
> > > > > > > > rather than simply deleting a zk node.
> > > > > > > > And one more question: where does kafka 0.9 stores the
> > > > consumer-group
> > > > > > > > information? In fact I also tried to delete the consumer
> group
> > > but
> > > > > the
> > > > > > > > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in
> > > 0.9.
> > > > > And
> > > > > > > also
> > > > > > > > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
> > > > --zookeeper
> > > > > > > > localhost:2181 --group group-name` seems broken.
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > 2015-12-29 16:46 GMT+01:00 Marko Bonaći <
> > > marko.bonaci@sematext.com
> > > > >:
> > > > > > > >
> > > > > > > > > I was refering to Dana Powers's answer in the link I posted
> > (to
> > > > > use a
> > > > > > > > > client API). You can find an example here:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> > > > > > > > >
> > > > > > > > > Marko Bonaći
> > > > > > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> > > > > > Management
> > > > > > > > > Solr & Elasticsearch Support
> > > > > > > > > Sematext <http://sematext.com/> | Contact
> > > > > > > > > <http://sematext.com/about/contact.html>
> > > > > > > > >
> > > > > > > > > On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <
> > > sslavic@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Then I guess @Before test, explicitly commit offset of 0.
> > > > > > > > > >
> > > > > > > > > > There doesn't seem to be a tool for committing offset,
> only
> > > for
> > > > > > > > > > checking/fetching current offset (see
> > > > > > > > > > http://kafka.apache.org/documentation.html#operations )
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 29, 2015 at 4:35 PM, Han JU <
> > > > ju.han.felix@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Stevo,
> > > > > > > > > > >
> > > > > > > > > > > But by deleting and recreating the topic, do I remove
> > also
> > > > the
> > > > > > > > messages
> > > > > > > > > > > ingested?
> > > > > > > > > > > My use case is that I ingest prepared messages once and
> > run
> > > > > > > consumer
> > > > > > > > > > tests
> > > > > > > > > > > multiple times, between each test run I reset the
> > consumer
> > > > > > group's
> > > > > > > > > offset
> > > > > > > > > > > so that each run starts from the beginning and
> consumers
> > > all
> > > > > the
> > > > > > > > > > messages.
> > > > > > > > > > >
> > > > > > > > > > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <
> > sslavic@gmail.com
> > > >:
> > > > > > > > > > >
> > > > > > > > > > > > Have you considered deleting and recreating topic
> used
> > in
> > > > > test?
> > > > > > > > > > > > Once topic is clean, read/poll once - any committed
> > > offset
> > > > > > should
> > > > > > > > be
> > > > > > > > > > > > outside of the range, and consumer should reset
> offset.
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU <
> > > > > > ju.han.felix@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hello,
> > > > > > > > > > > > >
> > > > > > > > > > > > > For local test purpose I need to frequently reset
> > > offset
> > > > > for
> > > > > > a
> > > > > > > > > > consumer
> > > > > > > > > > > > > group. In 0.8 I just delete the consumer group's zk
> > > node
> > > > > > under
> > > > > > > > > > > > > `/consumers`. But with the redesign of the 0.9, how
> > > > could I
> > > > > > > > achieve
> > > > > > > > > > the
> > > > > > > > > > > > > same thing?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks!
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > *JU Han*
> > > > > > > > > > > > >
> > > > > > > > > > > > > Software Engineer @ Teads.tv
> > > > > > > > > > > > >
> > > > > > > > > > > > > +33 0619608888
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > *JU Han*
> > > > > > > > > > >
> > > > > > > > > > > Software Engineer @ Teads.tv
> > > > > > > > > > >
> > > > > > > > > > > +33 0619608888
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *JU Han*
> > > > > > > >
> > > > > > > > Software Engineer @ Teads.tv
> > > > > > > >
> > > > > > > > +33 0619608888
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *JU Han*
> > > > > >
> > > > > > Software Engineer @ Teads.tv
> > > > > >
> > > > > > +33 0619608888
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *JU Han*
> > > >
> > > > Software Engineer @ Teads.tv
> > > >
> > > > +33 0619608888
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Stevo Slavić <ss...@gmail.com>.
IMO just command/tool to commit offset for given consumer group and topic
should work (maybe partition can be optional parameter, if partition is not
provided commit same offset for all partitions of given topic). Then one
can commit offset -1 for given consumer group and given topic, effectively
making offset invalid (outside of range).
On Mon, Jan 4, 2016 at 6:00 PM, Guozhang Wang <wa...@gmail.com> wrote:
> What you really want is to remove the committed offsets in Kafka servers,
> not the consumer group registry metadata (e.g. such as partition assignment
> information, etc), which will not be done immediately after the group is
> removed, but only after the offsets expiration has elapsed.
>
> We can add this feature as an admin command in future versions.
>
> Guozhang
>
>
> On Mon, Jan 4, 2016 at 2:12 AM, Stevo Slavić <ss...@gmail.com> wrote:
>
> > Created related ticket https://issues.apache.org/jira/browse/KAFKA-3057
> > Maybe request for extra operations support/tools could be a separate
> > ticket.
> >
> > On Mon, Jan 4, 2016 at 10:48 AM, Han JU <ju...@gmail.com> wrote:
> >
> > > Thanks a lot Guozhang!
> > > So currently there's no way to delete a consumer group in with the new
> > > consumer API? Do you plan to add it in the next versions?
> > >
> > >
> > > 2015-12-31 20:09 GMT+01:00 Guozhang Wang <wa...@gmail.com>:
> > >
> > > > Hello Han,
> > > >
> > > > 1. As Marko mentioned you can use "seek" in the 0.9 Java consumer to
> > > reset
> > > > your consuming offsets. Or if you are stopping the consumer between
> > your
> > > > test runs you can also commit() with offset 0 before you closing your
> > > > consumer at the end of each test.
> > > >
> > > > 2. In the 0.9 Java consumer both the consumer registry information
> and
> > > the
> > > > offsets are stored in Kafka servers instead of on the ZK, you can
> find
> > a
> > > > more detailed description and motivation of this change here:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > > >
> > > > So the "AdminUtils.deleteConsumerGroupInZK" will not help in removing
> > the
> > > > consumer registry information.
> > > >
> > > > 3. We have deprecated kafka.tools.ConsumerOffsetChecker in 0.9, see
> > > > "deprecations in 0.9.0" here:
> > > >
> > > > http://kafka.apache.org/documentation.html#upgrade_9_breaking
> > > >
> > > > Instead you can try to use bin/kafka-consumer-groups.sh
> > > > (kafka.admin.ConsumerGroupCommand).
> > > >
> > > > Guozhang
> > > >
> > > >
> > > >
> > > > On Wed, Dec 30, 2015 at 9:10 AM, Han JU <ju...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Marko,
> > > > >
> > > > > Yes we're currently using this on our production kafka 0.8. But it
> > does
> > > > not
> > > > > seem to work with the new consumer API in 0.9.
> > > > > To answer my own question about deleting consumer group in new
> > consumer
> > > > > API, it seems that it's currently not possible with the new
> consumer
> > > API
> > > > > (there's no delete related method in `AdminClient` of the new
> > consumer
> > > > > API).
> > > > >
> > > > >
> > > > > 2015-12-30 17:02 GMT+01:00 Marko Bonaći <marko.bonaci@sematext.com
> >:
> > > > >
> > > > > > If you want to monitor offset (ZK or Kafka based), try with
> > > QuantFind's
> > > > > > Kafka Offset Monitor.
> > > > > > If you use Docker, it's easy as:
> > > > > >
> > > > > > docker run -p 8080:8080 -e ZK=zk_hostname:2181
> > > > > > jpodeszwik/kafka-offset-monitor
> > > > > > and then opening browser to dockerhost:8080.
> > > > > >
> > > > > > If not in the Docker mood, use instructions here:
> > > > > > https://github.com/quantifind/KafkaOffsetMonitor
> > > > > >
> > > > > > Marko Bonaći
> > > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> > > Management
> > > > > > Solr & Elasticsearch Support
> > > > > > Sematext <http://sematext.com/> | Contact
> > > > > > <http://sematext.com/about/contact.html>
> > > > > >
> > > > > > On Wed, Dec 30, 2015 at 12:54 PM, Han JU <ju.han.felix@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Thanks guys. The `seek` seems a solution. But it's more
> > cumbersome
> > > > than
> > > > > > in
> > > > > > > 0.8 because I have to plug in some extra code in my consumer
> > > > > abstractions
> > > > > > > rather than simply deleting a zk node.
> > > > > > > And one more question: where does kafka 0.9 stores the
> > > consumer-group
> > > > > > > information? In fact I also tried to delete the consumer group
> > but
> > > > the
> > > > > > > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in
> > 0.9.
> > > > And
> > > > > > also
> > > > > > > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
> > > --zookeeper
> > > > > > > localhost:2181 --group group-name` seems broken.
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > 2015-12-29 16:46 GMT+01:00 Marko Bonaći <
> > marko.bonaci@sematext.com
> > > >:
> > > > > > >
> > > > > > > > I was refering to Dana Powers's answer in the link I posted
> (to
> > > > use a
> > > > > > > > client API). You can find an example here:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> > > > > > > >
> > > > > > > > Marko Bonaći
> > > > > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> > > > > Management
> > > > > > > > Solr & Elasticsearch Support
> > > > > > > > Sematext <http://sematext.com/> | Contact
> > > > > > > > <http://sematext.com/about/contact.html>
> > > > > > > >
> > > > > > > > On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <
> > sslavic@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Then I guess @Before test, explicitly commit offset of 0.
> > > > > > > > >
> > > > > > > > > There doesn't seem to be a tool for committing offset, only
> > for
> > > > > > > > > checking/fetching current offset (see
> > > > > > > > > http://kafka.apache.org/documentation.html#operations )
> > > > > > > > >
> > > > > > > > > On Tue, Dec 29, 2015 at 4:35 PM, Han JU <
> > > ju.han.felix@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Stevo,
> > > > > > > > > >
> > > > > > > > > > But by deleting and recreating the topic, do I remove
> also
> > > the
> > > > > > > messages
> > > > > > > > > > ingested?
> > > > > > > > > > My use case is that I ingest prepared messages once and
> run
> > > > > > consumer
> > > > > > > > > tests
> > > > > > > > > > multiple times, between each test run I reset the
> consumer
> > > > > group's
> > > > > > > > offset
> > > > > > > > > > so that each run starts from the beginning and consumers
> > all
> > > > the
> > > > > > > > > messages.
> > > > > > > > > >
> > > > > > > > > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <
> sslavic@gmail.com
> > >:
> > > > > > > > > >
> > > > > > > > > > > Have you considered deleting and recreating topic used
> in
> > > > test?
> > > > > > > > > > > Once topic is clean, read/poll once - any committed
> > offset
> > > > > should
> > > > > > > be
> > > > > > > > > > > outside of the range, and consumer should reset offset.
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU <
> > > > > ju.han.felix@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hello,
> > > > > > > > > > > >
> > > > > > > > > > > > For local test purpose I need to frequently reset
> > offset
> > > > for
> > > > > a
> > > > > > > > > consumer
> > > > > > > > > > > > group. In 0.8 I just delete the consumer group's zk
> > node
> > > > > under
> > > > > > > > > > > > `/consumers`. But with the redesign of the 0.9, how
> > > could I
> > > > > > > achieve
> > > > > > > > > the
> > > > > > > > > > > > same thing?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks!
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > > *JU Han*
> > > > > > > > > > > >
> > > > > > > > > > > > Software Engineer @ Teads.tv
> > > > > > > > > > > >
> > > > > > > > > > > > +33 0619608888
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > *JU Han*
> > > > > > > > > >
> > > > > > > > > > Software Engineer @ Teads.tv
> > > > > > > > > >
> > > > > > > > > > +33 0619608888
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *JU Han*
> > > > > > >
> > > > > > > Software Engineer @ Teads.tv
> > > > > > >
> > > > > > > +33 0619608888
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *JU Han*
> > > > >
> > > > > Software Engineer @ Teads.tv
> > > > >
> > > > > +33 0619608888
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> > >
> > >
> > > --
> > > *JU Han*
> > >
> > > Software Engineer @ Teads.tv
> > >
> > > +33 0619608888
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Guozhang Wang <wa...@gmail.com>.
What you really want is to remove the committed offsets in Kafka servers,
not the consumer group registry metadata (e.g. such as partition assignment
information, etc), which will not be done immediately after the group is
removed, but only after the offsets expiration has elapsed.
We can add this feature as an admin command in future versions.
Guozhang
On Mon, Jan 4, 2016 at 2:12 AM, Stevo Slavić <ss...@gmail.com> wrote:
> Created related ticket https://issues.apache.org/jira/browse/KAFKA-3057
> Maybe request for extra operations support/tools could be a separate
> ticket.
>
> On Mon, Jan 4, 2016 at 10:48 AM, Han JU <ju...@gmail.com> wrote:
>
> > Thanks a lot Guozhang!
> > So currently there's no way to delete a consumer group in with the new
> > consumer API? Do you plan to add it in the next versions?
> >
> >
> > 2015-12-31 20:09 GMT+01:00 Guozhang Wang <wa...@gmail.com>:
> >
> > > Hello Han,
> > >
> > > 1. As Marko mentioned you can use "seek" in the 0.9 Java consumer to
> > reset
> > > your consuming offsets. Or if you are stopping the consumer between
> your
> > > test runs you can also commit() with offset 0 before you closing your
> > > consumer at the end of each test.
> > >
> > > 2. In the 0.9 Java consumer both the consumer registry information and
> > the
> > > offsets are stored in Kafka servers instead of on the ZK, you can find
> a
> > > more detailed description and motivation of this change here:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> > >
> > > So the "AdminUtils.deleteConsumerGroupInZK" will not help in removing
> the
> > > consumer registry information.
> > >
> > > 3. We have deprecated kafka.tools.ConsumerOffsetChecker in 0.9, see
> > > "deprecations in 0.9.0" here:
> > >
> > > http://kafka.apache.org/documentation.html#upgrade_9_breaking
> > >
> > > Instead you can try to use bin/kafka-consumer-groups.sh
> > > (kafka.admin.ConsumerGroupCommand).
> > >
> > > Guozhang
> > >
> > >
> > >
> > > On Wed, Dec 30, 2015 at 9:10 AM, Han JU <ju...@gmail.com>
> wrote:
> > >
> > > > Hi Marko,
> > > >
> > > > Yes we're currently using this on our production kafka 0.8. But it
> does
> > > not
> > > > seem to work with the new consumer API in 0.9.
> > > > To answer my own question about deleting consumer group in new
> consumer
> > > > API, it seems that it's currently not possible with the new consumer
> > API
> > > > (there's no delete related method in `AdminClient` of the new
> consumer
> > > > API).
> > > >
> > > >
> > > > 2015-12-30 17:02 GMT+01:00 Marko Bonaći <ma...@sematext.com>:
> > > >
> > > > > If you want to monitor offset (ZK or Kafka based), try with
> > QuantFind's
> > > > > Kafka Offset Monitor.
> > > > > If you use Docker, it's easy as:
> > > > >
> > > > > docker run -p 8080:8080 -e ZK=zk_hostname:2181
> > > > > jpodeszwik/kafka-offset-monitor
> > > > > and then opening browser to dockerhost:8080.
> > > > >
> > > > > If not in the Docker mood, use instructions here:
> > > > > https://github.com/quantifind/KafkaOffsetMonitor
> > > > >
> > > > > Marko Bonaći
> > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> > Management
> > > > > Solr & Elasticsearch Support
> > > > > Sematext <http://sematext.com/> | Contact
> > > > > <http://sematext.com/about/contact.html>
> > > > >
> > > > > On Wed, Dec 30, 2015 at 12:54 PM, Han JU <ju...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Thanks guys. The `seek` seems a solution. But it's more
> cumbersome
> > > than
> > > > > in
> > > > > > 0.8 because I have to plug in some extra code in my consumer
> > > > abstractions
> > > > > > rather than simply deleting a zk node.
> > > > > > And one more question: where does kafka 0.9 stores the
> > consumer-group
> > > > > > information? In fact I also tried to delete the consumer group
> but
> > > the
> > > > > > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in
> 0.9.
> > > And
> > > > > also
> > > > > > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
> > --zookeeper
> > > > > > localhost:2181 --group group-name` seems broken.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > 2015-12-29 16:46 GMT+01:00 Marko Bonaći <
> marko.bonaci@sematext.com
> > >:
> > > > > >
> > > > > > > I was refering to Dana Powers's answer in the link I posted (to
> > > use a
> > > > > > > client API). You can find an example here:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> > > > > > >
> > > > > > > Marko Bonaći
> > > > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> > > > Management
> > > > > > > Solr & Elasticsearch Support
> > > > > > > Sematext <http://sematext.com/> | Contact
> > > > > > > <http://sematext.com/about/contact.html>
> > > > > > >
> > > > > > > On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <
> sslavic@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Then I guess @Before test, explicitly commit offset of 0.
> > > > > > > >
> > > > > > > > There doesn't seem to be a tool for committing offset, only
> for
> > > > > > > > checking/fetching current offset (see
> > > > > > > > http://kafka.apache.org/documentation.html#operations )
> > > > > > > >
> > > > > > > > On Tue, Dec 29, 2015 at 4:35 PM, Han JU <
> > ju.han.felix@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Stevo,
> > > > > > > > >
> > > > > > > > > But by deleting and recreating the topic, do I remove also
> > the
> > > > > > messages
> > > > > > > > > ingested?
> > > > > > > > > My use case is that I ingest prepared messages once and run
> > > > > consumer
> > > > > > > > tests
> > > > > > > > > multiple times, between each test run I reset the consumer
> > > > group's
> > > > > > > offset
> > > > > > > > > so that each run starts from the beginning and consumers
> all
> > > the
> > > > > > > > messages.
> > > > > > > > >
> > > > > > > > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <sslavic@gmail.com
> >:
> > > > > > > > >
> > > > > > > > > > Have you considered deleting and recreating topic used in
> > > test?
> > > > > > > > > > Once topic is clean, read/poll once - any committed
> offset
> > > > should
> > > > > > be
> > > > > > > > > > outside of the range, and consumer should reset offset.
> > > > > > > > > >
> > > > > > > > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU <
> > > > ju.han.felix@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hello,
> > > > > > > > > > >
> > > > > > > > > > > For local test purpose I need to frequently reset
> offset
> > > for
> > > > a
> > > > > > > > consumer
> > > > > > > > > > > group. In 0.8 I just delete the consumer group's zk
> node
> > > > under
> > > > > > > > > > > `/consumers`. But with the redesign of the 0.9, how
> > could I
> > > > > > achieve
> > > > > > > > the
> > > > > > > > > > > same thing?
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > *JU Han*
> > > > > > > > > > >
> > > > > > > > > > > Software Engineer @ Teads.tv
> > > > > > > > > > >
> > > > > > > > > > > +33 0619608888
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *JU Han*
> > > > > > > > >
> > > > > > > > > Software Engineer @ Teads.tv
> > > > > > > > >
> > > > > > > > > +33 0619608888
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *JU Han*
> > > > > >
> > > > > > Software Engineer @ Teads.tv
> > > > > >
> > > > > > +33 0619608888
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *JU Han*
> > > >
> > > > Software Engineer @ Teads.tv
> > > >
> > > > +33 0619608888
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> >
> > --
> > *JU Han*
> >
> > Software Engineer @ Teads.tv
> >
> > +33 0619608888
> >
>
--
-- Guozhang
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Stevo Slavić <ss...@gmail.com>.
Created related ticket https://issues.apache.org/jira/browse/KAFKA-3057
Maybe request for extra operations support/tools could be a separate ticket.
On Mon, Jan 4, 2016 at 10:48 AM, Han JU <ju...@gmail.com> wrote:
> Thanks a lot Guozhang!
> So currently there's no way to delete a consumer group in with the new
> consumer API? Do you plan to add it in the next versions?
>
>
> 2015-12-31 20:09 GMT+01:00 Guozhang Wang <wa...@gmail.com>:
>
> > Hello Han,
> >
> > 1. As Marko mentioned you can use "seek" in the 0.9 Java consumer to
> reset
> > your consuming offsets. Or if you are stopping the consumer between your
> > test runs you can also commit() with offset 0 before you closing your
> > consumer at the end of each test.
> >
> > 2. In the 0.9 Java consumer both the consumer registry information and
> the
> > offsets are stored in Kafka servers instead of on the ZK, you can find a
> > more detailed description and motivation of this change here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
> >
> > So the "AdminUtils.deleteConsumerGroupInZK" will not help in removing the
> > consumer registry information.
> >
> > 3. We have deprecated kafka.tools.ConsumerOffsetChecker in 0.9, see
> > "deprecations in 0.9.0" here:
> >
> > http://kafka.apache.org/documentation.html#upgrade_9_breaking
> >
> > Instead you can try to use bin/kafka-consumer-groups.sh
> > (kafka.admin.ConsumerGroupCommand).
> >
> > Guozhang
> >
> >
> >
> > On Wed, Dec 30, 2015 at 9:10 AM, Han JU <ju...@gmail.com> wrote:
> >
> > > Hi Marko,
> > >
> > > Yes we're currently using this on our production kafka 0.8. But it does
> > not
> > > seem to work with the new consumer API in 0.9.
> > > To answer my own question about deleting consumer group in new consumer
> > > API, it seems that it's currently not possible with the new consumer
> API
> > > (there's no delete related method in `AdminClient` of the new consumer
> > > API).
> > >
> > >
> > > 2015-12-30 17:02 GMT+01:00 Marko Bonaći <ma...@sematext.com>:
> > >
> > > > If you want to monitor offset (ZK or Kafka based), try with
> QuantFind's
> > > > Kafka Offset Monitor.
> > > > If you use Docker, it's easy as:
> > > >
> > > > docker run -p 8080:8080 -e ZK=zk_hostname:2181
> > > > jpodeszwik/kafka-offset-monitor
> > > > and then opening browser to dockerhost:8080.
> > > >
> > > > If not in the Docker mood, use instructions here:
> > > > https://github.com/quantifind/KafkaOffsetMonitor
> > > >
> > > > Marko Bonaći
> > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> Management
> > > > Solr & Elasticsearch Support
> > > > Sematext <http://sematext.com/> | Contact
> > > > <http://sematext.com/about/contact.html>
> > > >
> > > > On Wed, Dec 30, 2015 at 12:54 PM, Han JU <ju...@gmail.com>
> > wrote:
> > > >
> > > > > Thanks guys. The `seek` seems a solution. But it's more cumbersome
> > than
> > > > in
> > > > > 0.8 because I have to plug in some extra code in my consumer
> > > abstractions
> > > > > rather than simply deleting a zk node.
> > > > > And one more question: where does kafka 0.9 stores the
> consumer-group
> > > > > information? In fact I also tried to delete the consumer group but
> > the
> > > > > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in 0.9.
> > And
> > > > also
> > > > > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
> --zookeeper
> > > > > localhost:2181 --group group-name` seems broken.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > 2015-12-29 16:46 GMT+01:00 Marko Bonaći <marko.bonaci@sematext.com
> >:
> > > > >
> > > > > > I was refering to Dana Powers's answer in the link I posted (to
> > use a
> > > > > > client API). You can find an example here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> > > > > >
> > > > > > Marko Bonaći
> > > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> > > Management
> > > > > > Solr & Elasticsearch Support
> > > > > > Sematext <http://sematext.com/> | Contact
> > > > > > <http://sematext.com/about/contact.html>
> > > > > >
> > > > > > On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <sslavic@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Then I guess @Before test, explicitly commit offset of 0.
> > > > > > >
> > > > > > > There doesn't seem to be a tool for committing offset, only for
> > > > > > > checking/fetching current offset (see
> > > > > > > http://kafka.apache.org/documentation.html#operations )
> > > > > > >
> > > > > > > On Tue, Dec 29, 2015 at 4:35 PM, Han JU <
> ju.han.felix@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Stevo,
> > > > > > > >
> > > > > > > > But by deleting and recreating the topic, do I remove also
> the
> > > > > messages
> > > > > > > > ingested?
> > > > > > > > My use case is that I ingest prepared messages once and run
> > > > consumer
> > > > > > > tests
> > > > > > > > multiple times, between each test run I reset the consumer
> > > group's
> > > > > > offset
> > > > > > > > so that each run starts from the beginning and consumers all
> > the
> > > > > > > messages.
> > > > > > > >
> > > > > > > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <ss...@gmail.com>:
> > > > > > > >
> > > > > > > > > Have you considered deleting and recreating topic used in
> > test?
> > > > > > > > > Once topic is clean, read/poll once - any committed offset
> > > should
> > > > > be
> > > > > > > > > outside of the range, and consumer should reset offset.
> > > > > > > > >
> > > > > > > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU <
> > > ju.han.felix@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hello,
> > > > > > > > > >
> > > > > > > > > > For local test purpose I need to frequently reset offset
> > for
> > > a
> > > > > > > consumer
> > > > > > > > > > group. In 0.8 I just delete the consumer group's zk node
> > > under
> > > > > > > > > > `/consumers`. But with the redesign of the 0.9, how
> could I
> > > > > achieve
> > > > > > > the
> > > > > > > > > > same thing?
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > *JU Han*
> > > > > > > > > >
> > > > > > > > > > Software Engineer @ Teads.tv
> > > > > > > > > >
> > > > > > > > > > +33 0619608888
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > *JU Han*
> > > > > > > >
> > > > > > > > Software Engineer @ Teads.tv
> > > > > > > >
> > > > > > > > +33 0619608888
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *JU Han*
> > > > >
> > > > > Software Engineer @ Teads.tv
> > > > >
> > > > > +33 0619608888
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *JU Han*
> > >
> > > Software Engineer @ Teads.tv
> > >
> > > +33 0619608888
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
>
>
> --
> *JU Han*
>
> Software Engineer @ Teads.tv
>
> +33 0619608888
>
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Han JU <ju...@gmail.com>.
Thanks a lot Guozhang!
So currently there's no way to delete a consumer group in with the new
consumer API? Do you plan to add it in the next versions?
2015-12-31 20:09 GMT+01:00 Guozhang Wang <wa...@gmail.com>:
> Hello Han,
>
> 1. As Marko mentioned you can use "seek" in the 0.9 Java consumer to reset
> your consuming offsets. Or if you are stopping the consumer between your
> test runs you can also commit() with offset 0 before you closing your
> consumer at the end of each test.
>
> 2. In the 0.9 Java consumer both the consumer registry information and the
> offsets are stored in Kafka servers instead of on the ZK, you can find a
> more detailed description and motivation of this change here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
>
> So the "AdminUtils.deleteConsumerGroupInZK" will not help in removing the
> consumer registry information.
>
> 3. We have deprecated kafka.tools.ConsumerOffsetChecker in 0.9, see
> "deprecations in 0.9.0" here:
>
> http://kafka.apache.org/documentation.html#upgrade_9_breaking
>
> Instead you can try to use bin/kafka-consumer-groups.sh
> (kafka.admin.ConsumerGroupCommand).
>
> Guozhang
>
>
>
> On Wed, Dec 30, 2015 at 9:10 AM, Han JU <ju...@gmail.com> wrote:
>
> > Hi Marko,
> >
> > Yes we're currently using this on our production kafka 0.8. But it does
> not
> > seem to work with the new consumer API in 0.9.
> > To answer my own question about deleting consumer group in new consumer
> > API, it seems that it's currently not possible with the new consumer API
> > (there's no delete related method in `AdminClient` of the new consumer
> > API).
> >
> >
> > 2015-12-30 17:02 GMT+01:00 Marko Bonaći <ma...@sematext.com>:
> >
> > > If you want to monitor offset (ZK or Kafka based), try with QuantFind's
> > > Kafka Offset Monitor.
> > > If you use Docker, it's easy as:
> > >
> > > docker run -p 8080:8080 -e ZK=zk_hostname:2181
> > > jpodeszwik/kafka-offset-monitor
> > > and then opening browser to dockerhost:8080.
> > >
> > > If not in the Docker mood, use instructions here:
> > > https://github.com/quantifind/KafkaOffsetMonitor
> > >
> > > Marko Bonaći
> > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > > Solr & Elasticsearch Support
> > > Sematext <http://sematext.com/> | Contact
> > > <http://sematext.com/about/contact.html>
> > >
> > > On Wed, Dec 30, 2015 at 12:54 PM, Han JU <ju...@gmail.com>
> wrote:
> > >
> > > > Thanks guys. The `seek` seems a solution. But it's more cumbersome
> than
> > > in
> > > > 0.8 because I have to plug in some extra code in my consumer
> > abstractions
> > > > rather than simply deleting a zk node.
> > > > And one more question: where does kafka 0.9 stores the consumer-group
> > > > information? In fact I also tried to delete the consumer group but
> the
> > > > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in 0.9.
> And
> > > also
> > > > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper
> > > > localhost:2181 --group group-name` seems broken.
> > > >
> > > > Thanks!
> > > >
> > > > 2015-12-29 16:46 GMT+01:00 Marko Bonaći <ma...@sematext.com>:
> > > >
> > > > > I was refering to Dana Powers's answer in the link I posted (to
> use a
> > > > > client API). You can find an example here:
> > > > >
> > > > >
> > > >
> > >
> >
> http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> > > > >
> > > > > Marko Bonaći
> > > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> > Management
> > > > > Solr & Elasticsearch Support
> > > > > Sematext <http://sematext.com/> | Contact
> > > > > <http://sematext.com/about/contact.html>
> > > > >
> > > > > On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <ss...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Then I guess @Before test, explicitly commit offset of 0.
> > > > > >
> > > > > > There doesn't seem to be a tool for committing offset, only for
> > > > > > checking/fetching current offset (see
> > > > > > http://kafka.apache.org/documentation.html#operations )
> > > > > >
> > > > > > On Tue, Dec 29, 2015 at 4:35 PM, Han JU <ju...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi Stevo,
> > > > > > >
> > > > > > > But by deleting and recreating the topic, do I remove also the
> > > > messages
> > > > > > > ingested?
> > > > > > > My use case is that I ingest prepared messages once and run
> > > consumer
> > > > > > tests
> > > > > > > multiple times, between each test run I reset the consumer
> > group's
> > > > > offset
> > > > > > > so that each run starts from the beginning and consumers all
> the
> > > > > > messages.
> > > > > > >
> > > > > > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <ss...@gmail.com>:
> > > > > > >
> > > > > > > > Have you considered deleting and recreating topic used in
> test?
> > > > > > > > Once topic is clean, read/poll once - any committed offset
> > should
> > > > be
> > > > > > > > outside of the range, and consumer should reset offset.
> > > > > > > >
> > > > > > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU <
> > ju.han.felix@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > >
> > > > > > > > > For local test purpose I need to frequently reset offset
> for
> > a
> > > > > > consumer
> > > > > > > > > group. In 0.8 I just delete the consumer group's zk node
> > under
> > > > > > > > > `/consumers`. But with the redesign of the 0.9, how could I
> > > > achieve
> > > > > > the
> > > > > > > > > same thing?
> > > > > > > > >
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > *JU Han*
> > > > > > > > >
> > > > > > > > > Software Engineer @ Teads.tv
> > > > > > > > >
> > > > > > > > > +33 0619608888
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > *JU Han*
> > > > > > >
> > > > > > > Software Engineer @ Teads.tv
> > > > > > >
> > > > > > > +33 0619608888
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *JU Han*
> > > >
> > > > Software Engineer @ Teads.tv
> > > >
> > > > +33 0619608888
> > > >
> > >
> >
> >
> >
> > --
> > *JU Han*
> >
> > Software Engineer @ Teads.tv
> >
> > +33 0619608888
> >
>
>
>
> --
> -- Guozhang
>
--
*JU Han*
Software Engineer @ Teads.tv
+33 0619608888
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Han,
1. As Marko mentioned you can use "seek" in the 0.9 Java consumer to reset
your consuming offsets. Or if you are stopping the consumer between your
test runs you can also commit() with offset 0 before you closing your
consumer at the end of each test.
2. In the 0.9 Java consumer both the consumer registry information and the
offsets are stored in Kafka servers instead of on the ZK, you can find a
more detailed description and motivation of this change here:
https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Client+Re-Design
So the "AdminUtils.deleteConsumerGroupInZK" will not help in removing the
consumer registry information.
3. We have deprecated kafka.tools.ConsumerOffsetChecker in 0.9, see
"deprecations in 0.9.0" here:
http://kafka.apache.org/documentation.html#upgrade_9_breaking
Instead you can try to use bin/kafka-consumer-groups.sh
(kafka.admin.ConsumerGroupCommand).
Guozhang
On Wed, Dec 30, 2015 at 9:10 AM, Han JU <ju...@gmail.com> wrote:
> Hi Marko,
>
> Yes we're currently using this on our production kafka 0.8. But it does not
> seem to work with the new consumer API in 0.9.
> To answer my own question about deleting consumer group in new consumer
> API, it seems that it's currently not possible with the new consumer API
> (there's no delete related method in `AdminClient` of the new consumer
> API).
>
>
> 2015-12-30 17:02 GMT+01:00 Marko Bonaći <ma...@sematext.com>:
>
> > If you want to monitor offset (ZK or Kafka based), try with QuantFind's
> > Kafka Offset Monitor.
> > If you use Docker, it's easy as:
> >
> > docker run -p 8080:8080 -e ZK=zk_hostname:2181
> > jpodeszwik/kafka-offset-monitor
> > and then opening browser to dockerhost:8080.
> >
> > If not in the Docker mood, use instructions here:
> > https://github.com/quantifind/KafkaOffsetMonitor
> >
> > Marko Bonaći
> > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > Solr & Elasticsearch Support
> > Sematext <http://sematext.com/> | Contact
> > <http://sematext.com/about/contact.html>
> >
> > On Wed, Dec 30, 2015 at 12:54 PM, Han JU <ju...@gmail.com> wrote:
> >
> > > Thanks guys. The `seek` seems a solution. But it's more cumbersome than
> > in
> > > 0.8 because I have to plug in some extra code in my consumer
> abstractions
> > > rather than simply deleting a zk node.
> > > And one more question: where does kafka 0.9 stores the consumer-group
> > > information? In fact I also tried to delete the consumer group but the
> > > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in 0.9. And
> > also
> > > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper
> > > localhost:2181 --group group-name` seems broken.
> > >
> > > Thanks!
> > >
> > > 2015-12-29 16:46 GMT+01:00 Marko Bonaći <ma...@sematext.com>:
> > >
> > > > I was refering to Dana Powers's answer in the link I posted (to use a
> > > > client API). You can find an example here:
> > > >
> > > >
> > >
> >
> http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> > > >
> > > > Marko Bonaći
> > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> Management
> > > > Solr & Elasticsearch Support
> > > > Sematext <http://sematext.com/> | Contact
> > > > <http://sematext.com/about/contact.html>
> > > >
> > > > On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <ss...@gmail.com>
> > wrote:
> > > >
> > > > > Then I guess @Before test, explicitly commit offset of 0.
> > > > >
> > > > > There doesn't seem to be a tool for committing offset, only for
> > > > > checking/fetching current offset (see
> > > > > http://kafka.apache.org/documentation.html#operations )
> > > > >
> > > > > On Tue, Dec 29, 2015 at 4:35 PM, Han JU <ju...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Stevo,
> > > > > >
> > > > > > But by deleting and recreating the topic, do I remove also the
> > > messages
> > > > > > ingested?
> > > > > > My use case is that I ingest prepared messages once and run
> > consumer
> > > > > tests
> > > > > > multiple times, between each test run I reset the consumer
> group's
> > > > offset
> > > > > > so that each run starts from the beginning and consumers all the
> > > > > messages.
> > > > > >
> > > > > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <ss...@gmail.com>:
> > > > > >
> > > > > > > Have you considered deleting and recreating topic used in test?
> > > > > > > Once topic is clean, read/poll once - any committed offset
> should
> > > be
> > > > > > > outside of the range, and consumer should reset offset.
> > > > > > >
> > > > > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU <
> ju.han.felix@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > For local test purpose I need to frequently reset offset for
> a
> > > > > consumer
> > > > > > > > group. In 0.8 I just delete the consumer group's zk node
> under
> > > > > > > > `/consumers`. But with the redesign of the 0.9, how could I
> > > achieve
> > > > > the
> > > > > > > > same thing?
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > --
> > > > > > > > *JU Han*
> > > > > > > >
> > > > > > > > Software Engineer @ Teads.tv
> > > > > > > >
> > > > > > > > +33 0619608888
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *JU Han*
> > > > > >
> > > > > > Software Engineer @ Teads.tv
> > > > > >
> > > > > > +33 0619608888
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *JU Han*
> > >
> > > Software Engineer @ Teads.tv
> > >
> > > +33 0619608888
> > >
> >
>
>
>
> --
> *JU Han*
>
> Software Engineer @ Teads.tv
>
> +33 0619608888
>
--
-- Guozhang
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Marko Bonaći <ma...@sematext.com>.
Hi Han,
if it doesn't work you should file an issue, since it explicitly says in
the readme that it works with:
1 zookeeper built-in high-level consumer (based on Zookeeper)
2 kafka built-in offset management API (based on Kafka internal topic)
3 Storm Kafka Spout (based on Zookeeper by default)
I'm still on Kafka 0.8, so I can't shed any light on your issue.
Thx for the AdminClient info.
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Wed, Dec 30, 2015 at 6:10 PM, Han JU <ju...@gmail.com> wrote:
> Hi Marko,
>
> Yes we're currently using this on our production kafka 0.8. But it does not
> seem to work with the new consumer API in 0.9.
> To answer my own question about deleting consumer group in new consumer
> API, it seems that it's currently not possible with the new consumer API
> (there's no delete related method in `AdminClient` of the new consumer
> API).
>
>
> 2015-12-30 17:02 GMT+01:00 Marko Bonaći <ma...@sematext.com>:
>
> > If you want to monitor offset (ZK or Kafka based), try with QuantFind's
> > Kafka Offset Monitor.
> > If you use Docker, it's easy as:
> >
> > docker run -p 8080:8080 -e ZK=zk_hostname:2181
> > jpodeszwik/kafka-offset-monitor
> > and then opening browser to dockerhost:8080.
> >
> > If not in the Docker mood, use instructions here:
> > https://github.com/quantifind/KafkaOffsetMonitor
> >
> > Marko Bonaći
> > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > Solr & Elasticsearch Support
> > Sematext <http://sematext.com/> | Contact
> > <http://sematext.com/about/contact.html>
> >
> > On Wed, Dec 30, 2015 at 12:54 PM, Han JU <ju...@gmail.com> wrote:
> >
> > > Thanks guys. The `seek` seems a solution. But it's more cumbersome than
> > in
> > > 0.8 because I have to plug in some extra code in my consumer
> abstractions
> > > rather than simply deleting a zk node.
> > > And one more question: where does kafka 0.9 stores the consumer-group
> > > information? In fact I also tried to delete the consumer group but the
> > > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in 0.9. And
> > also
> > > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper
> > > localhost:2181 --group group-name` seems broken.
> > >
> > > Thanks!
> > >
> > > 2015-12-29 16:46 GMT+01:00 Marko Bonaći <ma...@sematext.com>:
> > >
> > > > I was refering to Dana Powers's answer in the link I posted (to use a
> > > > client API). You can find an example here:
> > > >
> > > >
> > >
> >
> http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> > > >
> > > > Marko Bonaći
> > > > Monitoring | Alerting | Anomaly Detection | Centralized Log
> Management
> > > > Solr & Elasticsearch Support
> > > > Sematext <http://sematext.com/> | Contact
> > > > <http://sematext.com/about/contact.html>
> > > >
> > > > On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <ss...@gmail.com>
> > wrote:
> > > >
> > > > > Then I guess @Before test, explicitly commit offset of 0.
> > > > >
> > > > > There doesn't seem to be a tool for committing offset, only for
> > > > > checking/fetching current offset (see
> > > > > http://kafka.apache.org/documentation.html#operations )
> > > > >
> > > > > On Tue, Dec 29, 2015 at 4:35 PM, Han JU <ju...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Stevo,
> > > > > >
> > > > > > But by deleting and recreating the topic, do I remove also the
> > > messages
> > > > > > ingested?
> > > > > > My use case is that I ingest prepared messages once and run
> > consumer
> > > > > tests
> > > > > > multiple times, between each test run I reset the consumer
> group's
> > > > offset
> > > > > > so that each run starts from the beginning and consumers all the
> > > > > messages.
> > > > > >
> > > > > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <ss...@gmail.com>:
> > > > > >
> > > > > > > Have you considered deleting and recreating topic used in test?
> > > > > > > Once topic is clean, read/poll once - any committed offset
> should
> > > be
> > > > > > > outside of the range, and consumer should reset offset.
> > > > > > >
> > > > > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU <
> ju.han.felix@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > For local test purpose I need to frequently reset offset for
> a
> > > > > consumer
> > > > > > > > group. In 0.8 I just delete the consumer group's zk node
> under
> > > > > > > > `/consumers`. But with the redesign of the 0.9, how could I
> > > achieve
> > > > > the
> > > > > > > > same thing?
> > > > > > > >
> > > > > > > > Thanks!
> > > > > > > >
> > > > > > > > --
> > > > > > > > *JU Han*
> > > > > > > >
> > > > > > > > Software Engineer @ Teads.tv
> > > > > > > >
> > > > > > > > +33 0619608888
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > *JU Han*
> > > > > >
> > > > > > Software Engineer @ Teads.tv
> > > > > >
> > > > > > +33 0619608888
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *JU Han*
> > >
> > > Software Engineer @ Teads.tv
> > >
> > > +33 0619608888
> > >
> >
>
>
>
> --
> *JU Han*
>
> Software Engineer @ Teads.tv
>
> +33 0619608888
>
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Han JU <ju...@gmail.com>.
Hi Marko,
Yes we're currently using this on our production kafka 0.8. But it does not
seem to work with the new consumer API in 0.9.
To answer my own question about deleting consumer group in new consumer
API, it seems that it's currently not possible with the new consumer API
(there's no delete related method in `AdminClient` of the new consumer API).
2015-12-30 17:02 GMT+01:00 Marko Bonaći <ma...@sematext.com>:
> If you want to monitor offset (ZK or Kafka based), try with QuantFind's
> Kafka Offset Monitor.
> If you use Docker, it's easy as:
>
> docker run -p 8080:8080 -e ZK=zk_hostname:2181
> jpodeszwik/kafka-offset-monitor
> and then opening browser to dockerhost:8080.
>
> If not in the Docker mood, use instructions here:
> https://github.com/quantifind/KafkaOffsetMonitor
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext <http://sematext.com/> | Contact
> <http://sematext.com/about/contact.html>
>
> On Wed, Dec 30, 2015 at 12:54 PM, Han JU <ju...@gmail.com> wrote:
>
> > Thanks guys. The `seek` seems a solution. But it's more cumbersome than
> in
> > 0.8 because I have to plug in some extra code in my consumer abstractions
> > rather than simply deleting a zk node.
> > And one more question: where does kafka 0.9 stores the consumer-group
> > information? In fact I also tried to delete the consumer group but the
> > `AdminUtils.deleteConsumerGroupInZK` does not seem to work in 0.9. And
> also
> > `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper
> > localhost:2181 --group group-name` seems broken.
> >
> > Thanks!
> >
> > 2015-12-29 16:46 GMT+01:00 Marko Bonaći <ma...@sematext.com>:
> >
> > > I was refering to Dana Powers's answer in the link I posted (to use a
> > > client API). You can find an example here:
> > >
> > >
> >
> http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> > >
> > > Marko Bonaći
> > > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > > Solr & Elasticsearch Support
> > > Sematext <http://sematext.com/> | Contact
> > > <http://sematext.com/about/contact.html>
> > >
> > > On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <ss...@gmail.com>
> wrote:
> > >
> > > > Then I guess @Before test, explicitly commit offset of 0.
> > > >
> > > > There doesn't seem to be a tool for committing offset, only for
> > > > checking/fetching current offset (see
> > > > http://kafka.apache.org/documentation.html#operations )
> > > >
> > > > On Tue, Dec 29, 2015 at 4:35 PM, Han JU <ju...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Stevo,
> > > > >
> > > > > But by deleting and recreating the topic, do I remove also the
> > messages
> > > > > ingested?
> > > > > My use case is that I ingest prepared messages once and run
> consumer
> > > > tests
> > > > > multiple times, between each test run I reset the consumer group's
> > > offset
> > > > > so that each run starts from the beginning and consumers all the
> > > > messages.
> > > > >
> > > > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <ss...@gmail.com>:
> > > > >
> > > > > > Have you considered deleting and recreating topic used in test?
> > > > > > Once topic is clean, read/poll once - any committed offset should
> > be
> > > > > > outside of the range, and consumer should reset offset.
> > > > > >
> > > > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU <ju...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > For local test purpose I need to frequently reset offset for a
> > > > consumer
> > > > > > > group. In 0.8 I just delete the consumer group's zk node under
> > > > > > > `/consumers`. But with the redesign of the 0.9, how could I
> > achieve
> > > > the
> > > > > > > same thing?
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > --
> > > > > > > *JU Han*
> > > > > > >
> > > > > > > Software Engineer @ Teads.tv
> > > > > > >
> > > > > > > +33 0619608888
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > *JU Han*
> > > > >
> > > > > Software Engineer @ Teads.tv
> > > > >
> > > > > +33 0619608888
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > *JU Han*
> >
> > Software Engineer @ Teads.tv
> >
> > +33 0619608888
> >
>
--
*JU Han*
Software Engineer @ Teads.tv
+33 0619608888
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Marko Bonaći <ma...@sematext.com>.
If you want to monitor offset (ZK or Kafka based), try with QuantFind's
Kafka Offset Monitor.
If you use Docker, it's easy as:
docker run -p 8080:8080 -e ZK=zk_hostname:2181
jpodeszwik/kafka-offset-monitor
and then opening browser to dockerhost:8080.
If not in the Docker mood, use instructions here:
https://github.com/quantifind/KafkaOffsetMonitor
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Wed, Dec 30, 2015 at 12:54 PM, Han JU <ju...@gmail.com> wrote:
> Thanks guys. The `seek` seems a solution. But it's more cumbersome than in
> 0.8 because I have to plug in some extra code in my consumer abstractions
> rather than simply deleting a zk node.
> And one more question: where does kafka 0.9 stores the consumer-group
> information? In fact I also tried to delete the consumer group but the
> `AdminUtils.deleteConsumerGroupInZK` does not seem to work in 0.9. And also
> `bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper
> localhost:2181 --group group-name` seems broken.
>
> Thanks!
>
> 2015-12-29 16:46 GMT+01:00 Marko Bonaći <ma...@sematext.com>:
>
> > I was refering to Dana Powers's answer in the link I posted (to use a
> > client API). You can find an example here:
> >
> >
> http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
> >
> > Marko Bonaći
> > Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> > Solr & Elasticsearch Support
> > Sematext <http://sematext.com/> | Contact
> > <http://sematext.com/about/contact.html>
> >
> > On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <ss...@gmail.com> wrote:
> >
> > > Then I guess @Before test, explicitly commit offset of 0.
> > >
> > > There doesn't seem to be a tool for committing offset, only for
> > > checking/fetching current offset (see
> > > http://kafka.apache.org/documentation.html#operations )
> > >
> > > On Tue, Dec 29, 2015 at 4:35 PM, Han JU <ju...@gmail.com>
> wrote:
> > >
> > > > Hi Stevo,
> > > >
> > > > But by deleting and recreating the topic, do I remove also the
> messages
> > > > ingested?
> > > > My use case is that I ingest prepared messages once and run consumer
> > > tests
> > > > multiple times, between each test run I reset the consumer group's
> > offset
> > > > so that each run starts from the beginning and consumers all the
> > > messages.
> > > >
> > > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <ss...@gmail.com>:
> > > >
> > > > > Have you considered deleting and recreating topic used in test?
> > > > > Once topic is clean, read/poll once - any committed offset should
> be
> > > > > outside of the range, and consumer should reset offset.
> > > > >
> > > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU <ju...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > For local test purpose I need to frequently reset offset for a
> > > consumer
> > > > > > group. In 0.8 I just delete the consumer group's zk node under
> > > > > > `/consumers`. But with the redesign of the 0.9, how could I
> achieve
> > > the
> > > > > > same thing?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > --
> > > > > > *JU Han*
> > > > > >
> > > > > > Software Engineer @ Teads.tv
> > > > > >
> > > > > > +33 0619608888
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > *JU Han*
> > > >
> > > > Software Engineer @ Teads.tv
> > > >
> > > > +33 0619608888
> > > >
> > >
> >
>
>
>
> --
> *JU Han*
>
> Software Engineer @ Teads.tv
>
> +33 0619608888
>
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Han JU <ju...@gmail.com>.
Thanks guys. The `seek` seems a solution. But it's more cumbersome than in
0.8 because I have to plug in some extra code in my consumer abstractions
rather than simply deleting a zk node.
And one more question: where does kafka 0.9 stores the consumer-group
information? In fact I also tried to delete the consumer group but the
`AdminUtils.deleteConsumerGroupInZK` does not seem to work in 0.9. And also
`bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper
localhost:2181 --group group-name` seems broken.
Thanks!
2015-12-29 16:46 GMT+01:00 Marko Bonaći <ma...@sematext.com>:
> I was refering to Dana Powers's answer in the link I posted (to use a
> client API). You can find an example here:
>
> http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
>
> Marko Bonaći
> Monitoring | Alerting | Anomaly Detection | Centralized Log Management
> Solr & Elasticsearch Support
> Sematext <http://sematext.com/> | Contact
> <http://sematext.com/about/contact.html>
>
> On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <ss...@gmail.com> wrote:
>
> > Then I guess @Before test, explicitly commit offset of 0.
> >
> > There doesn't seem to be a tool for committing offset, only for
> > checking/fetching current offset (see
> > http://kafka.apache.org/documentation.html#operations )
> >
> > On Tue, Dec 29, 2015 at 4:35 PM, Han JU <ju...@gmail.com> wrote:
> >
> > > Hi Stevo,
> > >
> > > But by deleting and recreating the topic, do I remove also the messages
> > > ingested?
> > > My use case is that I ingest prepared messages once and run consumer
> > tests
> > > multiple times, between each test run I reset the consumer group's
> offset
> > > so that each run starts from the beginning and consumers all the
> > messages.
> > >
> > > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <ss...@gmail.com>:
> > >
> > > > Have you considered deleting and recreating topic used in test?
> > > > Once topic is clean, read/poll once - any committed offset should be
> > > > outside of the range, and consumer should reset offset.
> > > >
> > > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU <ju...@gmail.com>
> > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > For local test purpose I need to frequently reset offset for a
> > consumer
> > > > > group. In 0.8 I just delete the consumer group's zk node under
> > > > > `/consumers`. But with the redesign of the 0.9, how could I achieve
> > the
> > > > > same thing?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > --
> > > > > *JU Han*
> > > > >
> > > > > Software Engineer @ Teads.tv
> > > > >
> > > > > +33 0619608888
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > *JU Han*
> > >
> > > Software Engineer @ Teads.tv
> > >
> > > +33 0619608888
> > >
> >
>
--
*JU Han*
Software Engineer @ Teads.tv
+33 0619608888
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Marko Bonaći <ma...@sematext.com>.
I was refering to Dana Powers's answer in the link I posted (to use a
client API). You can find an example here:
http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Tue, Dec 29, 2015 at 4:41 PM, Stevo Slavić <ss...@gmail.com> wrote:
> Then I guess @Before test, explicitly commit offset of 0.
>
> There doesn't seem to be a tool for committing offset, only for
> checking/fetching current offset (see
> http://kafka.apache.org/documentation.html#operations )
>
> On Tue, Dec 29, 2015 at 4:35 PM, Han JU <ju...@gmail.com> wrote:
>
> > Hi Stevo,
> >
> > But by deleting and recreating the topic, do I remove also the messages
> > ingested?
> > My use case is that I ingest prepared messages once and run consumer
> tests
> > multiple times, between each test run I reset the consumer group's offset
> > so that each run starts from the beginning and consumers all the
> messages.
> >
> > 2015-12-29 16:19 GMT+01:00 Stevo Slavić <ss...@gmail.com>:
> >
> > > Have you considered deleting and recreating topic used in test?
> > > Once topic is clean, read/poll once - any committed offset should be
> > > outside of the range, and consumer should reset offset.
> > >
> > > On Tue, Dec 29, 2015 at 4:11 PM, Han JU <ju...@gmail.com>
> wrote:
> > >
> > > > Hello,
> > > >
> > > > For local test purpose I need to frequently reset offset for a
> consumer
> > > > group. In 0.8 I just delete the consumer group's zk node under
> > > > `/consumers`. But with the redesign of the 0.9, how could I achieve
> the
> > > > same thing?
> > > >
> > > > Thanks!
> > > >
> > > > --
> > > > *JU Han*
> > > >
> > > > Software Engineer @ Teads.tv
> > > >
> > > > +33 0619608888
> > > >
> > >
> >
> >
> >
> > --
> > *JU Han*
> >
> > Software Engineer @ Teads.tv
> >
> > +33 0619608888
> >
>
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Stevo Slavić <ss...@gmail.com>.
Then I guess @Before test, explicitly commit offset of 0.
There doesn't seem to be a tool for committing offset, only for
checking/fetching current offset (see
http://kafka.apache.org/documentation.html#operations )
On Tue, Dec 29, 2015 at 4:35 PM, Han JU <ju...@gmail.com> wrote:
> Hi Stevo,
>
> But by deleting and recreating the topic, do I remove also the messages
> ingested?
> My use case is that I ingest prepared messages once and run consumer tests
> multiple times, between each test run I reset the consumer group's offset
> so that each run starts from the beginning and consumers all the messages.
>
> 2015-12-29 16:19 GMT+01:00 Stevo Slavić <ss...@gmail.com>:
>
> > Have you considered deleting and recreating topic used in test?
> > Once topic is clean, read/poll once - any committed offset should be
> > outside of the range, and consumer should reset offset.
> >
> > On Tue, Dec 29, 2015 at 4:11 PM, Han JU <ju...@gmail.com> wrote:
> >
> > > Hello,
> > >
> > > For local test purpose I need to frequently reset offset for a consumer
> > > group. In 0.8 I just delete the consumer group's zk node under
> > > `/consumers`. But with the redesign of the 0.9, how could I achieve the
> > > same thing?
> > >
> > > Thanks!
> > >
> > > --
> > > *JU Han*
> > >
> > > Software Engineer @ Teads.tv
> > >
> > > +33 0619608888
> > >
> >
>
>
>
> --
> *JU Han*
>
> Software Engineer @ Teads.tv
>
> +33 0619608888
>
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Han JU <ju...@gmail.com>.
Hi Stevo,
But by deleting and recreating the topic, do I remove also the messages
ingested?
My use case is that I ingest prepared messages once and run consumer tests
multiple times, between each test run I reset the consumer group's offset
so that each run starts from the beginning and consumers all the messages.
2015-12-29 16:19 GMT+01:00 Stevo Slavić <ss...@gmail.com>:
> Have you considered deleting and recreating topic used in test?
> Once topic is clean, read/poll once - any committed offset should be
> outside of the range, and consumer should reset offset.
>
> On Tue, Dec 29, 2015 at 4:11 PM, Han JU <ju...@gmail.com> wrote:
>
> > Hello,
> >
> > For local test purpose I need to frequently reset offset for a consumer
> > group. In 0.8 I just delete the consumer group's zk node under
> > `/consumers`. But with the redesign of the 0.9, how could I achieve the
> > same thing?
> >
> > Thanks!
> >
> > --
> > *JU Han*
> >
> > Software Engineer @ Teads.tv
> >
> > +33 0619608888
> >
>
--
*JU Han*
Software Engineer @ Teads.tv
+33 0619608888
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Stevo Slavić <ss...@gmail.com>.
Have you considered deleting and recreating topic used in test?
Once topic is clean, read/poll once - any committed offset should be
outside of the range, and consumer should reset offset.
On Tue, Dec 29, 2015 at 4:11 PM, Han JU <ju...@gmail.com> wrote:
> Hello,
>
> For local test purpose I need to frequently reset offset for a consumer
> group. In 0.8 I just delete the consumer group's zk node under
> `/consumers`. But with the redesign of the 0.9, how could I achieve the
> same thing?
>
> Thanks!
>
> --
> *JU Han*
>
> Software Engineer @ Teads.tv
>
> +33 0619608888
>
Re: How to reset a consumer-group's offset in kafka 0.9?
Posted by Marko Bonaći <ma...@sematext.com>.
We recently had the same question:
http://search-hadoop.com/m/uyzND1kM7q1gElhy1
Marko Bonaći
Monitoring | Alerting | Anomaly Detection | Centralized Log Management
Solr & Elasticsearch Support
Sematext <http://sematext.com/> | Contact
<http://sematext.com/about/contact.html>
On Tue, Dec 29, 2015 at 4:11 PM, Han JU <ju...@gmail.com> wrote:
> Hello,
>
> For local test purpose I need to frequently reset offset for a consumer
> group. In 0.8 I just delete the consumer group's zk node under
> `/consumers`. But with the redesign of the 0.9, how could I achieve the
> same thing?
>
> Thanks!
>
> --
> *JU Han*
>
> Software Engineer @ Teads.tv
>
> +33 0619608888
>