You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Liam Clarke-Hutchinson <li...@adscale.co.nz> on 2020/04/20 20:22:06 UTC
Re: kafka-consumer-groups.sh CURRENT-OFFSET column show "-" , what
mean about it ! thank you !
Hi,
It means the consumer group exists, and Kafka's aware of it's topic
subscriptions, but either the consumers haven't consumed yet (if using auto
offset commit) or the consumers haven't committed any offsets, if using
manual offset committing.
Does that make sense?
Kind regards,
Liam Clarke-Hutchinson
On Tue, Apr 21, 2020 at 8:12 PM 一直以来 <27...@qq.com> wrote:
> ghy@ghy-VirtualBox:~/T/k/bin$ ./kafka-consumer-groups.sh
> --bootstrap-server localhost:9081 --describe --group test
>
>
> GROUP TOPIC
> PARTITION CURRENT-OFFSET
> LOG-END-OFFSET LAG
> CONSUMER-ID
>
> HOST CLIENT-ID
> test myTopic3
> 4 -
> 1
> -
> consumer-test-1-d5f2f83d-59fa-4c3b-859d-4219e16479a8 /192.168.1.104
> consumer-test-1
> test myTopic3
> 1 -
> 1
> -
> consumer-test-1-d5f2f83d-59fa-4c3b-859d-4219e16479a8 /192.168.1.104
> consumer-test-1
> test myTopic3
> 2 -
> 0
> -
> consumer-test-1-d5f2f83d-59fa-4c3b-859d-4219e16479a8 /192.168.1.104
> consumer-test-1
> test myTopic3
> 3 -
> 1
> -
> consumer-test-1-d5f2f83d-59fa-4c3b-859d-4219e16479a8 /192.168.1.104
> consumer-test-1
> test myTopic3
> 0 -
> 2
> -
> consumer-test-1-d5f2f83d-59fa-4c3b-859d-4219e16479a8 /192.168.1.104
> consumer-test-1
Re: kafka-consumer-groups.sh CURRENT-OFFSET column show "-" , what
mean about it ! thank you !
Posted by Liam Clarke-Hutchinson <li...@adscale.co.nz>.
Yep, 0 is a non-null value, that is, the consumer group has a committed
offset, which in this case happens to be 0. The dash "-" implies an
unknown/null value.
In your example with your test topics - the earliest offset in the topic is
still 0, and there's not been much data through it, so maybe it makes sense
as a default value?
But look at the earliest offset on one of our topics, which has a retention
time of 7 days (as Kafka offsets are absolute for a topic, never relative):
# the -2 arg to time is shorthand for "earliest", -1 is "latest"
kafka-run-class kafka.tools.GetOffsetShell --broker-list kafka:9092 --topic
trx_reporting --time -2
trx_reporting:0:8681020918
trx_reporting:1:8675438549
trx_reporting:2:8681184674
If I saw a consumer group that listed an offset of 8681020918 on
partition 0 of topic trx_reporting, I would think it had previously been
consuming that topic. However, an offset of "-" makes it clear that it's
never consumed.
And it's important that we distinguish "My next offset to read from is 0"
from "I have never read from this partition" because of the configuration
property available to consumers, auto.offset.reset, which allows you to
modify a consumer's behaviour if the last committed offset is null:
auto.offset.reset: What to do when there is no initial offset in Kafka or
if the current offset does not exist any more on the server (e.g. because
that data has been deleted):
- earliest: automatically reset the offset to the earliest offset
- latest: automatically reset the offset to the latest offset
- none: throw exception to the consumer if no previous offset is found
for the consumer's group
- anything else: throw exception to the consumer.
Different applications will have business cases for each possible value.
And it requires being able to distinguish a consumer group with a committed
offset from a consumer group that's never committed an offset.
On Wed, Apr 22, 2020 at 8:02 PM 一直以来 <27...@qq.com> wrote:
> but show "0" and "_", show two value difference???
> thank you !
>
>
> ------------------ 原始邮件 ------------------
> 发件人: "Liam Clarke-Hutchinson"<liam.clarke@adscale.co.nz>;
> 发送时间: 2020年4月21日(星期二) 凌晨4:22
> 收件人: "users"<users@kafka.apache.org>;
>
> 主题: Re: kafka-consumer-groups.sh CURRENT-OFFSET column show "-" ,
> what mean about it ! thank you !
>
>
>
> Hi,
>
> It means the consumer group exists, and Kafka's aware of it's topic
> subscriptions, but either the consumers haven't consumed yet (if using auto
> offset commit) or the consumers haven't committed any offsets, if using
> manual offset committing.
>
> Does that make sense?
>
> Kind regards,
>
> Liam Clarke-Hutchinson
>
> On Tue, Apr 21, 2020 at 8:12 PM 一直以来 <279377921@qq.com> wrote:
>
> > ghy@ghy-VirtualBox:~/T/k/bin$ ./kafka-consumer-groups.sh
> > --bootstrap-server localhost:9081 --describe --group test
> >
> >
> > GROUP&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp;TOPIC&nbsp; &nbsp; &nbsp;
> > &nbsp; &nbsp; &nbsp;PARTITION&nbsp;
> CURRENT-OFFSET&nbsp;
> > LOG-END-OFFSET&nbsp; LAG&nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp;
> > &nbsp;CONSUMER-ID&nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; &nbsp;
> > &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; &nbsp;
> > &nbsp; &nbsp; &nbsp; HOST&nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; CLIENT-ID
> > test&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> myTopic3&nbsp; &nbsp; &nbsp;
> > &nbsp; 4&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> -&nbsp; &nbsp; &nbsp; &nbsp;
> > &nbsp; &nbsp; &nbsp; &nbsp;1&nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; &nbsp;
> > &nbsp; &nbsp;-&nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp;
> > &nbsp;consumer-test-1-d5f2f83d-59fa-4c3b-859d-4219e16479a8 /
> 192.168.1.104&nbsp;
> > consumer-test-1
> > test&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> myTopic3&nbsp; &nbsp; &nbsp;
> > &nbsp; 1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> -&nbsp; &nbsp; &nbsp; &nbsp;
> > &nbsp; &nbsp; &nbsp; &nbsp;1&nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; &nbsp;
> > &nbsp; &nbsp;-&nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp;
> > &nbsp;consumer-test-1-d5f2f83d-59fa-4c3b-859d-4219e16479a8 /
> 192.168.1.104&nbsp;
> > consumer-test-1
> > test&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> myTopic3&nbsp; &nbsp; &nbsp;
> > &nbsp; 2&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> -&nbsp; &nbsp; &nbsp; &nbsp;
> > &nbsp; &nbsp; &nbsp; &nbsp;0&nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; &nbsp;
> > &nbsp; &nbsp;-&nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp;
> > &nbsp;consumer-test-1-d5f2f83d-59fa-4c3b-859d-4219e16479a8 /
> 192.168.1.104&nbsp;
> > consumer-test-1
> > test&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> myTopic3&nbsp; &nbsp; &nbsp;
> > &nbsp; 3&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> -&nbsp; &nbsp; &nbsp; &nbsp;
> > &nbsp; &nbsp; &nbsp; &nbsp;1&nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; &nbsp;
> > &nbsp; &nbsp;-&nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp;
> > &nbsp;consumer-test-1-d5f2f83d-59fa-4c3b-859d-4219e16479a8 /
> 192.168.1.104&nbsp;
> > consumer-test-1
> > test&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> myTopic3&nbsp; &nbsp; &nbsp;
> > &nbsp; 0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> -&nbsp; &nbsp; &nbsp; &nbsp;
> > &nbsp; &nbsp; &nbsp; &nbsp;2&nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; &nbsp;
> > &nbsp; &nbsp;-&nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp;
> > &nbsp;consumer-test-1-d5f2f83d-59fa-4c3b-859d-4219e16479a8 /
> 192.168.1.104&nbsp;
> > consumer-test-1
回复： kafka-consumer-groups.sh CURRENT-OFFSET column show "-" , what mean about it ! thank you !
Posted by 一直以来 <27...@qq.com>.
but show "0" and "_", show two value difference???
thank you !
------------------ 原始邮件 ------------------
发件人: "Liam Clarke-Hutchinson"<liam.clarke@adscale.co.nz>;
发送时间: 2020年4月21日(星期二) 凌晨4:22
收件人: "users"<users@kafka.apache.org>;
主题: Re: kafka-consumer-groups.sh CURRENT-OFFSET column show "-" , what mean about it ! thank you !
Hi,
It means the consumer group exists, and Kafka's aware of it's topic
subscriptions, but either the consumers haven't consumed yet (if using auto
offset commit) or the consumers haven't committed any offsets, if using
manual offset committing.
Does that make sense?
Kind regards,
Liam Clarke-Hutchinson
On Tue, Apr 21, 2020 at 8:12 PM 一直以来 <279377921@qq.com> wrote:
> ghy@ghy-VirtualBox:~/T/k/bin$ ./kafka-consumer-groups.sh
> --bootstrap-server localhost:9081 --describe --group test
>
>
> GROUP&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;TOPIC&nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp;PARTITION&nbsp; CURRENT-OFFSET&nbsp;
> LOG-END-OFFSET&nbsp; LAG&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp;CONSUMER-ID&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; HOST&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; CLIENT-ID
> test&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; myTopic3&nbsp; &nbsp; &nbsp;
> &nbsp; 4&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; -&nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; &nbsp;1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp;-&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp;consumer-test-1-d5f2f83d-59fa-4c3b-859d-4219e16479a8 /192.168.1.104&nbsp;
> consumer-test-1
> test&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; myTopic3&nbsp; &nbsp; &nbsp;
> &nbsp; 1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; -&nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; &nbsp;1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp;-&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp;consumer-test-1-d5f2f83d-59fa-4c3b-859d-4219e16479a8 /192.168.1.104&nbsp;
> consumer-test-1
> test&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; myTopic3&nbsp; &nbsp; &nbsp;
> &nbsp; 2&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; -&nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; &nbsp;0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp;-&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp;consumer-test-1-d5f2f83d-59fa-4c3b-859d-4219e16479a8 /192.168.1.104&nbsp;
> consumer-test-1
> test&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; myTopic3&nbsp; &nbsp; &nbsp;
> &nbsp; 3&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; -&nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; &nbsp;1&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp;-&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp;consumer-test-1-d5f2f83d-59fa-4c3b-859d-4219e16479a8 /192.168.1.104&nbsp;
> consumer-test-1
> test&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; myTopic3&nbsp; &nbsp; &nbsp;
> &nbsp; 0&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; -&nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp; &nbsp; &nbsp;2&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp; &nbsp;-&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;
> &nbsp;consumer-test-1-d5f2f83d-59fa-4c3b-859d-4219e16479a8 /192.168.1.104&nbsp;
> consumer-test-1