You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "yuanjia8947@163.com" <yu...@163.com> on 2016/08/12 03:55:23 UTC
kafka-consumer-groups.sh delete group with new-consumer
Hi all，
When I use kafka-consumer-groups.sh to delete new-consumer's group, the shell note that "Option [delete] is not valid with [new-consumer]. Note that there's no need to delete group metadata for the new consumer as it is automatically deleted when the last member leaves."
How to understand it is automatically deleted when the last member leaves?
Does it means the consumer's position could lost if the consumer stop?
Thanks and Regards
YuanjiaLi
Re: Re: kafka-consumer-groups.sh delete group with new-consumer
Posted by Guozhang Wang <wa...@gmail.com>.
I think my previous answer was not complete thanks to Tao's reminder: in
fact, the retention is controlled both on the client and server side,
1) users can set the retention time period on the OffsetCommitRequest (in
latest version v2).
2) in the current Java client implementation, it is always set to "-1", but
we can let users to configure it moving forward.
3) on the server side, if the received request's retention time is set to
"-1", it will use its server-side config named "offsetsRetentionMinutes"
with default to 1440 as Tao mentioned; otherwise it will use the request
specified retention time.
Guozhang
On Sun, Aug 14, 2016 at 7:35 PM, yuanjia8947@163.com <yu...@163.com>
wrote:
> Hi Guozhang,
> Thanks for replies. And which is the expiration parameter? Can I
> config it on broker or on consumer?
>
> Thanks and Regards
>
>
>
> Yuanjia Li
>
> From: Guozhang Wang
> Date: 2016-08-13 03:05
> To: users@kafka.apache.org
> Subject: Re: kafka-consumer-groups.sh delete group with new-consumer
> Hi Yuanjia,
>
> New consumer's group registry information is stored on the Kafka brokers,
> not ZK any more, and the brokers use heartbeats to detect if the consumer
> has been failed and hence can be removed from the group or not.
>
> In addition, the consumer's committed offsets (i.e. "positions" as you
> mentioned it) has an expiration parameter (default is "forever") based on
> which the offsets will be automatically cleaned on the brokers as well.
>
> When all members are gone from a group, i.e. the group is empty, AND all
> the group's previous consumer's committed offsets have all been expired,
> the group registry will be deleted.
>
>
> Guozhang
>
> On Thu, Aug 11, 2016 at 8:55 PM, yuanjia8947@163.com <yu...@163.com>
> wrote:
>
> > Hi all，
> > When I use kafka-consumer-groups.sh to delete new-consumer's group,
> > the shell note that "Option [delete] is not valid with [new-consumer].
> Note
> > that there's no need to delete group metadata for the new consumer as it
> is
> > automatically deleted when the last member leaves."
> > How to understand it is automatically deleted when the last member
> > leaves?
> > Does it means the consumer's position could lost if the consumer
> stop?
> >
> >
> > Thanks and Regards
> >
> >
> >
> > YuanjiaLi
> >
>
>
>
> --
> -- Guozhang
>
--
-- Guozhang
Re: Re: kafka-consumer-groups.sh delete group with new-consumer
Posted by "yuanjia8947@163.com" <yu...@163.com>.
Hi Guozhang,
Thanks for replies. And which is the expiration parameter? Can I config it on broker or on consumer?
Thanks and Regards
Yuanjia Li
From: Guozhang Wang
Date: 2016-08-13 03:05
To: users@kafka.apache.org
Subject: Re: kafka-consumer-groups.sh delete group with new-consumer
Hi Yuanjia,
New consumer's group registry information is stored on the Kafka brokers,
not ZK any more, and the brokers use heartbeats to detect if the consumer
has been failed and hence can be removed from the group or not.
In addition, the consumer's committed offsets (i.e. "positions" as you
mentioned it) has an expiration parameter (default is "forever") based on
which the offsets will be automatically cleaned on the brokers as well.
When all members are gone from a group, i.e. the group is empty, AND all
the group's previous consumer's committed offsets have all been expired,
the group registry will be deleted.
Guozhang
On Thu, Aug 11, 2016 at 8:55 PM, yuanjia8947@163.com <yu...@163.com>
wrote:
> Hi all，
> When I use kafka-consumer-groups.sh to delete new-consumer's group,
> the shell note that "Option [delete] is not valid with [new-consumer]. Note
> that there's no need to delete group metadata for the new consumer as it is
> automatically deleted when the last member leaves."
> How to understand it is automatically deleted when the last member
> leaves?
> Does it means the consumer's position could lost if the consumer stop?
>
>
> Thanks and Regards
>
>
>
> YuanjiaLi
>
--
-- Guozhang
Re: kafka-consumer-groups.sh delete group with new-consumer
Posted by tao xiao <xi...@gmail.com>.
Shouldn't the default offset retention minutes be 1440?
offsets.retention.minutes=1440
On Sat, 13 Aug 2016 at 03:05 Guozhang Wang <wa...@gmail.com> wrote:
> Hi Yuanjia,
>
> New consumer's group registry information is stored on the Kafka brokers,
> not ZK any more, and the brokers use heartbeats to detect if the consumer
> has been failed and hence can be removed from the group or not.
>
> In addition, the consumer's committed offsets (i.e. "positions" as you
> mentioned it) has an expiration parameter (default is "forever") based on
> which the offsets will be automatically cleaned on the brokers as well.
>
> When all members are gone from a group, i.e. the group is empty, AND all
> the group's previous consumer's committed offsets have all been expired,
> the group registry will be deleted.
>
>
> Guozhang
>
> On Thu, Aug 11, 2016 at 8:55 PM, yuanjia8947@163.com <yu...@163.com>
> wrote:
>
> > Hi all，
> > When I use kafka-consumer-groups.sh to delete new-consumer's group,
> > the shell note that "Option [delete] is not valid with [new-consumer].
> Note
> > that there's no need to delete group metadata for the new consumer as it
> is
> > automatically deleted when the last member leaves."
> > How to understand it is automatically deleted when the last member
> > leaves?
> > Does it means the consumer's position could lost if the consumer
> stop?
> >
> >
> > Thanks and Regards
> >
> >
> >
> > YuanjiaLi
> >
>
>
>
> --
> -- Guozhang
>
Re: kafka-consumer-groups.sh delete group with new-consumer
Posted by R Krishna <kr...@gmail.com>.
Guozhang, don't we just need latest partition offsets per consumer? what is
the idea behind keeping all offsets "default" to forever?
On Fri, Aug 12, 2016 at 12:05 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Hi Yuanjia,
>
> New consumer's group registry information is stored on the Kafka brokers,
> not ZK any more, and the brokers use heartbeats to detect if the consumer
> has been failed and hence can be removed from the group or not.
>
> In addition, the consumer's committed offsets (i.e. "positions" as you
> mentioned it) has an expiration parameter (default is "forever") based on
> which the offsets will be automatically cleaned on the brokers as well.
>
> When all members are gone from a group, i.e. the group is empty, AND all
> the group's previous consumer's committed offsets have all been expired,
> the group registry will be deleted.
>
>
> Guozhang
>
> On Thu, Aug 11, 2016 at 8:55 PM, yuanjia8947@163.com <yu...@163.com>
> wrote:
>
> > Hi all，
> > When I use kafka-consumer-groups.sh to delete new-consumer's group,
> > the shell note that "Option [delete] is not valid with [new-consumer].
> Note
> > that there's no need to delete group metadata for the new consumer as it
> is
> > automatically deleted when the last member leaves."
> > How to understand it is automatically deleted when the last member
> > leaves?
> > Does it means the consumer's position could lost if the consumer
> stop?
> >
> >
> > Thanks and Regards
> >
> >
> >
> > YuanjiaLi
> >
>
>
>
> --
> -- Guozhang
>
--
Radha Krishna, Proddaturi
253-234-5657
Re: kafka-consumer-groups.sh delete group with new-consumer
Posted by Guozhang Wang <wa...@gmail.com>.
Hi Yuanjia,
New consumer's group registry information is stored on the Kafka brokers,
not ZK any more, and the brokers use heartbeats to detect if the consumer
has been failed and hence can be removed from the group or not.
In addition, the consumer's committed offsets (i.e. "positions" as you
mentioned it) has an expiration parameter (default is "forever") based on
which the offsets will be automatically cleaned on the brokers as well.
When all members are gone from a group, i.e. the group is empty, AND all
the group's previous consumer's committed offsets have all been expired,
the group registry will be deleted.
Guozhang
On Thu, Aug 11, 2016 at 8:55 PM, yuanjia8947@163.com <yu...@163.com>
wrote:
> Hi all，
> When I use kafka-consumer-groups.sh to delete new-consumer's group,
> the shell note that "Option [delete] is not valid with [new-consumer]. Note
> that there's no need to delete group metadata for the new consumer as it is
> automatically deleted when the last member leaves."
> How to understand it is automatically deleted when the last member
> leaves?
> Does it means the consumer's position could lost if the consumer stop?
>
>
> Thanks and Regards
>
>
>
> YuanjiaLi
>
--
-- Guozhang