You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by Alex D <a....@gmail.com> on 2018/10/03 14:11:20 UTC

[DISCUSS] KIP-379: Multiple Consumer Group Management

Hello, friends!

Welcome to the Multiple Consumer Group Management feature for
kafka-consumer-groups utility discussion thread.

KIP is available here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-379%3A+Multiple+Consumer+Group+Management

Pull Request: https://github.com/apache/kafka/pull/5726

JIRA ticket: https://issues.apache.org/jira/browse/KAFKA-7471

What do you think?

Thanks,
Alexander Dunayevsky

Re: [DISCUSS] KIP-379: Multiple Consumer Group Management

Posted by ea...@gmail.com.

Is there a reason for using multiple --group flags over having it accept a regex?

The topics command currently accepts a regex for most operations and doesn't support using multiple topics flags. It seems like it would be better to take a more standardized approach to providing this type of information.

> On Oct 19, 2018, at 10:28 AM, Suman B N <su...@gmail.com> wrote:
> 
> This eases debugging metadata information of consumer groups and offsets in
> case of client hungs which we have been facing frequently.
> +1 from me. Well done Alex!
> 
> -Suman
> 
> On Fri, Oct 19, 2018 at 8:36 PM Vahid Hashemian <va...@gmail.com>
> wrote:
> 
>> Thanks for proposing the KIP. Looks good to me overall.
>> 
>> I agree with Jason's suggestion that it would be best to keep the current
>> output format when a single '--group' is present. Because otherwise, there
>> would be an impact to users who rely on the current output format. Also,
>> starting with a GROUP column makes more sense to me.
>> 
>> Also, and for my own info, is there a valid scenario where we would want to
>> delete all consumer groups? It sounds to me like a potentially dangerous
>> feature. I would imagine that it would help more with dev/test
>> environments, where we normally have a few groups (for which the repeating
>> '--group' option should work).
>> 
>> Regards!
>> --Vahid
>> 
>> On Thu, Oct 18, 2018 at 11:28 PM Jason Gustafson <ja...@confluent.io>
>> wrote:
>> 
>>> Hi Alex,
>>> 
>>> Thanks for the KIP. I think it makes sense, especially since most of the
>>> group apis are intended for batching anyway.
>>> 
>>> The only questions I have are about compatibility. For example, the csv
>>> format for resetting offsets is changed, so will we continue to support
>> the
>>> old format? Also, if only one `--group` option is passed, do you think
>> it's
>>> worth leaving it the group column out of the `--describe` output? That
>>> would keep the describe output consistent with the current implementation
>>> for the current usage. Finally, and this is just a nitpick, but do you
>>> think it makes sense to put the group column first in the describe
>> output?
>>> 
>>> Thanks,
>>> Jason
>>> 
>>> 
>>>> On Wed, Oct 3, 2018 at 7:11 AM, Alex D <a....@gmail.com> wrote:
>>>> 
>>>> Hello, friends!
>>>> 
>>>> Welcome to the Multiple Consumer Group Management feature for
>>>> kafka-consumer-groups utility discussion thread.
>>>> 
>>>> KIP is available here:
>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-379%
>>>> 3A+Multiple+Consumer+Group+Management
>>>> 
>>>> Pull Request: https://github.com/apache/kafka/pull/5726
>>>> 
>>>> JIRA ticket: https://issues.apache.org/jira/browse/KAFKA-7471
>>>> 
>>>> What do you think?
>>>> 
>>>> Thanks,
>>>> Alexander Dunayevsky
>>>> 
>>> 
>> 
> 
> 
> -- 
> *Suman*
> *OlaCabs*

Re: [DISCUSS] KIP-379: Multiple Consumer Group Management

Posted by Suman B N <su...@gmail.com>.

This eases debugging metadata information of consumer groups and offsets in
case of client hungs which we have been facing frequently.
+1 from me. Well done Alex!

-Suman

On Fri, Oct 19, 2018 at 8:36 PM Vahid Hashemian <va...@gmail.com>
wrote:

> Thanks for proposing the KIP. Looks good to me overall.
>
> I agree with Jason's suggestion that it would be best to keep the current
> output format when a single '--group' is present. Because otherwise, there
> would be an impact to users who rely on the current output format. Also,
> starting with a GROUP column makes more sense to me.
>
> Also, and for my own info, is there a valid scenario where we would want to
> delete all consumer groups? It sounds to me like a potentially dangerous
> feature. I would imagine that it would help more with dev/test
> environments, where we normally have a few groups (for which the repeating
> '--group' option should work).
>
> Regards!
> --Vahid
>
> On Thu, Oct 18, 2018 at 11:28 PM Jason Gustafson <ja...@confluent.io>
> wrote:
>
> > Hi Alex,
> >
> > Thanks for the KIP. I think it makes sense, especially since most of the
> > group apis are intended for batching anyway.
> >
> > The only questions I have are about compatibility. For example, the csv
> > format for resetting offsets is changed, so will we continue to support
> the
> > old format? Also, if only one `--group` option is passed, do you think
> it's
> > worth leaving it the group column out of the `--describe` output? That
> > would keep the describe output consistent with the current implementation
> > for the current usage. Finally, and this is just a nitpick, but do you
> > think it makes sense to put the group column first in the describe
> output?
> >
> > Thanks,
> > Jason
> >
> >
> > On Wed, Oct 3, 2018 at 7:11 AM, Alex D <a....@gmail.com> wrote:
> >
> > > Hello, friends!
> > >
> > > Welcome to the Multiple Consumer Group Management feature for
> > > kafka-consumer-groups utility discussion thread.
> > >
> > > KIP is available here:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-379%
> > > 3A+Multiple+Consumer+Group+Management
> > >
> > > Pull Request: https://github.com/apache/kafka/pull/5726
> > >
> > > JIRA ticket: https://issues.apache.org/jira/browse/KAFKA-7471
> > >
> > > What do you think?
> > >
> > > Thanks,
> > > Alexander Dunayevsky
> > >
> >
>


-- 
*Suman*
*OlaCabs*

Re: [DISCUSS] KIP-379: Multiple Consumer Group Management

Posted by Vahid Hashemian <va...@gmail.com>.

Thanks for proposing the KIP. Looks good to me overall.

I agree with Jason's suggestion that it would be best to keep the current
output format when a single '--group' is present. Because otherwise, there
would be an impact to users who rely on the current output format. Also,
starting with a GROUP column makes more sense to me.

Also, and for my own info, is there a valid scenario where we would want to
delete all consumer groups? It sounds to me like a potentially dangerous
feature. I would imagine that it would help more with dev/test
environments, where we normally have a few groups (for which the repeating
'--group' option should work).

Regards!
--Vahid

On Thu, Oct 18, 2018 at 11:28 PM Jason Gustafson <ja...@confluent.io> wrote:

> Hi Alex,
>
> Thanks for the KIP. I think it makes sense, especially since most of the
> group apis are intended for batching anyway.
>
> The only questions I have are about compatibility. For example, the csv
> format for resetting offsets is changed, so will we continue to support the
> old format? Also, if only one `--group` option is passed, do you think it's
> worth leaving it the group column out of the `--describe` output? That
> would keep the describe output consistent with the current implementation
> for the current usage. Finally, and this is just a nitpick, but do you
> think it makes sense to put the group column first in the describe output?
>
> Thanks,
> Jason
>
>
> On Wed, Oct 3, 2018 at 7:11 AM, Alex D <a....@gmail.com> wrote:
>
> > Hello, friends!
> >
> > Welcome to the Multiple Consumer Group Management feature for
> > kafka-consumer-groups utility discussion thread.
> >
> > KIP is available here:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-379%
> > 3A+Multiple+Consumer+Group+Management
> >
> > Pull Request: https://github.com/apache/kafka/pull/5726
> >
> > JIRA ticket: https://issues.apache.org/jira/browse/KAFKA-7471
> >
> > What do you think?
> >
> > Thanks,
> > Alexander Dunayevsky
> >
>

Re: [DISCUSS] KIP-379: Multiple Consumer Group Management

Posted by Jason Gustafson <ja...@confluent.io>.

Hi Alex,

Thanks for the KIP. I think it makes sense, especially since most of the
group apis are intended for batching anyway.

The only questions I have are about compatibility. For example, the csv
format for resetting offsets is changed, so will we continue to support the
old format? Also, if only one `--group` option is passed, do you think it's
worth leaving it the group column out of the `--describe` output? That
would keep the describe output consistent with the current implementation
for the current usage. Finally, and this is just a nitpick, but do you
think it makes sense to put the group column first in the describe output?

Thanks,
Jason

On Wed, Oct 3, 2018 at 7:11 AM, Alex D <a....@gmail.com> wrote:

> Hello, friends!
>
> Welcome to the Multiple Consumer Group Management feature for
> kafka-consumer-groups utility discussion thread.
>
> KIP is available here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-379%
> 3A+Multiple+Consumer+Group+Management
>
> Pull Request: https://github.com/apache/kafka/pull/5726
>
> JIRA ticket: https://issues.apache.org/jira/browse/KAFKA-7471
>
> What do you think?
>
> Thanks,
> Alexander Dunayevsky
>