You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Vahid S Hashemian <va...@us.ibm.com> on 2018/08/02 21:23:54 UTC
ConsumerGroupCommand tool improvement?
Hi all,
A requirement has been raised by a colleague and I wanted to see if there
is any interest in the community in adding the functionality to Apache
Kafka.
ConsumerGroupCommand tool in describe ('--describe' or '--describe
--offsets') mode currently lists all topics the group has consumed from
and all consumers with assigned partitions for a single group.
The idea is to allow filtering of topics, consumers (client ids), and even
groups using regular expressions. This will allow the tool to handle use
cases such as
What's the status of a particular consumer (or consumers) in all the
groups they are consuming from? (for example to check if they are lagging
behind in all groups)
What consumer groups are consuming from a topic (or topics) and what's the
lag for each group?
Limit the existing result to the topics/consumers of interest (for groups
with several topics/consumers)
...
This would potentially lead to enhancing the AdminClient API as well.
If the community also sees a value in this, I could start drafting a KIP.
Thanks for your feedback.
--Vahid
Re: ConsumerGroupCommand tool improvement?
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Colin,
Thanks for the feedback!
I understand your concerns (I was thinking about making this improvement
in a way that's fully backward compatible, with similar regex syntax as in
some of the existing tools).
In any case, If there is not enough interest in the user community I'll
rest my case.
Thanks!
--Vahid
From: Colin McCabe <cm...@apache.org>
To: users@kafka.apache.org
Date: 08/06/2018 12:44 PM
Subject: Re: ConsumerGroupCommand tool improvement?
On Mon, Aug 6, 2018, at 12:04, Vahid S Hashemian wrote:
> Hi Colin,
>
> Thanks for considering the idea and sharing your feedback.
>
> The improvements I proposed can be achieved, to some extend, using the
> AdminClient API and the Consumer Group CLI tool. But they won't fully
> support the proposal.
>
> For example,
> Regular expressions are not supported on the groups
> Topic / Client filtering is not supported across all groups
>
> So the reason for proposing the idea was to see if other Kafka users are
> also interested in some of these features so we can remove the burden of
> them writing custom code around existing consumer group features, and
make
> those features built into Kafka Consumer Group Command and AdminClient
> API.
Hmm. If you're writing Java code that calls the APIs, though, this is
easy, right? You can filter the groups in the cluster with a regular
expression with just a single line of code in Java 8.
> adminClient.listConsumerGroups().all().stream().
> filter(listing -> listing.groupId().matches(myRegex)).
> collect(Collectors.toList());
An option to filter ListConsumerGroups by a regular expression might
actually be less easy-to-use for most users than this simple filter, since
users would have to read the JavaDoc for our APIs.
Maybe there are some use-cases where it makes sense to add regex support
to the command-line tools, though.
I guess the reason why I am pushing back on this is that regular
expressions add a lot of complexity to the API and make it harder for us
to meet our backwards compatibility guarantees. For example, Java regular
expressions changed slightly between Java 7 and Java 8
best,
Colin
>
> Thanks again!
> --Vahid
>
>
>
> From: Colin McCabe <cm...@apache.org>
> To: users@kafka.apache.org
> Date: 08/03/2018 04:16 PM
> Subject: Re: ConsumerGroupCommand tool improvement?
>
>
>
> Hi Vahid,
>
> Interesting idea.
>
> It seems like if you're using the AdminClient APIs programmatically, you
> can just do the filtering yourself in a more flexible way than what we
> could provide.
>
> On the other hand, if you're using the ./bin/consumer-groups.sh
> command-line tool, why not use grep or a similar tool to filter the
> output? Maybe there is some additional functionality in supporting
> regexes in the command-line tool, but it also seems like it might be
kind
> of complex as well. Do you have some examples where having regex
support
> int the tool would be much easier than the traditional way of piping the
> output to grep, awk, and sed?
>
> best,
> Colin
>
>
> On Thu, Aug 2, 2018, at 14:23, Vahid S Hashemian wrote:
> > Hi all,
> >
> > A requirement has been raised by a colleague and I wanted to see if
> there
> > is any interest in the community in adding the functionality to Apache
> > Kafka.
> >
> > ConsumerGroupCommand tool in describe ('--describe' or '--describe
> > --offsets') mode currently lists all topics the group has consumed
from
> > and all consumers with assigned partitions for a single group.
> > The idea is to allow filtering of topics, consumers (client ids), and
> even
> > groups using regular expressions. This will allow the tool to handle
use
>
> > cases such as
> > What's the status of a particular consumer (or consumers) in all the
> > groups they are consuming from? (for example to check if they are
> lagging
> > behind in all groups)
> > What consumer groups are consuming from a topic (or topics) and what's
> the
> > lag for each group?
> > Limit the existing result to the topics/consumers of interest (for
> groups
> > with several topics/consumers)
> > ...
> >
> > This would potentially lead to enhancing the AdminClient API as well.
> >
> > If the community also sees a value in this, I could start drafting a
> KIP.
> >
> > Thanks for your feedback.
> > --Vahid
> >
>
>
>
>
>
Re: ConsumerGroupCommand tool improvement?
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Colin,
Thanks for the feedback!
I understand your concerns (I was thinking about making this improvement
in a way that's fully backward compatible, with similar regex syntax as in
some of the existing tools).
In any case, If there is not enough interest in the user community I'll
rest my case.
Thanks!
--Vahid
From: Colin McCabe <cm...@apache.org>
To: users@kafka.apache.org
Date: 08/06/2018 12:44 PM
Subject: Re: ConsumerGroupCommand tool improvement?
On Mon, Aug 6, 2018, at 12:04, Vahid S Hashemian wrote:
> Hi Colin,
>
> Thanks for considering the idea and sharing your feedback.
>
> The improvements I proposed can be achieved, to some extend, using the
> AdminClient API and the Consumer Group CLI tool. But they won't fully
> support the proposal.
>
> For example,
> Regular expressions are not supported on the groups
> Topic / Client filtering is not supported across all groups
>
> So the reason for proposing the idea was to see if other Kafka users are
> also interested in some of these features so we can remove the burden of
> them writing custom code around existing consumer group features, and
make
> those features built into Kafka Consumer Group Command and AdminClient
> API.
Hmm. If you're writing Java code that calls the APIs, though, this is
easy, right? You can filter the groups in the cluster with a regular
expression with just a single line of code in Java 8.
> adminClient.listConsumerGroups().all().stream().
> filter(listing -> listing.groupId().matches(myRegex)).
> collect(Collectors.toList());
An option to filter ListConsumerGroups by a regular expression might
actually be less easy-to-use for most users than this simple filter, since
users would have to read the JavaDoc for our APIs.
Maybe there are some use-cases where it makes sense to add regex support
to the command-line tools, though.
I guess the reason why I am pushing back on this is that regular
expressions add a lot of complexity to the API and make it harder for us
to meet our backwards compatibility guarantees. For example, Java regular
expressions changed slightly between Java 7 and Java 8
best,
Colin
>
> Thanks again!
> --Vahid
>
>
>
> From: Colin McCabe <cm...@apache.org>
> To: users@kafka.apache.org
> Date: 08/03/2018 04:16 PM
> Subject: Re: ConsumerGroupCommand tool improvement?
>
>
>
> Hi Vahid,
>
> Interesting idea.
>
> It seems like if you're using the AdminClient APIs programmatically, you
> can just do the filtering yourself in a more flexible way than what we
> could provide.
>
> On the other hand, if you're using the ./bin/consumer-groups.sh
> command-line tool, why not use grep or a similar tool to filter the
> output? Maybe there is some additional functionality in supporting
> regexes in the command-line tool, but it also seems like it might be
kind
> of complex as well. Do you have some examples where having regex
support
> int the tool would be much easier than the traditional way of piping the
> output to grep, awk, and sed?
>
> best,
> Colin
>
>
> On Thu, Aug 2, 2018, at 14:23, Vahid S Hashemian wrote:
> > Hi all,
> >
> > A requirement has been raised by a colleague and I wanted to see if
> there
> > is any interest in the community in adding the functionality to Apache
> > Kafka.
> >
> > ConsumerGroupCommand tool in describe ('--describe' or '--describe
> > --offsets') mode currently lists all topics the group has consumed
from
> > and all consumers with assigned partitions for a single group.
> > The idea is to allow filtering of topics, consumers (client ids), and
> even
> > groups using regular expressions. This will allow the tool to handle
use
>
> > cases such as
> > What's the status of a particular consumer (or consumers) in all the
> > groups they are consuming from? (for example to check if they are
> lagging
> > behind in all groups)
> > What consumer groups are consuming from a topic (or topics) and what's
> the
> > lag for each group?
> > Limit the existing result to the topics/consumers of interest (for
> groups
> > with several topics/consumers)
> > ...
> >
> > This would potentially lead to enhancing the AdminClient API as well.
> >
> > If the community also sees a value in this, I could start drafting a
> KIP.
> >
> > Thanks for your feedback.
> > --Vahid
> >
>
>
>
>
>
Re: ConsumerGroupCommand tool improvement?
Posted by Colin McCabe <cm...@apache.org>.
On Mon, Aug 6, 2018, at 12:04, Vahid S Hashemian wrote:
> Hi Colin,
>
> Thanks for considering the idea and sharing your feedback.
>
> The improvements I proposed can be achieved, to some extend, using the
> AdminClient API and the Consumer Group CLI tool. But they won't fully
> support the proposal.
>
> For example,
> Regular expressions are not supported on the groups
> Topic / Client filtering is not supported across all groups
>
> So the reason for proposing the idea was to see if other Kafka users are
> also interested in some of these features so we can remove the burden of
> them writing custom code around existing consumer group features, and make
> those features built into Kafka Consumer Group Command and AdminClient
> API.
Hmm. If you're writing Java code that calls the APIs, though, this is easy, right? You can filter the groups in the cluster with a regular expression with just a single line of code in Java 8.
> adminClient.listConsumerGroups().all().stream().
> filter(listing -> listing.groupId().matches(myRegex)).
> collect(Collectors.toList());
An option to filter ListConsumerGroups by a regular expression might actually be less easy-to-use for most users than this simple filter, since users would have to read the JavaDoc for our APIs.
Maybe there are some use-cases where it makes sense to add regex support to the command-line tools, though.
I guess the reason why I am pushing back on this is that regular expressions add a lot of complexity to the API and make it harder for us to meet our backwards compatibility guarantees. For example, Java regular expressions changed slightly between Java 7 and Java 8
best,
Colin
>
> Thanks again!
> --Vahid
>
>
>
> From: Colin McCabe <cm...@apache.org>
> To: users@kafka.apache.org
> Date: 08/03/2018 04:16 PM
> Subject: Re: ConsumerGroupCommand tool improvement?
>
>
>
> Hi Vahid,
>
> Interesting idea.
>
> It seems like if you're using the AdminClient APIs programmatically, you
> can just do the filtering yourself in a more flexible way than what we
> could provide.
>
> On the other hand, if you're using the ./bin/consumer-groups.sh
> command-line tool, why not use grep or a similar tool to filter the
> output? Maybe there is some additional functionality in supporting
> regexes in the command-line tool, but it also seems like it might be kind
> of complex as well. Do you have some examples where having regex support
> int the tool would be much easier than the traditional way of piping the
> output to grep, awk, and sed?
>
> best,
> Colin
>
>
> On Thu, Aug 2, 2018, at 14:23, Vahid S Hashemian wrote:
> > Hi all,
> >
> > A requirement has been raised by a colleague and I wanted to see if
> there
> > is any interest in the community in adding the functionality to Apache
> > Kafka.
> >
> > ConsumerGroupCommand tool in describe ('--describe' or '--describe
> > --offsets') mode currently lists all topics the group has consumed from
> > and all consumers with assigned partitions for a single group.
> > The idea is to allow filtering of topics, consumers (client ids), and
> even
> > groups using regular expressions. This will allow the tool to handle use
>
> > cases such as
> > What's the status of a particular consumer (or consumers) in all the
> > groups they are consuming from? (for example to check if they are
> lagging
> > behind in all groups)
> > What consumer groups are consuming from a topic (or topics) and what's
> the
> > lag for each group?
> > Limit the existing result to the topics/consumers of interest (for
> groups
> > with several topics/consumers)
> > ...
> >
> > This would potentially lead to enhancing the AdminClient API as well.
> >
> > If the community also sees a value in this, I could start drafting a
> KIP.
> >
> > Thanks for your feedback.
> > --Vahid
> >
>
>
>
>
>
Re: ConsumerGroupCommand tool improvement?
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Colin,
Thanks for considering the idea and sharing your feedback.
The improvements I proposed can be achieved, to some extend, using the
AdminClient API and the Consumer Group CLI tool. But they won't fully
support the proposal.
For example,
Regular expressions are not supported on the groups
Topic / Client filtering is not supported across all groups
So the reason for proposing the idea was to see if other Kafka users are
also interested in some of these features so we can remove the burden of
them writing custom code around existing consumer group features, and make
those features built into Kafka Consumer Group Command and AdminClient
API.
Thanks again!
--Vahid
From: Colin McCabe <cm...@apache.org>
To: users@kafka.apache.org
Date: 08/03/2018 04:16 PM
Subject: Re: ConsumerGroupCommand tool improvement?
Hi Vahid,
Interesting idea.
It seems like if you're using the AdminClient APIs programmatically, you
can just do the filtering yourself in a more flexible way than what we
could provide.
On the other hand, if you're using the ./bin/consumer-groups.sh
command-line tool, why not use grep or a similar tool to filter the
output? Maybe there is some additional functionality in supporting
regexes in the command-line tool, but it also seems like it might be kind
of complex as well. Do you have some examples where having regex support
int the tool would be much easier than the traditional way of piping the
output to grep, awk, and sed?
best,
Colin
On Thu, Aug 2, 2018, at 14:23, Vahid S Hashemian wrote:
> Hi all,
>
> A requirement has been raised by a colleague and I wanted to see if
there
> is any interest in the community in adding the functionality to Apache
> Kafka.
>
> ConsumerGroupCommand tool in describe ('--describe' or '--describe
> --offsets') mode currently lists all topics the group has consumed from
> and all consumers with assigned partitions for a single group.
> The idea is to allow filtering of topics, consumers (client ids), and
even
> groups using regular expressions. This will allow the tool to handle use
> cases such as
> What's the status of a particular consumer (or consumers) in all the
> groups they are consuming from? (for example to check if they are
lagging
> behind in all groups)
> What consumer groups are consuming from a topic (or topics) and what's
the
> lag for each group?
> Limit the existing result to the topics/consumers of interest (for
groups
> with several topics/consumers)
> ...
>
> This would potentially lead to enhancing the AdminClient API as well.
>
> If the community also sees a value in this, I could start drafting a
KIP.
>
> Thanks for your feedback.
> --Vahid
>
Re: ConsumerGroupCommand tool improvement?
Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Colin,
Thanks for considering the idea and sharing your feedback.
The improvements I proposed can be achieved, to some extend, using the
AdminClient API and the Consumer Group CLI tool. But they won't fully
support the proposal.
For example,
Regular expressions are not supported on the groups
Topic / Client filtering is not supported across all groups
So the reason for proposing the idea was to see if other Kafka users are
also interested in some of these features so we can remove the burden of
them writing custom code around existing consumer group features, and make
those features built into Kafka Consumer Group Command and AdminClient
API.
Thanks again!
--Vahid
From: Colin McCabe <cm...@apache.org>
To: users@kafka.apache.org
Date: 08/03/2018 04:16 PM
Subject: Re: ConsumerGroupCommand tool improvement?
Hi Vahid,
Interesting idea.
It seems like if you're using the AdminClient APIs programmatically, you
can just do the filtering yourself in a more flexible way than what we
could provide.
On the other hand, if you're using the ./bin/consumer-groups.sh
command-line tool, why not use grep or a similar tool to filter the
output? Maybe there is some additional functionality in supporting
regexes in the command-line tool, but it also seems like it might be kind
of complex as well. Do you have some examples where having regex support
int the tool would be much easier than the traditional way of piping the
output to grep, awk, and sed?
best,
Colin
On Thu, Aug 2, 2018, at 14:23, Vahid S Hashemian wrote:
> Hi all,
>
> A requirement has been raised by a colleague and I wanted to see if
there
> is any interest in the community in adding the functionality to Apache
> Kafka.
>
> ConsumerGroupCommand tool in describe ('--describe' or '--describe
> --offsets') mode currently lists all topics the group has consumed from
> and all consumers with assigned partitions for a single group.
> The idea is to allow filtering of topics, consumers (client ids), and
even
> groups using regular expressions. This will allow the tool to handle use
> cases such as
> What's the status of a particular consumer (or consumers) in all the
> groups they are consuming from? (for example to check if they are
lagging
> behind in all groups)
> What consumer groups are consuming from a topic (or topics) and what's
the
> lag for each group?
> Limit the existing result to the topics/consumers of interest (for
groups
> with several topics/consumers)
> ...
>
> This would potentially lead to enhancing the AdminClient API as well.
>
> If the community also sees a value in this, I could start drafting a
KIP.
>
> Thanks for your feedback.
> --Vahid
>
Re: ConsumerGroupCommand tool improvement?
Posted by Colin McCabe <cm...@apache.org>.
Hi Vahid,
Interesting idea.
It seems like if you're using the AdminClient APIs programmatically, you can just do the filtering yourself in a more flexible way than what we could provide.
On the other hand, if you're using the ./bin/consumer-groups.sh command-line tool, why not use grep or a similar tool to filter the output? Maybe there is some additional functionality in supporting regexes in the command-line tool, but it also seems like it might be kind of complex as well. Do you have some examples where having regex support int the tool would be much easier than the traditional way of piping the output to grep, awk, and sed?
best,
Colin
On Thu, Aug 2, 2018, at 14:23, Vahid S Hashemian wrote:
> Hi all,
>
> A requirement has been raised by a colleague and I wanted to see if there
> is any interest in the community in adding the functionality to Apache
> Kafka.
>
> ConsumerGroupCommand tool in describe ('--describe' or '--describe
> --offsets') mode currently lists all topics the group has consumed from
> and all consumers with assigned partitions for a single group.
> The idea is to allow filtering of topics, consumers (client ids), and even
> groups using regular expressions. This will allow the tool to handle use
> cases such as
> What's the status of a particular consumer (or consumers) in all the
> groups they are consuming from? (for example to check if they are lagging
> behind in all groups)
> What consumer groups are consuming from a topic (or topics) and what's the
> lag for each group?
> Limit the existing result to the topics/consumers of interest (for groups
> with several topics/consumers)
> ...
>
> This would potentially lead to enhancing the AdminClient API as well.
>
> If the community also sees a value in this, I could start drafting a KIP.
>
> Thanks for your feedback.
> --Vahid
>