You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Adam Bellemare <ad...@gmail.com> on 2019/08/19 19:00:40 UTC
ACL for group creation?
Hi All
I am looking through the Confluent docs and core Kafka docs and don't see
an ACL for group creation:
https://docs.confluent.io/current/kafka/authorization.html#acl-format
and
https://kafka.apache.org/documentation/#security_authz
My scenario is simple: We use the consumer group as the means of
identifying a single application, including tooling for managing
application resets, offset management, lag monitoring, etc. We often have
situations where someone resets their consumer group by appending an
incremented integer ("cg" to "cg1"), but it throws the rest of the
monitoring and management tooling out of whack.
Is there a reason why we do not have ACL-based CREATE restrictions to a
particular consumer group? I am willing to do the work to implement this
and test it out, but I wanted to validate that there isn't a reason I am
missing.
Thanks
Adam
Re: ACL for group creation?
Posted by David Jacot <dj...@confluent.io>.
Hi Adam,
Assuming you have a cluster where access to any groups is denied by
default, you can allow
a principal to only have access to one group.
Something like the following:
kafka-acls.sh --add --allow-principal User:CN=my-principal --operation Read
--topic 'my-topic' --group 'my-group'
With this, *my-principal* will only be able to join the consumer group
*my-group*. If the principal
uses another group, such as *my-group2*, it will get:
org.apache.kafka.common.errors.GroupAuthorizationException: Not authorized
to access group: my-group2
I hope it helps.
Best,
David
On Thu, Aug 22, 2019 at 4:36 PM Adam Bellemare <ad...@gmail.com>
wrote:
> I see the consumer group much like the username in relational database
> access credentials. We routinely use the consumer group name as the means
> of identifying the consumer for operational things, such as alerting based
> on consumer group lag, autoscaling based on lag, and tooling around
> manipulating the consumer group offset for a particular service. The
> consumer group allows us to know, operationally, which offsets we are
> observing or manipulating, and ideally we would like to limit the set of
> consumer groups possible.
>
> In practice, I regularly see people simply append an incrementing integer
> to the end of their consumer group (cg, cg1, cg2, cg1234) when they want to
> reset their application, INSTEAD of following offset reset or kafka-streams
> application reset procedures. Sure, it would be nice to get everyone to
> follow recommended procedures, but people do what they CAN do, not what
> they're supposed to do. This has significant impact on the brokers and
> surrounding tooling (orphaned internal topics with indefinite retention,
> falsely-firing lag monitoring, broken auto-scaling), and instead of us
> building out layers of supportive tooling to isolate ourselves from it, why
> not simply set up ACLs to enforce which consumer groups an application can
> and cannot use?
>
> Does this need a KIP? Or is a bug report simply enough? I am unsure how
> compatibility would work, so I am leaning towards a KIP...
>
> Thanks
> Adam
>
> On Wed, Aug 21, 2019 at 5:59 PM Colin McCabe <cm...@apache.org> wrote:
>
> > I think it's worth considering separating out the permissions needed to
> > create a consumer group from the permissions needed to join one. We
> > distinguish these permissions for topics, and people generally find it
> > useful. We could start checking CREATE on GROUP, perhaps? It might be
> > hard to do in a compatible way.
> >
> > cheers,
> > Colin
> >
> >
> > On Wed, Aug 21, 2019, at 12:05, Adam Bellemare wrote:
> > > +users mailing list
> > >
> > > David,
> > >
> > > I don't think I really understand your email. Are you saying that this
> > can
> > > already be achieved only using the READ ACL?
> > >
> > > Thanks
> > > Adam
> > >
> > >
> > >
> > > On Wed, Aug 21, 2019 at 3:58 AM David Jacot <dj...@confluent.io>
> wrote:
> > >
> > > > Hello,
> > > >
> > > > It would be better to ask such question on the user mailing list.
> > > >
> > > > The reason is that the group is created automatically when a consumer
> > > > joins it. It is not created explicitly so it can be restricted.
> > > >
> > > > In your case, you could setup a ACL to authorize the application to
> > only
> > > > use the group you have defined. It would prevent the application from
> > > > creating new groups. (READ Acl on Group resource with a specific
> name).
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On Mon, Aug 19, 2019 at 9:01 PM Adam Bellemare <
> > adam.bellemare@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All
> > > > >
> > > > > I am looking through the Confluent docs and core Kafka docs and
> > don't see
> > > > > an ACL for group creation:
> > > > >
> > https://docs.confluent.io/current/kafka/authorization.html#acl-format
> > > > > and
> > > > > https://kafka.apache.org/documentation/#security_authz
> > > > >
> > > > > My scenario is simple: We use the consumer group as the means of
> > > > > identifying a single application, including tooling for managing
> > > > > application resets, offset management, lag monitoring, etc. We
> often
> > have
> > > > > situations where someone resets their consumer group by appending
> an
> > > > > incremented integer ("cg" to "cg1"), but it throws the rest of the
> > > > > monitoring and management tooling out of whack.
> > > > >
> > > > > Is there a reason why we do not have ACL-based CREATE restrictions
> > to a
> > > > > particular consumer group? I am willing to do the work to implement
> > this
> > > > > and test it out, but I wanted to validate that there isn't a reason
> > I am
> > > > > missing.
> > > > >
> > > > > Thanks
> > > > > Adam
> > > > >
> > > >
> > >
> >
>
Re: ACL for group creation?
Posted by Adam Bellemare <ad...@gmail.com>.
I see the consumer group much like the username in relational database
access credentials. We routinely use the consumer group name as the means
of identifying the consumer for operational things, such as alerting based
on consumer group lag, autoscaling based on lag, and tooling around
manipulating the consumer group offset for a particular service. The
consumer group allows us to know, operationally, which offsets we are
observing or manipulating, and ideally we would like to limit the set of
consumer groups possible.
In practice, I regularly see people simply append an incrementing integer
to the end of their consumer group (cg, cg1, cg2, cg1234) when they want to
reset their application, INSTEAD of following offset reset or kafka-streams
application reset procedures. Sure, it would be nice to get everyone to
follow recommended procedures, but people do what they CAN do, not what
they're supposed to do. This has significant impact on the brokers and
surrounding tooling (orphaned internal topics with indefinite retention,
falsely-firing lag monitoring, broken auto-scaling), and instead of us
building out layers of supportive tooling to isolate ourselves from it, why
not simply set up ACLs to enforce which consumer groups an application can
and cannot use?
Does this need a KIP? Or is a bug report simply enough? I am unsure how
compatibility would work, so I am leaning towards a KIP...
Thanks
Adam
On Wed, Aug 21, 2019 at 5:59 PM Colin McCabe <cm...@apache.org> wrote:
> I think it's worth considering separating out the permissions needed to
> create a consumer group from the permissions needed to join one. We
> distinguish these permissions for topics, and people generally find it
> useful. We could start checking CREATE on GROUP, perhaps? It might be
> hard to do in a compatible way.
>
> cheers,
> Colin
>
>
> On Wed, Aug 21, 2019, at 12:05, Adam Bellemare wrote:
> > +users mailing list
> >
> > David,
> >
> > I don't think I really understand your email. Are you saying that this
> can
> > already be achieved only using the READ ACL?
> >
> > Thanks
> > Adam
> >
> >
> >
> > On Wed, Aug 21, 2019 at 3:58 AM David Jacot <dj...@confluent.io> wrote:
> >
> > > Hello,
> > >
> > > It would be better to ask such question on the user mailing list.
> > >
> > > The reason is that the group is created automatically when a consumer
> > > joins it. It is not created explicitly so it can be restricted.
> > >
> > > In your case, you could setup a ACL to authorize the application to
> only
> > > use the group you have defined. It would prevent the application from
> > > creating new groups. (READ Acl on Group resource with a specific name).
> > >
> > > Best,
> > > David
> > >
> > > On Mon, Aug 19, 2019 at 9:01 PM Adam Bellemare <
> adam.bellemare@gmail.com>
> > > wrote:
> > >
> > > > Hi All
> > > >
> > > > I am looking through the Confluent docs and core Kafka docs and
> don't see
> > > > an ACL for group creation:
> > > >
> https://docs.confluent.io/current/kafka/authorization.html#acl-format
> > > > and
> > > > https://kafka.apache.org/documentation/#security_authz
> > > >
> > > > My scenario is simple: We use the consumer group as the means of
> > > > identifying a single application, including tooling for managing
> > > > application resets, offset management, lag monitoring, etc. We often
> have
> > > > situations where someone resets their consumer group by appending an
> > > > incremented integer ("cg" to "cg1"), but it throws the rest of the
> > > > monitoring and management tooling out of whack.
> > > >
> > > > Is there a reason why we do not have ACL-based CREATE restrictions
> to a
> > > > particular consumer group? I am willing to do the work to implement
> this
> > > > and test it out, but I wanted to validate that there isn't a reason
> I am
> > > > missing.
> > > >
> > > > Thanks
> > > > Adam
> > > >
> > >
> >
>
Re: ACL for group creation?
Posted by Colin McCabe <cm...@apache.org>.
I think it's worth considering separating out the permissions needed to create a consumer group from the permissions needed to join one. We distinguish these permissions for topics, and people generally find it useful. We could start checking CREATE on GROUP, perhaps? It might be hard to do in a compatible way.
cheers,
Colin
On Wed, Aug 21, 2019, at 12:05, Adam Bellemare wrote:
> +users mailing list
>
> David,
>
> I don't think I really understand your email. Are you saying that this can
> already be achieved only using the READ ACL?
>
> Thanks
> Adam
>
>
>
> On Wed, Aug 21, 2019 at 3:58 AM David Jacot <dj...@confluent.io> wrote:
>
> > Hello,
> >
> > It would be better to ask such question on the user mailing list.
> >
> > The reason is that the group is created automatically when a consumer
> > joins it. It is not created explicitly so it can be restricted.
> >
> > In your case, you could setup a ACL to authorize the application to only
> > use the group you have defined. It would prevent the application from
> > creating new groups. (READ Acl on Group resource with a specific name).
> >
> > Best,
> > David
> >
> > On Mon, Aug 19, 2019 at 9:01 PM Adam Bellemare <ad...@gmail.com>
> > wrote:
> >
> > > Hi All
> > >
> > > I am looking through the Confluent docs and core Kafka docs and don't see
> > > an ACL for group creation:
> > > https://docs.confluent.io/current/kafka/authorization.html#acl-format
> > > and
> > > https://kafka.apache.org/documentation/#security_authz
> > >
> > > My scenario is simple: We use the consumer group as the means of
> > > identifying a single application, including tooling for managing
> > > application resets, offset management, lag monitoring, etc. We often have
> > > situations where someone resets their consumer group by appending an
> > > incremented integer ("cg" to "cg1"), but it throws the rest of the
> > > monitoring and management tooling out of whack.
> > >
> > > Is there a reason why we do not have ACL-based CREATE restrictions to a
> > > particular consumer group? I am willing to do the work to implement this
> > > and test it out, but I wanted to validate that there isn't a reason I am
> > > missing.
> > >
> > > Thanks
> > > Adam
> > >
> >
>
Re: ACL for group creation?
Posted by Adam Bellemare <ad...@gmail.com>.
+users mailing list
David,
I don't think I really understand your email. Are you saying that this can
already be achieved only using the READ ACL?
Thanks
Adam
On Wed, Aug 21, 2019 at 3:58 AM David Jacot <dj...@confluent.io> wrote:
> Hello,
>
> It would be better to ask such question on the user mailing list.
>
> The reason is that the group is created automatically when a consumer
> joins it. It is not created explicitly so it can be restricted.
>
> In your case, you could setup a ACL to authorize the application to only
> use the group you have defined. It would prevent the application from
> creating new groups. (READ Acl on Group resource with a specific name).
>
> Best,
> David
>
> On Mon, Aug 19, 2019 at 9:01 PM Adam Bellemare <ad...@gmail.com>
> wrote:
>
> > Hi All
> >
> > I am looking through the Confluent docs and core Kafka docs and don't see
> > an ACL for group creation:
> > https://docs.confluent.io/current/kafka/authorization.html#acl-format
> > and
> > https://kafka.apache.org/documentation/#security_authz
> >
> > My scenario is simple: We use the consumer group as the means of
> > identifying a single application, including tooling for managing
> > application resets, offset management, lag monitoring, etc. We often have
> > situations where someone resets their consumer group by appending an
> > incremented integer ("cg" to "cg1"), but it throws the rest of the
> > monitoring and management tooling out of whack.
> >
> > Is there a reason why we do not have ACL-based CREATE restrictions to a
> > particular consumer group? I am willing to do the work to implement this
> > and test it out, but I wanted to validate that there isn't a reason I am
> > missing.
> >
> > Thanks
> > Adam
> >
>
Re: ACL for group creation?
Posted by Adam Bellemare <ad...@gmail.com>.
+users mailing list
David,
I don't think I really understand your email. Are you saying that this can
already be achieved only using the READ ACL?
Thanks
Adam
On Wed, Aug 21, 2019 at 3:58 AM David Jacot <dj...@confluent.io> wrote:
> Hello,
>
> It would be better to ask such question on the user mailing list.
>
> The reason is that the group is created automatically when a consumer
> joins it. It is not created explicitly so it can be restricted.
>
> In your case, you could setup a ACL to authorize the application to only
> use the group you have defined. It would prevent the application from
> creating new groups. (READ Acl on Group resource with a specific name).
>
> Best,
> David
>
> On Mon, Aug 19, 2019 at 9:01 PM Adam Bellemare <ad...@gmail.com>
> wrote:
>
> > Hi All
> >
> > I am looking through the Confluent docs and core Kafka docs and don't see
> > an ACL for group creation:
> > https://docs.confluent.io/current/kafka/authorization.html#acl-format
> > and
> > https://kafka.apache.org/documentation/#security_authz
> >
> > My scenario is simple: We use the consumer group as the means of
> > identifying a single application, including tooling for managing
> > application resets, offset management, lag monitoring, etc. We often have
> > situations where someone resets their consumer group by appending an
> > incremented integer ("cg" to "cg1"), but it throws the rest of the
> > monitoring and management tooling out of whack.
> >
> > Is there a reason why we do not have ACL-based CREATE restrictions to a
> > particular consumer group? I am willing to do the work to implement this
> > and test it out, but I wanted to validate that there isn't a reason I am
> > missing.
> >
> > Thanks
> > Adam
> >
>
Re: ACL for group creation?
Posted by David Jacot <dj...@confluent.io>.
Hello,
It would be better to ask such question on the user mailing list.
The reason is that the group is created automatically when a consumer
joins it. It is not created explicitly so it can be restricted.
In your case, you could setup a ACL to authorize the application to only
use the group you have defined. It would prevent the application from
creating new groups. (READ Acl on Group resource with a specific name).
Best,
David
On Mon, Aug 19, 2019 at 9:01 PM Adam Bellemare <ad...@gmail.com>
wrote:
> Hi All
>
> I am looking through the Confluent docs and core Kafka docs and don't see
> an ACL for group creation:
> https://docs.confluent.io/current/kafka/authorization.html#acl-format
> and
> https://kafka.apache.org/documentation/#security_authz
>
> My scenario is simple: We use the consumer group as the means of
> identifying a single application, including tooling for managing
> application resets, offset management, lag monitoring, etc. We often have
> situations where someone resets their consumer group by appending an
> incremented integer ("cg" to "cg1"), but it throws the rest of the
> monitoring and management tooling out of whack.
>
> Is there a reason why we do not have ACL-based CREATE restrictions to a
> particular consumer group? I am willing to do the work to implement this
> and test it out, but I wanted to validate that there isn't a reason I am
> missing.
>
> Thanks
> Adam
>