You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Mickael Maison <mi...@gmail.com> on 2018/04/03 16:22:05 UTC

Kafka Authorizer interface review

Hi all,

Over the past few months the IBM Message Hub team has "played quite a
bit" with the pluggable Authorizer interface and I'll try to give a
summary of our findings.

First when implementing a custom Authorizer, we found it hard having a
global view of all the Resource/Operation required for each ApiKey. We
ended up building a table (by looking at KafkaApis.scala) of all the
combinations that can be triggered. We posted this table in the wiki,
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Authorizations,
hopefully that will help others too.

We found the overview it provides necessary and it should probably be
in the docs/javadocs.

The biggest limitation for us were the permissions required to create
topics. This is what we targeted with KIP-277:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-277+-+Fine+Grained+ACL+for+CreateTopics+API

Some of our other findings:
- There is now way to distinguish between topic and record deletion.
If a Principal has Delete on a Topic, it can do both. With regulations
like GDPR, we can expect the DeleteRecords API to gain popularity and
it's a bit scary that it also allows to delete the topic.
- We also can't distinguish between DescribeLogDirs, DescribeAcls and
ListGroups as they both require Describe on the Cluster resource.
While ListGroups is pretty common for "normal" users, the other 2 are
a bit more on the admin side.
- OffsetCommit only requires Read on Group even though it's
technically a write operation. I think this was already discussed at
some point on the mailing list.

Changing permissions is an expensive process and so far we've not
attempted to come up with alternatives (apart from KIP-277). There is
also a balance between granularity and ease of use, requiring
administrators to set and maintain many permissions is not really an
improvement!

Thanks

Re: Kafka Authorizer interface review

Posted by Vahid S Hashemian <va...@us.ibm.com>.
Hi Mickael,

Thanks for detailed description on these authorization issues.
I agree they need to be reviewed and fixed in the areas you specified, or 
even at a higher that simplifies their maintenance as the matrix is 
expanded or needs to be modified.
FYI, KIP-231 also attempts at addressing the issue with ListGroups API in 
a backward compatible way.

--Vahid




From:   Mickael Maison <mi...@gmail.com>
To:     dev <de...@kafka.apache.org>
Date:   04/03/2018 09:22 AM
Subject:        Kafka Authorizer interface review



Hi all,

Over the past few months the IBM Message Hub team has "played quite a
bit" with the pluggable Authorizer interface and I'll try to give a
summary of our findings.

First when implementing a custom Authorizer, we found it hard having a
global view of all the Resource/Operation required for each ApiKey. We
ended up building a table (by looking at KafkaApis.scala) of all the
combinations that can be triggered. We posted this table in the wiki,
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_Kafka-2BAuthorizations&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=OsQVJInvqT_yR7tSsG5DRrkZg56iCzthkvSJiiLoF8k&s=z-seYP7KLIJFfpEX1yR0ficIYpXGIAiLOTb3gqaOa0k&e=
,
hopefully that will help others too.

We found the overview it provides necessary and it should probably be
in the docs/javadocs.

The biggest limitation for us were the permissions required to create
topics. This is what we targeted with KIP-277:
https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_KAFKA_KIP-2D277-2B-2D-2BFine-2BGrained-2BACL-2Bfor-2BCreateTopics-2BAPI&d=DwIBaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Q_itwloTQj3_xUKl7Nzswo6KE4Nj-kjJc7uSVcviKUc&m=OsQVJInvqT_yR7tSsG5DRrkZg56iCzthkvSJiiLoF8k&s=IQ2jGoMdgsjG4wkRaL5dV19m3eYMo4e9joyjani2pd8&e=


Some of our other findings:
- There is now way to distinguish between topic and record deletion.
If a Principal has Delete on a Topic, it can do both. With regulations
like GDPR, we can expect the DeleteRecords API to gain popularity and
it's a bit scary that it also allows to delete the topic.
- We also can't distinguish between DescribeLogDirs, DescribeAcls and
ListGroups as they both require Describe on the Cluster resource.
While ListGroups is pretty common for "normal" users, the other 2 are
a bit more on the admin side.
- OffsetCommit only requires Read on Group even though it's
technically a write operation. I think this was already discussed at
some point on the mailing list.

Changing permissions is an expensive process and so far we've not
attempted to come up with alternatives (apart from KIP-277). There is
also a balance between granularity and ease of use, requiring
administrators to set and maintain many permissions is not really an
improvement!

Thanks

Re: Kafka Authorizer interface review

Posted by Ted Yu <yu...@gmail.com>.
bq. you too are concerned about the current delete record/topic limitation

Yes.
I think this is a security hole.

On Tue, Apr 3, 2018 at 9:37 AM, Mickael Maison <mi...@gmail.com>
wrote:

> Yes this is indeed a typo!
>
> And yes we're considering filing another KIP but I thought collecting
> all our feedback and providing a full summary might be beneficial for
> others.
> I see you too are concerned about the current delete record/topic
> limitation.
>
> On Tue, Apr 3, 2018 at 5:26 PM, Ted Yu <yu...@gmail.com> wrote:
> > bq. There is now way to distinguish between topic and record deletion.
> >
> > I guess you meant 'no way' above.
> > I think deleting a topic has higher impact than deleting records.
> >
> > Have you considered filing KIP to distinguish the two operations ?
> >
> > Cheers
> >
> > On Tue, Apr 3, 2018 at 9:22 AM, Mickael Maison <mickael.maison@gmail.com
> >
> > wrote:
> >
> >> Hi all,
> >>
> >> Over the past few months the IBM Message Hub team has "played quite a
> >> bit" with the pluggable Authorizer interface and I'll try to give a
> >> summary of our findings.
> >>
> >> First when implementing a custom Authorizer, we found it hard having a
> >> global view of all the Resource/Operation required for each ApiKey. We
> >> ended up building a table (by looking at KafkaApis.scala) of all the
> >> combinations that can be triggered. We posted this table in the wiki,
> >> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Authorizations,
> >> hopefully that will help others too.
> >>
> >> We found the overview it provides necessary and it should probably be
> >> in the docs/javadocs.
> >>
> >> The biggest limitation for us were the permissions required to create
> >> topics. This is what we targeted with KIP-277:
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> 277+-+Fine+Grained+ACL+for+CreateTopics+API
> >>
> >> Some of our other findings:
> >> - There is now way to distinguish between topic and record deletion.
> >> If a Principal has Delete on a Topic, it can do both. With regulations
> >> like GDPR, we can expect the DeleteRecords API to gain popularity and
> >> it's a bit scary that it also allows to delete the topic.
> >> - We also can't distinguish between DescribeLogDirs, DescribeAcls and
> >> ListGroups as they both require Describe on the Cluster resource.
> >> While ListGroups is pretty common for "normal" users, the other 2 are
> >> a bit more on the admin side.
> >> - OffsetCommit only requires Read on Group even though it's
> >> technically a write operation. I think this was already discussed at
> >> some point on the mailing list.
> >>
> >> Changing permissions is an expensive process and so far we've not
> >> attempted to come up with alternatives (apart from KIP-277). There is
> >> also a balance between granularity and ease of use, requiring
> >> administrators to set and maintain many permissions is not really an
> >> improvement!
> >>
> >> Thanks
> >>
>

Re: Kafka Authorizer interface review

Posted by Mickael Maison <mi...@gmail.com>.
Yes this is indeed a typo!

And yes we're considering filing another KIP but I thought collecting
all our feedback and providing a full summary might be beneficial for
others.
I see you too are concerned about the current delete record/topic limitation.

On Tue, Apr 3, 2018 at 5:26 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. There is now way to distinguish between topic and record deletion.
>
> I guess you meant 'no way' above.
> I think deleting a topic has higher impact than deleting records.
>
> Have you considered filing KIP to distinguish the two operations ?
>
> Cheers
>
> On Tue, Apr 3, 2018 at 9:22 AM, Mickael Maison <mi...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> Over the past few months the IBM Message Hub team has "played quite a
>> bit" with the pluggable Authorizer interface and I'll try to give a
>> summary of our findings.
>>
>> First when implementing a custom Authorizer, we found it hard having a
>> global view of all the Resource/Operation required for each ApiKey. We
>> ended up building a table (by looking at KafkaApis.scala) of all the
>> combinations that can be triggered. We posted this table in the wiki,
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Authorizations,
>> hopefully that will help others too.
>>
>> We found the overview it provides necessary and it should probably be
>> in the docs/javadocs.
>>
>> The biggest limitation for us were the permissions required to create
>> topics. This is what we targeted with KIP-277:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 277+-+Fine+Grained+ACL+for+CreateTopics+API
>>
>> Some of our other findings:
>> - There is now way to distinguish between topic and record deletion.
>> If a Principal has Delete on a Topic, it can do both. With regulations
>> like GDPR, we can expect the DeleteRecords API to gain popularity and
>> it's a bit scary that it also allows to delete the topic.
>> - We also can't distinguish between DescribeLogDirs, DescribeAcls and
>> ListGroups as they both require Describe on the Cluster resource.
>> While ListGroups is pretty common for "normal" users, the other 2 are
>> a bit more on the admin side.
>> - OffsetCommit only requires Read on Group even though it's
>> technically a write operation. I think this was already discussed at
>> some point on the mailing list.
>>
>> Changing permissions is an expensive process and so far we've not
>> attempted to come up with alternatives (apart from KIP-277). There is
>> also a balance between granularity and ease of use, requiring
>> administrators to set and maintain many permissions is not really an
>> improvement!
>>
>> Thanks
>>

Re: Kafka Authorizer interface review

Posted by Ted Yu <yu...@gmail.com>.
bq. There is now way to distinguish between topic and record deletion.

I guess you meant 'no way' above.
I think deleting a topic has higher impact than deleting records.

Have you considered filing KIP to distinguish the two operations ?

Cheers

On Tue, Apr 3, 2018 at 9:22 AM, Mickael Maison <mi...@gmail.com>
wrote:

> Hi all,
>
> Over the past few months the IBM Message Hub team has "played quite a
> bit" with the pluggable Authorizer interface and I'll try to give a
> summary of our findings.
>
> First when implementing a custom Authorizer, we found it hard having a
> global view of all the Resource/Operation required for each ApiKey. We
> ended up building a table (by looking at KafkaApis.scala) of all the
> combinations that can be triggered. We posted this table in the wiki,
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Authorizations,
> hopefully that will help others too.
>
> We found the overview it provides necessary and it should probably be
> in the docs/javadocs.
>
> The biggest limitation for us were the permissions required to create
> topics. This is what we targeted with KIP-277:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 277+-+Fine+Grained+ACL+for+CreateTopics+API
>
> Some of our other findings:
> - There is now way to distinguish between topic and record deletion.
> If a Principal has Delete on a Topic, it can do both. With regulations
> like GDPR, we can expect the DeleteRecords API to gain popularity and
> it's a bit scary that it also allows to delete the topic.
> - We also can't distinguish between DescribeLogDirs, DescribeAcls and
> ListGroups as they both require Describe on the Cluster resource.
> While ListGroups is pretty common for "normal" users, the other 2 are
> a bit more on the admin side.
> - OffsetCommit only requires Read on Group even though it's
> technically a write operation. I think this was already discussed at
> some point on the mailing list.
>
> Changing permissions is an expensive process and so far we've not
> attempted to come up with alternatives (apart from KIP-277). There is
> also a balance between granularity and ease of use, requiring
> administrators to set and maintain many permissions is not really an
> improvement!
>
> Thanks
>