[DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Hi,

KIP-11 is open for discussion , I have updated the wiki with the design and open questions.

Thanks
Parth

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Hi Bosco,

Thanks for taking the time to review. I will update the doc with a policy
example and I will add CONNECT as an operation.

For Admin APIs, I agree the best thing to do right now is just wait for
KIP-4 to be submitted. I will update the doc to reflect the same.

Thanks
Parth

On 3/9/15, 11:48 AM, "Don Bosco Durai" <bo...@apache.org> wrote:

>Parth
>
>Overall it is looking good. Couple of questionsŠ
>
>- Can you give an example how the policies will look like in the default
>implementation?
>- In the operations, can we support ³CONNECT² also? This can be used
>during Session connection
>- Regarding access control for ³Topic Creation², since we can¹t do it on
>the server side, can we de-scope it for? And plan it as a future feature
>request?
>
>Thanks
>
>Bosco
>
> 
>
>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io> wrote:
>
>>Hi Parth,
>>            Thanks for putting this together. Overall it looks good to
>>            me. Although AdminUtils is a concern KIP-4  can probably fix
>>            that part.
>>Thanks,
>>Harsha
>>
>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>> Forgot to add links to wiki and jira.
>>> 
>>> Link to wiki:
>>> 
>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization
>>>+
>>>Interface
>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>> 
>>> Thanks
>>> Parth
>>> 
>>> From: Parth Brahmbhatt
>>> <pb...@hortonworks.com>>
>>> Date: Thursday, March 5, 2015 at 10:33 AM
>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org>"
>>> <de...@kafka.apache.org>>
>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>> 
>>> Hi,
>>> 
>>> KIP-11 is open for discussion , I have updated the wiki with the design
>>> and open questions.
>>> 
>>> Thanks
>>> Parth
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

I have added the following to list of open questions based on the hangout
discussion:

* The owner field of a topic in current proposal is set to the user who
created the topic and this user has all access to the topic. There was
suggestion on making this a list of users who can share ownership.
alternatively we can keep the user as a single entity but the user
creating the topic will have to ensure that the topic acls are configured
to allow admin access to all the other users that wants to assume
co-ownership.


It will be great if we can at least agree on the following things:
* The newly proposed broker configs, their types and names
* The Authorizer interface and the Acl structure
* The command line options being added, their name and types
* The new structure of topic config which is being stored in zookeeper

Thanks
Parth

On 4/15/15, 12:53 PM, "Parth Brahmbhatt" <pb...@hortonworks.com>
wrote:

>Yes, it can be turned off completely. We are proposing to add
>authorizer.class.name as a broker config. The value of this config can be
>null/unspecified (which is the default) in which case no authorization
>will be performed. It can be set to any FQCN of any class that implements
>Authorizer so you can plugin custom authorizers.
>
>Authentication is a different beast and you should look at other security
>related KIPs, here is the top level jira
>https://issues.apache.org/jira/browse/KAFKA-1682 and top level document
>https://cwiki.apache.org/confluence/display/KAFKA/Security
>
>Thanks
>Parth
>
>On 4/15/15, 11:56 AM, "Tong Li" <li...@us.ibm.com> wrote:
>
>>
>>Parth,
>>     If one wants to use his or her own access control including
>>authentication system, with this design what will be needed to be done?
>>Can
>>one completely turn this off so that the system behaves exactly same as
>>it
>>is today?
>>
>>Thanks.
>>
>>Tong
>>
>>Sent from my iPhone
>>
>>> On Apr 15, 2015, at 1:51 PM, Parth Brahmbhatt
>><pb...@hortonworks.com> wrote:
>>>
>>> Hi Michael,
>>>
>>> There is code in kafka codebase that reads and interprets the topic
>>config JSON which has acls, owner and logconfig properties. There are 3
>>use
>>cases that we are supporting with current proposal:
>>>
>>>   *   You use out of box simpleAcl authorizer which is tied to the acl
>>stored in topic config and the format is locked down.
>>>   *   You have a custom authorizer and a custom ACL store.
>>>Ranger/Argus
>>falls under this as they have their own acl store and ui that users use
>>to
>>configure acls on the cluster and cluster resources  like topic. It is
>>upto
>>the custom authorizer to leverage the kafka acl configs or completely
>>ignore them as they have set a user expectation that only acls configured
>>via their ui/system will be effective.
>>>   *   You have a custom authorizer but no custom Acl store. You are
>>completely tied to Acl structure that we have provided in out of box
>>implementation.
>>>
>>> Thanks
>>> Parth
>>>
>>> On 4/15/15, 10:31 AM, "Michael Herstine"
>><mh...@linkedin.com.INVALID>>
>>wrote:
>>>
>>> Hi Parth,
>>>
>>> One question that occurred to me at the end of today’s hangout: how
>>>tied
>>> are we to a particular ACL representation under your proposal? I know
>>that
>>> TopicConfigCache will just contain JSON— if a particular site decides
>>they
>>> want to represent their ACLs differently, and swap out the authorizer
>>> implementation, will that work?  I guess what I’m asking is whether
>>> there’s any code in the Kafka codebase that will interpret that JSON,
>>>or
>>> does that logic live exclusively in the authorizer?
>>>
>>> On 4/14/15, 10:56 PM, "Don Bosco Durai"
>><bo...@apache.org>> wrote:
>>>
>>> I also feel, having just IP would be more appropriate. Host lookup will
>>> unnecessary slow things down and would be insecure as you pointed out.
>>>
>>> With IP, it will be also able to setup policies (in future if needed)
>>with
>>> ranges or netmasks and it would be more scalable.
>>>
>>> Bosco
>>>
>>>
>>> On 4/14/15, 1:40 PM, "Michael Herstine"
>><mh...@linkedin.com.INVALID>>
>>> wrote:
>>>
>>> Hi Parth,
>>>
>>> Sorry to chime in so late, but I’ve got a minor question on the KIP.
>>>
>>> Several methods take a parameter named “host” of type String. Is that
>>> intended to be a hostname, or an IP address? If the former, I’m curious
>>> as
>>> to how that’s found (in my experience, when accepting an incoming
>>>socket
>>> connection, you only know the IP address, and there isn’t a way to map
>>> that to a hostname without a round trip to a DNS server, which is
>>> insecure
>>> anyway).
>>>
>>>
>>> On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
>><pb...@hortonworks.com>>
>>> wrote:
>>>
>>> Hi all,
>>>
>>> I have modified the KIP to reflect the recent change request from the
>>> reviewers. I have been working on the code and I have the server side
>>> code
>>> for authorization ready. I am now modifying the command line utilities.
>>> I
>>> would really appreciate if some of the committers can spend sometime to
>>> review the KIP so we can make progress on this.
>>>
>>> Thanks
>>> Parth
>>>
>>> On 3/18/15, 2:20 PM, "Michael Herstine"
>><mh...@linkedin.com.INVALID>>
>>> wrote:
>>>
>>> Hi Parth,
>>>
>>> Thanks! A few questions:
>>>
>>> 1. Do you want to permit rules in your ACLs that DENY access as well as
>>> ALLOW? This can be handy setting up rules that have exceptions. E.g.
>>> “Allow principal P to READ resource R from all hosts” with “Deny
>>> principal
>>> P READ access to resource R from host H1” in combination would allow P
>>> to
>>> READ R from all hosts *except* H1.
>>>
>>> 2. When a topic is newly created, will there be an ACL created for it?
>>> If
>>> not, would that not deny subsequent access to it?
>>>
>>> (nit) Maybe use Principal instead of String to represent principals?
>>>
>>>
>>> On 3/9/15, 11:48 AM, "Don Bosco Durai"
>><bo...@apache.org>> wrote:
>>>
>>> Parth
>>>
>>> Overall it is looking good. Couple of questionsŠ
>>>
>>> - Can you give an example how the policies will look like in the
>>> default
>>> implementation?
>>> - In the operations, can we support ³CONNECT² also? This can be used
>>> during Session connection
>>> - Regarding access control for ³Topic Creation², since we can¹t do it
>>> on
>>> the server side, can we de-scope it for? And plan it as a future
>>> feature
>>> request?
>>>
>>> Thanks
>>>
>>> Bosco
>>>
>>>
>>> On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io>>
>>wrote:
>>>
>>> Hi Parth,
>>>             Thanks for putting this together. Overall it looks good
>>> to
>>>             me. Although AdminUtils is a concern KIP-4  can probably
>>> fix
>>>             that part.
>>> Thanks,
>>> Harsha
>>>
>>> On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>> Forgot to add links to wiki and jira.
>>> Link to wiki:
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
>>> t
>>> i
>>> o
>>> n
>>> +
>>> Interface
>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>> Thanks
>>> Parth
>>> From: Parth Brahmbhatt
>>>
>><pb...@hortonworks.com><mailto:p
>>b
>>rahmbhatt@hortonworks.com>>
>>
>>> Date: Thursday, March 5, 2015 at 10:33 AM
>>> To:
>>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:dev@kafka.apach
>>e
>>.org>"
>>
>>>
>><de...@kafka.apache.org><mailto:dev@kafka.apach
>>e
>>.org>>
>>
>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>> Hi,
>>> KIP-11 is open for discussion , I have updated the wiki with the
>>> design
>>> and open questions.
>>> Thanks
>>> Parth
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Yes, it can be turned off completely. We are proposing to add
authorizer.class.name as a broker config. The value of this config can be
null/unspecified (which is the default) in which case no authorization
will be performed. It can be set to any FQCN of any class that implements
Authorizer so you can plugin custom authorizers.

Authentication is a different beast and you should look at other security
related KIPs, here is the top level jira
https://issues.apache.org/jira/browse/KAFKA-1682 and top level document
https://cwiki.apache.org/confluence/display/KAFKA/Security

Thanks
Parth

On 4/15/15, 11:56 AM, "Tong Li" <li...@us.ibm.com> wrote:

>
>Parth,
>     If one wants to use his or her own access control including
>authentication system, with this design what will be needed to be done?
>Can
>one completely turn this off so that the system behaves exactly same as it
>is today?
>
>Thanks.
>
>Tong
>
>Sent from my iPhone
>
>> On Apr 15, 2015, at 1:51 PM, Parth Brahmbhatt
><pb...@hortonworks.com> wrote:
>>
>> Hi Michael,
>>
>> There is code in kafka codebase that reads and interprets the topic
>config JSON which has acls, owner and logconfig properties. There are 3
>use
>cases that we are supporting with current proposal:
>>
>>   *   You use out of box simpleAcl authorizer which is tied to the acl
>stored in topic config and the format is locked down.
>>   *   You have a custom authorizer and a custom ACL store.  Ranger/Argus
>falls under this as they have their own acl store and ui that users use to
>configure acls on the cluster and cluster resources  like topic. It is
>upto
>the custom authorizer to leverage the kafka acl configs or completely
>ignore them as they have set a user expectation that only acls configured
>via their ui/system will be effective.
>>   *   You have a custom authorizer but no custom Acl store. You are
>completely tied to Acl structure that we have provided in out of box
>implementation.
>>
>> Thanks
>> Parth
>>
>> On 4/15/15, 10:31 AM, "Michael Herstine"
><mh...@linkedin.com.INVALID>>
>wrote:
>>
>> Hi Parth,
>>
>> One question that occurred to me at the end of today’s hangout: how tied
>> are we to a particular ACL representation under your proposal? I know
>that
>> TopicConfigCache will just contain JSON— if a particular site decides
>they
>> want to represent their ACLs differently, and swap out the authorizer
>> implementation, will that work?  I guess what I’m asking is whether
>> there’s any code in the Kafka codebase that will interpret that JSON, or
>> does that logic live exclusively in the authorizer?
>>
>> On 4/14/15, 10:56 PM, "Don Bosco Durai"
><bo...@apache.org>> wrote:
>>
>> I also feel, having just IP would be more appropriate. Host lookup will
>> unnecessary slow things down and would be insecure as you pointed out.
>>
>> With IP, it will be also able to setup policies (in future if needed)
>with
>> ranges or netmasks and it would be more scalable.
>>
>> Bosco
>>
>>
>> On 4/14/15, 1:40 PM, "Michael Herstine"
><mh...@linkedin.com.INVALID>>
>> wrote:
>>
>> Hi Parth,
>>
>> Sorry to chime in so late, but I’ve got a minor question on the KIP.
>>
>> Several methods take a parameter named “host” of type String. Is that
>> intended to be a hostname, or an IP address? If the former, I’m curious
>> as
>> to how that’s found (in my experience, when accepting an incoming socket
>> connection, you only know the IP address, and there isn’t a way to map
>> that to a hostname without a round trip to a DNS server, which is
>> insecure
>> anyway).
>>
>>
>> On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
><pb...@hortonworks.com>>
>> wrote:
>>
>> Hi all,
>>
>> I have modified the KIP to reflect the recent change request from the
>> reviewers. I have been working on the code and I have the server side
>> code
>> for authorization ready. I am now modifying the command line utilities.
>> I
>> would really appreciate if some of the committers can spend sometime to
>> review the KIP so we can make progress on this.
>>
>> Thanks
>> Parth
>>
>> On 3/18/15, 2:20 PM, "Michael Herstine"
><mh...@linkedin.com.INVALID>>
>> wrote:
>>
>> Hi Parth,
>>
>> Thanks! A few questions:
>>
>> 1. Do you want to permit rules in your ACLs that DENY access as well as
>> ALLOW? This can be handy setting up rules that have exceptions. E.g.
>> “Allow principal P to READ resource R from all hosts” with “Deny
>> principal
>> P READ access to resource R from host H1” in combination would allow P
>> to
>> READ R from all hosts *except* H1.
>>
>> 2. When a topic is newly created, will there be an ACL created for it?
>> If
>> not, would that not deny subsequent access to it?
>>
>> (nit) Maybe use Principal instead of String to represent principals?
>>
>>
>> On 3/9/15, 11:48 AM, "Don Bosco Durai"
><bo...@apache.org>> wrote:
>>
>> Parth
>>
>> Overall it is looking good. Couple of questionsŠ
>>
>> - Can you give an example how the policies will look like in the
>> default
>> implementation?
>> - In the operations, can we support ³CONNECT² also? This can be used
>> during Session connection
>> - Regarding access control for ³Topic Creation², since we can¹t do it
>> on
>> the server side, can we de-scope it for? And plan it as a future
>> feature
>> request?
>>
>> Thanks
>>
>> Bosco
>>
>>
>> On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io>>
>wrote:
>>
>> Hi Parth,
>>             Thanks for putting this together. Overall it looks good
>> to
>>             me. Although AdminUtils is a concern KIP-4  can probably
>> fix
>>             that part.
>> Thanks,
>> Harsha
>>
>> On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>> Forgot to add links to wiki and jira.
>> Link to wiki:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
>> t
>> i
>> o
>> n
>> +
>> Interface
>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>> Thanks
>> Parth
>> From: Parth Brahmbhatt
>>
><pb...@hortonworks.com><mailto:pb
>rahmbhatt@hortonworks.com>>
>
>> Date: Thursday, March 5, 2015 at 10:33 AM
>> To:
>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:dev@kafka.apache
>.org>"
>
>>
><de...@kafka.apache.org><mailto:dev@kafka.apache
>.org>>
>
>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>> Hi,
>> KIP-11 is open for discussion , I have updated the wiki with the
>> design
>> and open questions.
>> Thanks
>> Parth
>>
>>
>>
>>
>>
>>
>>
>>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Changed Edit to Alter.

I did not think about it that way but Sriharsha raised the same point in a
private conversation. I did not think about it that way but I agree it
makes sense. If no one objects I think in default implementation we can
infer that if user have READ or WRITE access he gets DESCRIBE for free.

Thanks
Parth

On 4/21/15, 2:04 PM, "Jay Kreps" <ja...@gmail.com> wrote:

>Also, I think I may have missed this but does READ imply you also have
>DESCRIBE? A reader will need access to both read offsets (to determine
>their own initial position) as well as commit offsets. Currently, though
>fetching offsets is under DESCRIBE only and commit offsets is under READ.
>If READ=>DESCRIBE are there any other implied permissions like that?
>
>-Jay
>
>On Tue, Apr 21, 2015 at 1:59 PM, Jay Kreps <ja...@gmail.com> wrote:
>
>> Hey Parth,
>>
>> Great write-up!
>>
>> One super minor thing: could we change the "EDIT" permission to be
>>called
>> "ALTER"? The request name in KIP-4 is Alter and the command line tool
>>has
>> always been alter (or we could go the other way and change those to
>>EDIT).
>> Not sure that one is any better than the other but consistency is always
>> nice.
>>
>> -Jay
>>
>> On Tue, Apr 21, 2015 at 9:06 AM, Jun Rao <ju...@confluent.io> wrote:
>>
>>> Harsha, Parth,
>>>
>>> Thanks for the clarification. This makes sense. Perhaps we can clarify
>>>the
>>> meaning of those rules in the wiki.
>>>
>>> Related to this, it seems that we need to support wildcard in
>>>cli/request
>>> protocol for topics?
>>>
>>> Jun
>>>
>>> On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>>> pbrahmbhatt@hortonworks.com> wrote:
>>>
>>> > The iptables on unix supports the DENY operator, not that it should
>>> > matter. The deny operator can also be used to specify ³allow user1 to
>>> READ
>>> > from topic1 from all hosts but host1,host2². Again we could add a
>>>host
>>> > group semantic and extra complexity around that, not sure if its
>>>worth
>>> it.
>>> > In addition with DENY operator you are now not forced to create a
>>> special
>>> > group just to support the authorization use case. I am not convinced
>>> that
>>> > the operator it self is really all that confusing. There are 3
>>>practical
>>> > use cases:
>>> > - Resource with no acl what so ever -> allow access to everyone (
>>>just
>>> for
>>> > backward compatibility, I would much rather fail close and force
>>>users
>>> to
>>> > explicitly grant acls that allows access to all users.)
>>> > - Resource with some acl attached -> only users that have a matching
>>> allow
>>> > acl are allowed (i.e. ³allow READ access to topic1 to user1 from all
>>> > hosts², only user1 has READ access and no other user has access of
>>>any
>>> > kind)
>>> > - Resource with some allow and some deny acl attached -> users are
>>> allowed
>>> > to perform operation only when they satisfy allow acl and do not have
>>> > conflicting deny acl. Users that have no acl(allow or deny) will
>>>still
>>> not
>>> > have any access. (i.e. ³allow READ access to topic1 to user1 from all
>>> > hosts except host1 and host², only user1 has access but not from
>>>host1
>>> an
>>> > host2)
>>> >
>>> > I think we need to make a decision on deny primarily because with
>>> > introduction of acl management API, Acl is now a public class that
>>>will
>>> be
>>> > used by Ranger/Santry and other authroization providers. In Current
>>> design
>>> > the acl has a permissionType enum field with possible values of Allow
>>> and
>>> > Deny. If we chose to remove deny we can assume all acls to be of
>>>allow
>>> > type and remove the permissionType field completely.
>>> >
>>> > Thanks
>>> > Parth
>>> >
>>> > On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>>> >
>>> > >I think thats how its done in pretty much any system I can think of.
>>> > >
>>> >
>>> >
>>>
>>
>>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Jay Kreps <ja...@gmail.com>]

Also, I think I may have missed this but does READ imply you also have
DESCRIBE? A reader will need access to both read offsets (to determine
their own initial position) as well as commit offsets. Currently, though
fetching offsets is under DESCRIBE only and commit offsets is under READ.
If READ=>DESCRIBE are there any other implied permissions like that?

-Jay

On Tue, Apr 21, 2015 at 1:59 PM, Jay Kreps <ja...@gmail.com> wrote:

> Hey Parth,
>
> Great write-up!
>
> One super minor thing: could we change the "EDIT" permission to be called
> "ALTER"? The request name in KIP-4 is Alter and the command line tool has
> always been alter (or we could go the other way and change those to EDIT).
> Not sure that one is any better than the other but consistency is always
> nice.
>
> -Jay
>
> On Tue, Apr 21, 2015 at 9:06 AM, Jun Rao <ju...@confluent.io> wrote:
>
>> Harsha, Parth,
>>
>> Thanks for the clarification. This makes sense. Perhaps we can clarify the
>> meaning of those rules in the wiki.
>>
>> Related to this, it seems that we need to support wildcard in cli/request
>> protocol for topics?
>>
>> Jun
>>
>> On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>> pbrahmbhatt@hortonworks.com> wrote:
>>
>> > The iptables on unix supports the DENY operator, not that it should
>> > matter. The deny operator can also be used to specify ³allow user1 to
>> READ
>> > from topic1 from all hosts but host1,host2². Again we could add a host
>> > group semantic and extra complexity around that, not sure if its worth
>> it.
>> > In addition with DENY operator you are now not forced to create a
>> special
>> > group just to support the authorization use case. I am not convinced
>> that
>> > the operator it self is really all that confusing. There are 3 practical
>> > use cases:
>> > - Resource with no acl what so ever -> allow access to everyone ( just
>> for
>> > backward compatibility, I would much rather fail close and force users
>> to
>> > explicitly grant acls that allows access to all users.)
>> > - Resource with some acl attached -> only users that have a matching
>> allow
>> > acl are allowed (i.e. ³allow READ access to topic1 to user1 from all
>> > hosts², only user1 has READ access and no other user has access of any
>> > kind)
>> > - Resource with some allow and some deny acl attached -> users are
>> allowed
>> > to perform operation only when they satisfy allow acl and do not have
>> > conflicting deny acl. Users that have no acl(allow or deny) will still
>> not
>> > have any access. (i.e. ³allow READ access to topic1 to user1 from all
>> > hosts except host1 and host², only user1 has access but not from host1
>> an
>> > host2)
>> >
>> > I think we need to make a decision on deny primarily because with
>> > introduction of acl management API, Acl is now a public class that will
>> be
>> > used by Ranger/Santry and other authroization providers. In Current
>> design
>> > the acl has a permissionType enum field with possible values of Allow
>> and
>> > Deny. If we chose to remove deny we can assume all acls to be of allow
>> > type and remove the permissionType field completely.
>> >
>> > Thanks
>> > Parth
>> >
>> > On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>> >
>> > >I think thats how its done in pretty much any system I can think of.
>> > >
>> >
>> >
>>
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Jay Kreps <ja...@gmail.com>]

Hey Parth,

Great write-up!

One super minor thing: could we change the "EDIT" permission to be called
"ALTER"? The request name in KIP-4 is Alter and the command line tool has
always been alter (or we could go the other way and change those to EDIT).
Not sure that one is any better than the other but consistency is always
nice.

-Jay

On Tue, Apr 21, 2015 at 9:06 AM, Jun Rao <ju...@confluent.io> wrote:

> Harsha, Parth,
>
> Thanks for the clarification. This makes sense. Perhaps we can clarify the
> meaning of those rules in the wiki.
>
> Related to this, it seems that we need to support wildcard in cli/request
> protocol for topics?
>
> Jun
>
> On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
> pbrahmbhatt@hortonworks.com> wrote:
>
> > The iptables on unix supports the DENY operator, not that it should
> > matter. The deny operator can also be used to specify ³allow user1 to
> READ
> > from topic1 from all hosts but host1,host2². Again we could add a host
> > group semantic and extra complexity around that, not sure if its worth
> it.
> > In addition with DENY operator you are now not forced to create a special
> > group just to support the authorization use case. I am not convinced that
> > the operator it self is really all that confusing. There are 3 practical
> > use cases:
> > - Resource with no acl what so ever -> allow access to everyone ( just
> for
> > backward compatibility, I would much rather fail close and force users to
> > explicitly grant acls that allows access to all users.)
> > - Resource with some acl attached -> only users that have a matching
> allow
> > acl are allowed (i.e. ³allow READ access to topic1 to user1 from all
> > hosts², only user1 has READ access and no other user has access of any
> > kind)
> > - Resource with some allow and some deny acl attached -> users are
> allowed
> > to perform operation only when they satisfy allow acl and do not have
> > conflicting deny acl. Users that have no acl(allow or deny) will still
> not
> > have any access. (i.e. ³allow READ access to topic1 to user1 from all
> > hosts except host1 and host², only user1 has access but not from host1 an
> > host2)
> >
> > I think we need to make a decision on deny primarily because with
> > introduction of acl management API, Acl is now a public class that will
> be
> > used by Ranger/Santry and other authroization providers. In Current
> design
> > the acl has a permissionType enum field with possible values of Allow and
> > Deny. If we chose to remove deny we can assume all acls to be of allow
> > type and remove the permissionType field completely.
> >
> > Thanks
> > Parth
> >
> > On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
> >
> > >I think thats how its done in pretty much any system I can think of.
> > >
> >
> >
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Tom Graves <tg...@yahoo.com.INVALID>]

Sorry I must be missing it, that is the doc I'm looking at.
I don't see how you specify the operation?
I don't see a normal --host that goes with list.  I see revokehost and granthost, neither I would think apply to list since --principal. says it applies to --list.
Copying text here in case I'm looking at the wrong thing:
   
   - kafka-acl.sh --help   
A tool to manage acls for kafka cluster and topics. Following are the options   
--topic <topicname> name of topic whose acls you want to add/remove.   
--cluster this will add/edit cluster level acls where you can control which users can create topics or list all topics for the cluster or send control messages to brokers.   
--add indicates you are trying to add Acl   
--remove indicates you are trying to remove acl   
--grantprincipal <principalType: principalName> a comma separated list of principalType: principalName where currently supported principalType is "user". When this option is not present but --granthost is specified it will default to * : * which is wild card that matches all types and all users.    
--granthost <host1,host2> a comma separated list of hosts from which the --grantpricipal is allowed the actions. When this option is not present but --grantprincipal is specified, it defaults to * which is wildcard matching all hosts.   
--revokeprincipal <principalType: principalName> a comma separated list of principalType: principalName where currently supported principalType is "user". When this option is not present but --revokehost  is specified , it defaults to * : * which is wild card that matches all types and all users. Revoke should only be used to limit the access added by some other grants. A revoke without an accompanying grant is meaningless as by default only principals in grant lists are allowed access based on their acls and everyone else is denied.   
----revokehost <host1,host2> a comma separated list of hosts from which the --revokeprincipal will be denied actions. When this option is not present but --revokeprincipal is specified, it defaults to * which is wildcard matching all hosts.   
--removeAll removes all acls for a topic, only works with topic and not with cluster.   
--list list all acls   
--principal <principalType: principalName> Used only with --list to list all acls for a principal
 



     On Wednesday, April 22, 2015 1:08 PM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:
   

 Please see all the available options here https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface#KIP-11-AuthorizationInterface-AclManagement(CLI) . I think it covers both hosts and operations and allows to specify a list for both.
ThanksParth
From: Tom Graves <tg...@yahoo.com>
Reply-To: Tom Graves <tg...@yahoo.com>
Date: Wednesday, April 22, 2015 at 11:02 AM
To: Parth Brahmbhatt <pb...@hortonworks.com>, "dev@kafka.apache.org" <de...@kafka.apache.org>
Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security

Thanks for the explanations Parth.
On the configs questions, the way I see it is its more likely to accidentally give everyone access, especially since you have to run a separate command to change the acls. If there was some config for defaults, a cluster admin could change that to be nobody or certain set of users, then grant others permissions.  This would also remove the race between commands.  This is something you can always add later though if people request it. 
So in kafka-acl.sh how do I actually tell it what the operation is?kafka-acl.sh --topic testtopic --add --grandprincipal user:joe,user:kate 
where does READ, WRITE, etc go?  Can specify as a list so I don't have to run this a bunch of times for each.
Do you want to have a --host option for --list so that admins could see what acls apply to specific host(s)?
Tom


On Wednesday, April 22, 2015 11:38 AM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:



FYI, I have modified the KIP to include group as resource. In order to
access “joinGroup” and “commitOFfset” APIs the user will need a read
permission on topic and WRITE permission on group.

I plan to open a VOTE thread by noon if there are no more concerns.

Thanks
Parth

On 4/22/15, 9:03 AM, "Tom Graves" <tg...@yahoo.com.INVALID> wrote:

>Hey everyone,
>Sorry to jump in on the conversation so late. I'm new to Kafka. I'll
>apologize in advance if you have already covered some of my questions.  I
>read through the wiki and had some comments and questions.
>1) public enum Operation needs EDIT changed to ALTER

>    Done.

>2) Does the Authorizer class need a setAcls?  Rather then just add to be
>able to set to explicit list and overwrite what was there?  I see the
>kafka-acl.sh lists a removeall so I guess you could do removeall and then
>add.  I also don't see a removeall in the Authorizer class, is it going
>to loop through them all to remove each one?

    There is an overloaded version of removeAcls in the interface that takes
in resource as the only input and as described in the javadoc all the acls
attached to that resource will be deleted. To cover the setAcl use case
the caller can first call remove and then add.

>3) Can someone tell me what the use case to do acls based on the hosts?
>I can see some possibilities just wondering if we can concrete ones where
>one user is allowed from one host but not another.

    I am not sure if I understand the question given the use case you
described in your question is what we are trying to cover with use of
hosts in Acl. There are some additional use cases like “allow access to
any user from host1,host2” but I think primarily it gives the admins the
ability to define acls at a more granular level.

>4) I'm a bit unclear how the "resource" works in the Authorizer class.
>From what I see we have 2 resources - topics and cluster.  If I want to
>add an acl to allow "joe" to CREATE for the cluster then I call addAcls
>with  Acl("user: joe", ALLOW, Set(*), Set(CREATE)) and "cluster"?  What
>if I want to call addAcls for DESCRIBE on a topic?  Is the resource then
>"topic" or is it the topic name?

    We now have 3 resources(added group), please see the updated doc. The
CREATE acl that you described is correct. For any topic operation you
should use topic name as the resource name and for group the user will
provide groupId as resource name.

>5) reassigning partitions is a CLUSTER_ACTION or superuser?  Its not
>totally clear to me the differences between these.  what about increasing
># of partitions?

    I see this as an alter topic operation so it is at topic level and the
user must have alter permissions on topic.    

>6) groups are mentioned, are we supporting right away or is that a follow
>on item? (is there going to be a kafka.supergroups)

    I think it can be a separate jira just for braking down the code review
in smaller chunk. We will support it in first version but I think if we
can not do it for any reason that should not block a release with all the
other authZ work. We made deliberate design choices (like introducing a
principalType in KafkaPrinciapl) to allow supporting groups as an
incremental change.

>7) Are there config options for setting acls when I create my topic?  Or
>do I have to create my topic and then run the kafka-acl.sh script to set
>them?  Although its very small, there would be possible race there that
>someone could start producing to topic before acls are set.

    We discussed this yesterday and we agreed to go with kafka-acl.sh. Yes
there is a very very small window of vulnerability but I think that really
does not warrant to change the decision in this case.

>8) are there configs for cluster level acl defaults?  Or does it default
>to superusers on bringing up new cluster and you have to modify with cli.
>thanks,Tom

    No defaults, the default is superusers will have full access. I don’t
think making assumptions about ones security requirement should be our
burden.

>
>    On Tuesday, April 21, 2015 7:10 PM, Parth Brahmbhatt
><pb...@hortonworks.com> wrote:
>  
>
> I have added the notes to KIP-11 Open question sections.
>
>Thanks
>Parth
>
>On 4/21/15, 4:49 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>
>>Adding my notes from today's call to the thread:
>>
>>** Deny or Allow all by default? We will add a configuration to
>>control this. The configuration will default to “allow” for backward
>>compatibility. Security admins can set it to "deny"
>>
>>** Storing ACLs for default authorizers: We'll store them in ZK. We'll
>>support pointing the authorizer to any ZK.
>>The use of ZK will be internal to the default authorizer. Authorizer
>>reads ACLs from cache every hour. We proposed having mechanism
>>(possibly via new ZK node) to tell broker to refresh the cache
>>immediately.
>>
>>** Support deny as permission type - we agreed to keep this.
>>
>>** Mapping operations to API: We may need to add Group as a resource,
>>with JoinGroup and OffsetCommit require privilege on the consumer
>>group.
>>This can be something we pass now and authorizers can support in
>>future. - Jay will write specifics to the mailing list discussion.
>>
>>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps <ja...@gmail.com> wrote:
>>> Following up on the KIP discussion. Two options for authorizing
>>>consumers
>>> to read topic "t" as part of group "g":
>>> 1. READ permission on resource /topic/t
>>> 2. READ permission on resource /topic/t AND WRITE permission on
>>>/group/g
>>>
>>> The advantage of (1) is that it is simpler. The disadvantage is that
>>>any
>>> member of any group that reads from t can commit offsets as any other
>>> member of a different group. This doesn't effect data security (who can
>>> access what) but it is a bit of a management issue--a malicious person
>>>can
>>> cause data loss or duplicates for another consumer by committing
>>>offset.
>>>
>>> I think I favor (2) but it's worth it to think it through.
>>>
>>> -Jay
>>>
>>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
>>> pbrahmbhatt@hortonworks.com> wrote:
>>>
>>>> Hey Jun,
>>>>
>>>> Yes and we support wild cards for all acl entities principal, hosts
>>>>and
>>>> operation.
>>>>
>>>> Thanks
>>>> Parth
>>>>
>>>> On 4/21/15, 9:06 AM, "Jun Rao" <ju...@confluent.io> wrote:
>>>>
>>>> >Harsha, Parth,
>>>> >
>>>> >Thanks for the clarification. This makes sense. Perhaps we can
>>>>clarify the
>>>> >meaning of those rules in the wiki.
>>>> >
>>>> >Related to this, it seems that we need to support wildcard in
>>>>cli/request
>>>> >protocol for topics?
>>>> >
>>>> >Jun
>>>> >
>>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>>>> >pbrahmbhatt@hortonworks.com> wrote:
>>>> >
>>>> >> The iptables on unix supports the DENY operator, not that it should
>>>> >> matter. The deny operator can also be used to specify ³allow user1
>>>>to
>>>> >>READ
>>>> >> from topic1 from all hosts but host1,host2². Again we could add a
>>>>host
>>>> >> group semantic and extra complexity around that, not sure if its
>>>>worth
>>>> >>it.
>>>> >> In addition with DENY operator you are now not forced to create a
>>>> >>special
>>>> >> group just to support the authorization use case. I am not
>>>>convinced
>>>> >>that
>>>> >> the operator it self is really all that confusing. There are 3
>>>>practical
>>>> >> use cases:
>>>> >> - Resource with no acl what so ever -> allow access to everyone (
>>>>just
>>>> >>for
>>>> >> backward compatibility, I would much rather fail close and force
>>>>users
>>>> >>to
>>>> >> explicitly grant acls that allows access to all users.)
>>>> >> - Resource with some acl attached -> only users that have a
>>>>matching
>>>> >>allow
>>>> >> acl are allowed (i.e. ³allow READ access to topic1 to user1 from
>>>>all
>>>> >> hosts², only user1 has READ access and no other user has access of
>>>>any
>>>> >> kind)
>>>> >> - Resource with some allow and some deny acl attached -> users are
>>>> >>allowed
>>>> >> to perform operation only when they satisfy allow acl and do not
>>>>have
>>>> >> conflicting deny acl. Users that have no acl(allow or deny) will
>>>>still
>>>> >>not
>>>> >> have any access. (i.e. ³allow READ access to topic1 to user1 from
>>>>all
>>>> >> hosts except host1 and host², only user1 has access but not from
>>>>host1
>>>> >>an
>>>> >> host2)
>>>> >>
>>>> >> I think we need to make a decision on deny primarily because with
>>>> >> introduction of acl management API, Acl is now a public class that
>>>>will
>>>> >>be
>>>> >> used by Ranger/Santry and other authroization providers. In Current
>>>> >>design
>>>> >> the acl has a permissionType enum field with possible values of
>>>>Allow
>>>> >>and
>>>> >> Deny. If we chose to remove deny we can assume all acls to be of
>>>>allow
>>>> >> type and remove the permissionType field completely.
>>>> >>
>>>> >> Thanks
>>>> >> Parth
>>>> >>
>>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>>>> >>
>>>> >> >I think thats how its done in pretty much any system I can think
>>>>of.
>>>> >> >
>>>> >>
>>>> >>
>>>>
>>>>
>
>
>
>  





  

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Tom Graves <tg...@yahoo.com.INVALID>]

What about specifying the operation when I go to add or remove?
Tom 


     On Wednesday, April 22, 2015 1:23 PM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:
   

 Sorry I missed your last questions. I am +0 on adding ―host option for
―list, we could add it for symmetry. Again if this is only a CLI change it
can be added later if you mean adding this in authorizer interface then we
should make a decision now.

Given a choice I would like to actually keep only one option which is
resource based get (remove even the get based on principal). I see those
(getAcl for principal or host) as special filtering case which can easily
be achieved by a third party tool by doing "list all topics" and calling
getAcls for each topic and applying filtering logic on that.  I really
don’t see the need to make those first class citizens of the authorizer
interface given these kind of queries will be issued outside of broker JVM
so they will not benefit from the caching and because the storage will be
indexed on resource both these options even as a first class API will just
scan all topic acls and apply filtering logic.

Thanks
Parth

On 4/22/15, 11:08 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>
wrote:

>Please see all the available options here
>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+I
>nterface#KIP-11-AuthorizationInterface-AclManagement(CLI) . I think it
>covers both hosts and operations and allows to specify a list for both.
>
>Thanks
>Parth
>
>From: Tom Graves <tg...@yahoo.com>>
>Reply-To: Tom Graves <tg...@yahoo.com>>
>Date: Wednesday, April 22, 2015 at 11:02 AM
>To: Parth Brahmbhatt
><pb...@hortonworks.com>>,
>"dev@kafka.apache.org<ma...@kafka.apache.org>"
><de...@kafka.apache.org>>
>Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
>
>Thanks for the explanations Parth.
>
>On the configs questions, the way I see it is its more likely to
>accidentally give everyone access, especially since you have to run a
>separate command to change the acls. If there was some config for
>defaults, a cluster admin could change that to be nobody or certain set
>of users, then grant others permissions.  This would also remove the race
>between commands.  This is something you can always add later though if
>people request it.
>
>So in kafka-acl.sh how do I actually tell it what the operation is?
>kafka-acl.sh --topic testtopic --add --grandprincipal user:joe,user:kate
>
>where does READ, WRITE, etc go?  Can specify as a list so I don't have to
>run this a bunch of times for each.
>
>Do you want to have a --host option for --list so that admins could see
>what acls apply to specific host(s)?
>
>Tom
>
>
>
>On Wednesday, April 22, 2015 11:38 AM, Parth Brahmbhatt
><pb...@hortonworks.com>> wrote:
>
>
>
>FYI, I have modified the KIP to include group as resource. In order to
>access “joinGroup” and “commitOFfset” APIs the user will need a read
>permission on topic and WRITE permission on group.
>
>I plan to open a VOTE thread by noon if there are no more concerns.
>
>Thanks
>Parth
>
>On 4/22/15, 9:03 AM, "Tom Graves"
><tg...@yahoo.com.INVALID>> wrote:
>
>>Hey everyone,
>>Sorry to jump in on the conversation so late. I'm new to Kafka. I'll
>>apologize in advance if you have already covered some of my questions.  I
>>read through the wiki and had some comments and questions.
>>1) public enum Operation needs EDIT changed to ALTER
>
>>    Done.
>
>>2) Does the Authorizer class need a setAcls?  Rather then just add to be
>>able to set to explicit list and overwrite what was there?  I see the
>>kafka-acl.sh lists a removeall so I guess you could do removeall and then
>>add.  I also don't see a removeall in the Authorizer class, is it going
>>to loop through them all to remove each one?
>
>    There is an overloaded version of removeAcls in the interface that
>takes
>in resource as the only input and as described in the javadoc all the acls
>attached to that resource will be deleted. To cover the setAcl use case
>the caller can first call remove and then add.
>
>>3) Can someone tell me what the use case to do acls based on the hosts?
>>I can see some possibilities just wondering if we can concrete ones where
>>one user is allowed from one host but not another.
>
>    I am not sure if I understand the question given the use case you
>described in your question is what we are trying to cover with use of
>hosts in Acl. There are some additional use cases like “allow access to
>any user from host1,host2” but I think primarily it gives the admins the
>ability to define acls at a more granular level.
>
>>4) I'm a bit unclear how the "resource" works in the Authorizer class.
>>From what I see we have 2 resources - topics and cluster.  If I want to
>>add an acl to allow "joe" to CREATE for the cluster then I call addAcls
>>with  Acl("user: joe", ALLOW, Set(*), Set(CREATE)) and "cluster"?  What
>>if I want to call addAcls for DESCRIBE on a topic?  Is the resource then
>>"topic" or is it the topic name?
>
>    We now have 3 resources(added group), please see the updated doc. The
>CREATE acl that you described is correct. For any topic operation you
>should use topic name as the resource name and for group the user will
>provide groupId as resource name.
>
>>5) reassigning partitions is a CLUSTER_ACTION or superuser?  Its not
>>totally clear to me the differences between these.  what about increasing
>># of partitions?
>
>    I see this as an alter topic operation so it is at topic level and the
>user must have alter permissions on topic.
>
>>6) groups are mentioned, are we supporting right away or is that a follow
>>on item? (is there going to be a kafka.supergroups)
>
>    I think it can be a separate jira just for braking down the code
>review
>in smaller chunk. We will support it in first version but I think if we
>can not do it for any reason that should not block a release with all the
>other authZ work. We made deliberate design choices (like introducing a
>principalType in KafkaPrinciapl) to allow supporting groups as an
>incremental change.
>
>>7) Are there config options for setting acls when I create my topic?  Or
>>do I have to create my topic and then run the kafka-acl.sh script to set
>>them?  Although its very small, there would be possible race there that
>>someone could start producing to topic before acls are set.
>
>    We discussed this yesterday and we agreed to go with kafka-acl.sh. Yes
>there is a very very small window of vulnerability but I think that really
>does not warrant to change the decision in this case.
>
>>8) are there configs for cluster level acl defaults?  Or does it default
>>to superusers on bringing up new cluster and you have to modify with cli.
>>thanks,Tom
>
>    No defaults, the default is superusers will have full access. I don’t
>think making assumptions about ones security requirement should be our
>burden.
>
>
>>
>>    On Tuesday, April 21, 2015 7:10 PM, Parth Brahmbhatt
>><pb...@hortonworks.com>> wrote:
>>
>>
>> I have added the notes to KIP-11 Open question sections.
>>
>>Thanks
>>Parth
>>
>>On 4/21/15, 4:49 PM, "Gwen Shapira"
>><gs...@cloudera.com>> wrote:
>>
>>>Adding my notes from today's call to the thread:
>>>
>>>** Deny or Allow all by default? We will add a configuration to
>>>control this. The configuration will default to “allow” for backward
>>>compatibility. Security admins can set it to "deny"
>>>
>>>** Storing ACLs for default authorizers: We'll store them in ZK. We'll
>>>support pointing the authorizer to any ZK.
>>>The use of ZK will be internal to the default authorizer. Authorizer
>>>reads ACLs from cache every hour. We proposed having mechanism
>>>(possibly via new ZK node) to tell broker to refresh the cache
>>>immediately.
>>>
>>>** Support deny as permission type - we agreed to keep this.
>>>
>>>** Mapping operations to API: We may need to add Group as a resource,
>>>with JoinGroup and OffsetCommit require privilege on the consumer
>>>group.
>>>This can be something we pass now and authorizers can support in
>>>future. - Jay will write specifics to the mailing list discussion.
>>>
>>>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps
>>><ja...@gmail.com>> wrote:
>>>> Following up on the KIP discussion. Two options for authorizing
>>>>consumers
>>>> to read topic "t" as part of group "g":
>>>> 1. READ permission on resource /topic/t
>>>> 2. READ permission on resource /topic/t AND WRITE permission on
>>>>/group/g
>>>>
>>>> The advantage of (1) is that it is simpler. The disadvantage is that
>>>>any
>>>> member of any group that reads from t can commit offsets as any other
>>>> member of a different group. This doesn't effect data security (who
>>>>can
>>>> access what) but it is a bit of a management issue--a malicious person
>>>>can
>>>> cause data loss or duplicates for another consumer by committing
>>>>offset.
>>>>
>>>> I think I favor (2) but it's worth it to think it through.
>>>>
>>>> -Jay
>>>>
>>>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
>>>> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
>>>>wrote:
>>>>
>>>>> Hey Jun,
>>>>>
>>>>> Yes and we support wild cards for all acl entities principal, hosts
>>>>>and
>>>>> operation.
>>>>>
>>>>> Thanks
>>>>> Parth
>>>>>
>>>>> On 4/21/15, 9:06 AM, "Jun Rao"
>>>>><ju...@confluent.io>> wrote:
>>>>>
>>>>> >Harsha, Parth,
>>>>> >
>>>>> >Thanks for the clarification. This makes sense. Perhaps we can
>>>>>clarify the
>>>>> >meaning of those rules in the wiki.
>>>>> >
>>>>> >Related to this, it seems that we need to support wildcard in
>>>>>cli/request
>>>>> >protocol for topics?
>>>>> >
>>>>> >Jun
>>>>> >
>>>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>>>>> >pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
>>>>>wrote:
>>>>> >
>>>>> >> The iptables on unix supports the DENY operator, not that it
>>>>>should
>>>>> >> matter. The deny operator can also be used to specify ³allow user1
>>>>>to
>>>>> >>READ
>>>>> >> from topic1 from all hosts but host1,host2². Again we could add a
>>>>>host
>>>>> >> group semantic and extra complexity around that, not sure if its
>>>>>worth
>>>>> >>it.
>>>>> >> In addition with DENY operator you are now not forced to create a
>>>>> >>special
>>>>> >> group just to support the authorization use case. I am not
>>>>>convinced
>>>>> >>that
>>>>> >> the operator it self is really all that confusing. There are 3
>>>>>practical
>>>>> >> use cases:
>>>>> >> - Resource with no acl what so ever -> allow access to everyone (
>>>>>just
>>>>> >>for
>>>>> >> backward compatibility, I would much rather fail close and force
>>>>>users
>>>>> >>to
>>>>> >> explicitly grant acls that allows access to all users.)
>>>>> >> - Resource with some acl attached -> only users that have a
>>>>>matching
>>>>> >>allow
>>>>> >> acl are allowed (i.e. ³allow READ access to topic1 to user1 from
>>>>>all
>>>>> >> hosts², only user1 has READ access and no other user has access of
>>>>>any
>>>>> >> kind)
>>>>> >> - Resource with some allow and some deny acl attached -> users are
>>>>> >>allowed
>>>>> >> to perform operation only when they satisfy allow acl and do not
>>>>>have
>>>>> >> conflicting deny acl. Users that have no acl(allow or deny) will
>>>>>still
>>>>> >>not
>>>>> >> have any access. (i.e. ³allow READ access to topic1 to user1 from
>>>>>all
>>>>> >> hosts except host1 and host², only user1 has access but not from
>>>>>host1
>>>>> >>an
>>>>> >> host2)
>>>>> >>
>>>>> >> I think we need to make a decision on deny primarily because with
>>>>> >> introduction of acl management API, Acl is now a public class that
>>>>>will
>>>>> >>be
>>>>> >> used by Ranger/Santry and other authroization providers. In
>>>>>Current
>>>>> >>design
>>>>> >> the acl has a permissionType enum field with possible values of
>>>>>Allow
>>>>> >>and
>>>>> >> Deny. If we chose to remove deny we can assume all acls to be of
>>>>>allow
>>>>> >> type and remove the permissionType field completely.
>>>>> >>
>>>>> >> Thanks
>>>>> >> Parth
>>>>> >>
>>>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira"
>>>>><gs...@cloudera.com>> wrote:
>>>>> >>
>>>>> >> >I think thats how its done in pretty much any system I can think
>>>>>of.
>>>>> >> >
>>>>> >>
>>>>> >>
>>>>>
>>>>>
>>
>>
>>
>>
>
>
>



  

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Tong Li <li...@us.ibm.com>]

same question here as well on hosts. If I would like to secure a topic,
regardless where the topic is, I would like it to be secured. It is hard to
imagine that one would like a topic to be secured on one host, but not on
the other unless a topic spreads over multiple hosts really meant to be
totally different things. Then it will be really needed.

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
litong01@us.ibm.com



From:	Jeff Holoman <jh...@cloudera.com>
To:	dev@kafka.apache.org
Cc:	Tom Graves <tg...@yahoo.com>
Date:	04/22/2015 02:40 PM
Subject:	Re: [DISCUSS] KIP-11- Authorization design for kafka security



Parth,

This is a long thread, so trying to keep up here, sorry if this has been
covered before. First, great job on the KIP proposal and work so far.

Are we sure that we want to tie host level access to a given user? My
understanding is that the ACL will be (omitting some fields)

user_a, host1, host2, host3
user_b, host1, host2, host3

So there would potentially be a lot of redundancy in the configs. Does it
make sense to have hosts be at the same level as principal in the
hierarchy? This way you could just blanket the allowed / denied hosts and
only have to worry about the users. So if you follow this, then

we can wildcard the user so we can have a separate list of just host-based
access. What's the order that the perms would be evaluated if a there was
more than one match on a principal ?

Is the thought that there wouldn't usually be much overlap on hosts? I
guess I can imagine a scenario where I want to offline/online access to a
particular hosts or set of hosts and if there was overlap, I'm doing a
bunch of alter commands for just a single host. Maybe this is too contrived
an example?

I agree that having this level of granularity gives flexibility but I
wonder if people will actually use it and not just * the hosts for a given
user and create separate "global" list as i mentioned above?

The only other system I know of that ties users with hosts for access is
MySql and I don't love that model. Companies usually standardize on group
authorization anyway, are we complicating that issue with the inclusion of
hosts attached to users? Additionally I worry about the debt of big JSON
configs in the first place, most non-developers find them non-intuitive
already, so anything to ease this I think would be beneficial.


Thanks

Jeff

On Wed, Apr 22, 2015 at 2:22 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:

> Sorry I missed your last questions. I am +0 on adding ―host option for
> ―list, we could add it for symmetry. Again if this is only a CLI change
it
> can be added later if you mean adding this in authorizer interface then
we
> should make a decision now.
>
> Given a choice I would like to actually keep only one option which is
> resource based get (remove even the get based on principal). I see those
> (getAcl for principal or host) as special filtering case which can easily
> be achieved by a third party tool by doing "list all topics" and calling
> getAcls for each topic and applying filtering logic on that.  I really
> don’t see the need to make those first class citizens of the authorizer
> interface given these kind of queries will be issued outside of broker
JVM
> so they will not benefit from the caching and because the storage will be
> indexed on resource both these options even as a first class API will
just
> scan all topic acls and apply filtering logic.
>
> Thanks
> Parth
>
> On 4/22/15, 11:08 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>
> wrote:
>
> >Please see all the available options here
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization
+I
> >nterface#KIP-11-AuthorizationInterface-AclManagement(CLI) . I think it
> >covers both hosts and operations and allows to specify a list for both.
> >
> >Thanks
> >Parth
> >
> >From: Tom Graves <tg...@yahoo.com>>
> >Reply-To: Tom Graves <tg...@yahoo.com>>
> >Date: Wednesday, April 22, 2015 at 11:02 AM
> >To: Parth Brahmbhatt
> ><pb...@hortonworks.com>>,
> >"dev@kafka.apache.org<ma...@kafka.apache.org>"
> ><de...@kafka.apache.org>>
> >Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
> >
> >Thanks for the explanations Parth.
> >
> >On the configs questions, the way I see it is its more likely to
> >accidentally give everyone access, especially since you have to run a
> >separate command to change the acls. If there was some config for
> >defaults, a cluster admin could change that to be nobody or certain set
> >of users, then grant others permissions.  This would also remove the
race
> >between commands.  This is something you can always add later though if
> >people request it.
> >
> >So in kafka-acl.sh how do I actually tell it what the operation is?
> >kafka-acl.sh --topic testtopic --add --grandprincipal user:joe,user:kate
> >
> >where does READ, WRITE, etc go?  Can specify as a list so I don't have
to
> >run this a bunch of times for each.
> >
> >Do you want to have a --host option for --list so that admins could see
> >what acls apply to specific host(s)?
> >
> >Tom
> >
> >
> >
> >On Wednesday, April 22, 2015 11:38 AM, Parth Brahmbhatt
> ><pb...@hortonworks.com>> wrote:
> >
> >
> >
> >FYI, I have modified the KIP to include group as resource. In order to
> >access “joinGroup” and “commitOFfset” APIs the user will need a read
> >permission on topic and WRITE permission on group.
> >
> >I plan to open a VOTE thread by noon if there are no more concerns.
> >
> >Thanks
> >Parth
> >
> >On 4/22/15, 9:03 AM, "Tom Graves"
> ><tg...@yahoo.com.INVALID>>
> wrote:
> >
> >>Hey everyone,
> >>Sorry to jump in on the conversation so late. I'm new to Kafka. I'll
> >>apologize in advance if you have already covered some of my questions.
I
> >>read through the wiki and had some comments and questions.
> >>1) public enum Operation needs EDIT changed to ALTER
> >
> >>    Done.
> >
> >>2) Does the Authorizer class need a setAcls?  Rather then just add to
be
> >>able to set to explicit list and overwrite what was there?  I see the
> >>kafka-acl.sh lists a removeall so I guess you could do removeall and
then
> >>add.  I also don't see a removeall in the Authorizer class, is it going
> >>to loop through them all to remove each one?
> >
> >    There is an overloaded version of removeAcls in the interface that
> >takes
> >in resource as the only input and as described in the javadoc all the
acls
> >attached to that resource will be deleted. To cover the setAcl use case
> >the caller can first call remove and then add.
> >
> >>3) Can someone tell me what the use case to do acls based on the hosts?
> >>I can see some possibilities just wondering if we can concrete ones
where
> >>one user is allowed from one host but not another.
> >
> >    I am not sure if I understand the question given the use case you
> >described in your question is what we are trying to cover with use of
> >hosts in Acl. There are some additional use cases like “allow access to
> >any user from host1,host2” but I think primarily it gives the admins the
> >ability to define acls at a more granular level.
> >
> >>4) I'm a bit unclear how the "resource" works in the Authorizer class.
> >>From what I see we have 2 resources - topics and cluster.  If I want to
> >>add an acl to allow "joe" to CREATE for the cluster then I call addAcls
> >>with  Acl("user: joe", ALLOW, Set(*), Set(CREATE)) and "cluster"?  What
> >>if I want to call addAcls for DESCRIBE on a topic?  Is the resource
then
> >>"topic" or is it the topic name?
> >
> >    We now have 3 resources(added group), please see the updated doc.
The
> >CREATE acl that you described is correct. For any topic operation you
> >should use topic name as the resource name and for group the user will
> >provide groupId as resource name.
> >
> >>5) reassigning partitions is a CLUSTER_ACTION or superuser?  Its not
> >>totally clear to me the differences between these.  what about
increasing
> >># of partitions?
> >
> >    I see this as an alter topic operation so it is at topic level and
the
> >user must have alter permissions on topic.
> >
> >>6) groups are mentioned, are we supporting right away or is that a
follow
> >>on item? (is there going to be a kafka.supergroups)
> >
> >    I think it can be a separate jira just for braking down the code
> >review
> >in smaller chunk. We will support it in first version but I think if we
> >can not do it for any reason that should not block a release with all
the
> >other authZ work. We made deliberate design choices (like introducing a
> >principalType in KafkaPrinciapl) to allow supporting groups as an
> >incremental change.
> >
> >>7) Are there config options for setting acls when I create my topic?
Or
> >>do I have to create my topic and then run the kafka-acl.sh script to
set
> >>them?  Although its very small, there would be possible race there that
> >>someone could start producing to topic before acls are set.
> >
> >    We discussed this yesterday and we agreed to go with kafka-acl.sh.
Yes
> >there is a very very small window of vulnerability but I think that
really
> >does not warrant to change the decision in this case.
> >
> >>8) are there configs for cluster level acl defaults?  Or does it
default
> >>to superusers on bringing up new cluster and you have to modify with
cli.
> >>thanks,Tom
> >
> >    No defaults, the default is superusers will have full access. I
don’t
> >think making assumptions about ones security requirement should be our
> >burden.
> >
> >
> >>
> >>    On Tuesday, April 21, 2015 7:10 PM, Parth Brahmbhatt
> >><pb...@hortonworks.com>>
wrote:
> >>
> >>
> >> I have added the notes to KIP-11 Open question sections.
> >>
> >>Thanks
> >>Parth
> >>
> >>On 4/21/15, 4:49 PM, "Gwen Shapira"
> >><gs...@cloudera.com>> wrote:
> >>
> >>>Adding my notes from today's call to the thread:
> >>>
> >>>** Deny or Allow all by default? We will add a configuration to
> >>>control this. The configuration will default to “allow” for backward
> >>>compatibility. Security admins can set it to "deny"
> >>>
> >>>** Storing ACLs for default authorizers: We'll store them in ZK. We'll
> >>>support pointing the authorizer to any ZK.
> >>>The use of ZK will be internal to the default authorizer. Authorizer
> >>>reads ACLs from cache every hour. We proposed having mechanism
> >>>(possibly via new ZK node) to tell broker to refresh the cache
> >>>immediately.
> >>>
> >>>** Support deny as permission type - we agreed to keep this.
> >>>
> >>>** Mapping operations to API: We may need to add Group as a resource,
> >>>with JoinGroup and OffsetCommit require privilege on the consumer
> >>>group.
> >>>This can be something we pass now and authorizers can support in
> >>>future. - Jay will write specifics to the mailing list discussion.
> >>>
> >>>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps
> >>><ja...@gmail.com>> wrote:
> >>>> Following up on the KIP discussion. Two options for authorizing
> >>>>consumers
> >>>> to read topic "t" as part of group "g":
> >>>> 1. READ permission on resource /topic/t
> >>>> 2. READ permission on resource /topic/t AND WRITE permission on
> >>>>/group/g
> >>>>
> >>>> The advantage of (1) is that it is simpler. The disadvantage is that
> >>>>any
> >>>> member of any group that reads from t can commit offsets as any
other
> >>>> member of a different group. This doesn't effect data security (who
> >>>>can
> >>>> access what) but it is a bit of a management issue--a malicious
person
> >>>>can
> >>>> cause data loss or duplicates for another consumer by committing
> >>>>offset.
> >>>>
> >>>> I think I favor (2) but it's worth it to think it through.
> >>>>
> >>>> -Jay
> >>>>
> >>>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
> >>>> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
> >>>>wrote:
> >>>>
> >>>>> Hey Jun,
> >>>>>
> >>>>> Yes and we support wild cards for all acl entities principal, hosts
> >>>>>and
> >>>>> operation.
> >>>>>
> >>>>> Thanks
> >>>>> Parth
> >>>>>
> >>>>> On 4/21/15, 9:06 AM, "Jun Rao"
> >>>>><ju...@confluent.io>> wrote:
> >>>>>
> >>>>> >Harsha, Parth,
> >>>>> >
> >>>>> >Thanks for the clarification. This makes sense. Perhaps we can
> >>>>>clarify the
> >>>>> >meaning of those rules in the wiki.
> >>>>> >
> >>>>> >Related to this, it seems that we need to support wildcard in
> >>>>>cli/request
> >>>>> >protocol for topics?
> >>>>> >
> >>>>> >Jun
> >>>>> >
> >>>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
> >>>>> >pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
> >>>>>wrote:
> >>>>> >
> >>>>> >> The iptables on unix supports the DENY operator, not that it
> >>>>>should
> >>>>> >> matter. The deny operator can also be used to specify ³allow
user1
> >>>>>to
> >>>>> >>READ
> >>>>> >> from topic1 from all hosts but host1,host2². Again we could add
a
> >>>>>host
> >>>>> >> group semantic and extra complexity around that, not sure if its
> >>>>>worth
> >>>>> >>it.
> >>>>> >> In addition with DENY operator you are now not forced to create
a
> >>>>> >>special
> >>>>> >> group just to support the authorization use case. I am not
> >>>>>convinced
> >>>>> >>that
> >>>>> >> the operator it self is really all that confusing. There are 3
> >>>>>practical
> >>>>> >> use cases:
> >>>>> >> - Resource with no acl what so ever -> allow access to everyone
(
> >>>>>just
> >>>>> >>for
> >>>>> >> backward compatibility, I would much rather fail close and force
> >>>>>users
> >>>>> >>to
> >>>>> >> explicitly grant acls that allows access to all users.)
> >>>>> >> - Resource with some acl attached -> only users that have a
> >>>>>matching
> >>>>> >>allow
> >>>>> >> acl are allowed (i.e. ³allow READ access to topic1 to user1 from
> >>>>>all
> >>>>> >> hosts², only user1 has READ access and no other user has access
of
> >>>>>any
> >>>>> >> kind)
> >>>>> >> - Resource with some allow and some deny acl attached -> users
are
> >>>>> >>allowed
> >>>>> >> to perform operation only when they satisfy allow acl and do not
> >>>>>have
> >>>>> >> conflicting deny acl. Users that have no acl(allow or deny) will
> >>>>>still
> >>>>> >>not
> >>>>> >> have any access. (i.e. ³allow READ access to topic1 to user1
from
> >>>>>all
> >>>>> >> hosts except host1 and host², only user1 has access but not from
> >>>>>host1
> >>>>> >>an
> >>>>> >> host2)
> >>>>> >>
> >>>>> >> I think we need to make a decision on deny primarily because
with
> >>>>> >> introduction of acl management API, Acl is now a public class
that
> >>>>>will
> >>>>> >>be
> >>>>> >> used by Ranger/Santry and other authroization providers. In
> >>>>>Current
> >>>>> >>design
> >>>>> >> the acl has a permissionType enum field with possible values of
> >>>>>Allow
> >>>>> >>and
> >>>>> >> Deny. If we chose to remove deny we can assume all acls to be of
> >>>>>allow
> >>>>> >> type and remove the permissionType field completely.
> >>>>> >>
> >>>>> >> Thanks
> >>>>> >> Parth
> >>>>> >>
> >>>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira"
> >>>>><gs...@cloudera.com>> wrote:
> >>>>> >>
> >>>>> >> >I think thats how its done in pretty much any system I can
think
> >>>>>of.
> >>>>> >> >
> >>>>> >>
> >>>>> >>
> >>>>>
> >>>>>
> >>
> >>
> >>
> >>
> >
> >
> >
>
>


--
Jeff Holoman
Systems Engineer

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Hi Jeff,

Thanks a lot for the review. I think you have a valid point about acls
being duplicated and the simplest solution would be to modify acls class
so they hold a set of principals instead of single principal. i.e

<user_a,user_b> has <READ,WRITE,DESCRIBE> Permissions on <Topic1> from
<Host1, Host2, Host3>.

I think the evaluation order only matters for the permissionType which is
Deny acls should be evaluated before allow acls. To give you an example
suppose we have following acls

acl1 -> user1 is allowed to READ from all hosts.
acl2 -> host1 is allowed to READ regardless of who is the user.
acl3 -> host2 is allowed to READ regardless of who is the user.

acl4 -> user1 is denied to READ from host1.

As stated in the KIP we first evaluate DENY so if user1 tries to access
from host1 he will be denied(acl4), even though both user1 and host1 has
acl’s for allow with wildcards (acl1, acl2).
If user1 tried to READ from host2 , the action will be allowed and it does
not matter if we match acl3 or acl1 so I don’t think the evaluation order
matters here.

“Will people actually use hosts with users?” I really don’t know but given
ACl’s are part of our Public APIs I thought it is better to try and cover
more use cases. If others think this extra complexity is not worth the
value its adding please raise your concerns so we can discuss if it should
be removed from the acl structure. Note that even in absence of hosts from
ACL users will still be able to whitelist/blacklist host as long as we
start supporting principalType = “host”, easy to add and can be an
incremental improvement. They will however loose the ability to restrict
access to users just from a set of hosts.

We agreed to offer a CLI to overcome the JSON acl config
https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+In
terface#KIP-11-AuthorizationInterface-AclManagement(CLI). I still like
Jsons but that probably has something to do with me being a developer :-).

Thanks
Parth

On 4/22/15, 11:38 AM, "Jeff Holoman" <jh...@cloudera.com> wrote:

>Parth,
>
>This is a long thread, so trying to keep up here, sorry if this has been
>covered before. First, great job on the KIP proposal and work so far.
>
>Are we sure that we want to tie host level access to a given user? My
>understanding is that the ACL will be (omitting some fields)
>
>user_a, host1, host2, host3
>user_b, host1, host2, host3
>
>So there would potentially be a lot of redundancy in the configs. Does it
>make sense to have hosts be at the same level as principal in the
>hierarchy? This way you could just blanket the allowed / denied hosts and
>only have to worry about the users. So if you follow this, then
>
>we can wildcard the user so we can have a separate list of just host-based
>access. What's the order that the perms would be evaluated if a there was
>more than one match on a principal ?
>
>Is the thought that there wouldn't usually be much overlap on hosts? I
>guess I can imagine a scenario where I want to offline/online access to a
>particular hosts or set of hosts and if there was overlap, I'm doing a
>bunch of alter commands for just a single host. Maybe this is too
>contrived
>an example?
>
>I agree that having this level of granularity gives flexibility but I
>wonder if people will actually use it and not just * the hosts for a given
>user and create separate "global" list as i mentioned above?
>
>The only other system I know of that ties users with hosts for access is
>MySql and I don't love that model. Companies usually standardize on group
>authorization anyway, are we complicating that issue with the inclusion of
>hosts attached to users? Additionally I worry about the debt of big JSON
>configs in the first place, most non-developers find them non-intuitive
>already, so anything to ease this I think would be beneficial.
>
>
>Thanks
>
>Jeff
>
>On Wed, Apr 22, 2015 at 2:22 PM, Parth Brahmbhatt <
>pbrahmbhatt@hortonworks.com> wrote:
>
>> Sorry I missed your last questions. I am +0 on adding ―host option for
>> ―list, we could add it for symmetry. Again if this is only a CLI change
>>it
>> can be added later if you mean adding this in authorizer interface then
>>we
>> should make a decision now.
>>
>> Given a choice I would like to actually keep only one option which is
>> resource based get (remove even the get based on principal). I see those
>> (getAcl for principal or host) as special filtering case which can
>>easily
>> be achieved by a third party tool by doing "list all topics" and calling
>> getAcls for each topic and applying filtering logic on that.  I really
>> don’t see the need to make those first class citizens of the authorizer
>> interface given these kind of queries will be issued outside of broker
>>JVM
>> so they will not benefit from the caching and because the storage will
>>be
>> indexed on resource both these options even as a first class API will
>>just
>> scan all topic acls and apply filtering logic.
>>
>> Thanks
>> Parth
>>
>> On 4/22/15, 11:08 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>
>> wrote:
>>
>> >Please see all the available options here
>> >
>> 
>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+
>>I
>> >nterface#KIP-11-AuthorizationInterface-AclManagement(CLI) . I think it
>> >covers both hosts and operations and allows to specify a list for both.
>> >
>> >Thanks
>> >Parth
>> >
>> >From: Tom Graves <tg...@yahoo.com>>
>> >Reply-To: Tom Graves
>><tg...@yahoo.com>>
>> >Date: Wednesday, April 22, 2015 at 11:02 AM
>> >To: Parth Brahmbhatt
>> ><pb...@hortonworks.com>>,
>> >"dev@kafka.apache.org<ma...@kafka.apache.org>"
>> ><de...@kafka.apache.org>>
>> >Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
>> >
>> >Thanks for the explanations Parth.
>> >
>> >On the configs questions, the way I see it is its more likely to
>> >accidentally give everyone access, especially since you have to run a
>> >separate command to change the acls. If there was some config for
>> >defaults, a cluster admin could change that to be nobody or certain set
>> >of users, then grant others permissions.  This would also remove the
>>race
>> >between commands.  This is something you can always add later though if
>> >people request it.
>> >
>> >So in kafka-acl.sh how do I actually tell it what the operation is?
>> >kafka-acl.sh --topic testtopic --add --grandprincipal
>>user:joe,user:kate
>> >
>> >where does READ, WRITE, etc go?  Can specify as a list so I don't have
>>to
>> >run this a bunch of times for each.
>> >
>> >Do you want to have a --host option for --list so that admins could see
>> >what acls apply to specific host(s)?
>> >
>> >Tom
>> >
>> >
>> >
>> >On Wednesday, April 22, 2015 11:38 AM, Parth Brahmbhatt
>> ><pb...@hortonworks.com>>
>>wrote:
>> >
>> >
>> >
>> >FYI, I have modified the KIP to include group as resource. In order to
>> >access “joinGroup” and “commitOFfset” APIs the user will need a read
>> >permission on topic and WRITE permission on group.
>> >
>> >I plan to open a VOTE thread by noon if there are no more concerns.
>> >
>> >Thanks
>> >Parth
>> >
>> >On 4/22/15, 9:03 AM, "Tom Graves"
>> ><tg...@yahoo.com.INVALID>>
>> wrote:
>> >
>> >>Hey everyone,
>> >>Sorry to jump in on the conversation so late. I'm new to Kafka. I'll
>> >>apologize in advance if you have already covered some of my
>>questions.  I
>> >>read through the wiki and had some comments and questions.
>> >>1) public enum Operation needs EDIT changed to ALTER
>> >
>> >>    Done.
>> >
>> >>2) Does the Authorizer class need a setAcls?  Rather then just add to
>>be
>> >>able to set to explicit list and overwrite what was there?  I see the
>> >>kafka-acl.sh lists a removeall so I guess you could do removeall and
>>then
>> >>add.  I also don't see a removeall in the Authorizer class, is it
>>going
>> >>to loop through them all to remove each one?
>> >
>> >    There is an overloaded version of removeAcls in the interface that
>> >takes
>> >in resource as the only input and as described in the javadoc all the
>>acls
>> >attached to that resource will be deleted. To cover the setAcl use case
>> >the caller can first call remove and then add.
>> >
>> >>3) Can someone tell me what the use case to do acls based on the
>>hosts?
>> >>I can see some possibilities just wondering if we can concrete ones
>>where
>> >>one user is allowed from one host but not another.
>> >
>> >    I am not sure if I understand the question given the use case you
>> >described in your question is what we are trying to cover with use of
>> >hosts in Acl. There are some additional use cases like “allow access to
>> >any user from host1,host2” but I think primarily it gives the admins
>>the
>> >ability to define acls at a more granular level.
>> >
>> >>4) I'm a bit unclear how the "resource" works in the Authorizer class.
>> >>From what I see we have 2 resources - topics and cluster.  If I want
>>to
>> >>add an acl to allow "joe" to CREATE for the cluster then I call
>>addAcls
>> >>with  Acl("user: joe", ALLOW, Set(*), Set(CREATE)) and "cluster"?
>>What
>> >>if I want to call addAcls for DESCRIBE on a topic?  Is the resource
>>then
>> >>"topic" or is it the topic name?
>> >
>> >    We now have 3 resources(added group), please see the updated doc.
>>The
>> >CREATE acl that you described is correct. For any topic operation you
>> >should use topic name as the resource name and for group the user will
>> >provide groupId as resource name.
>> >
>> >>5) reassigning partitions is a CLUSTER_ACTION or superuser?  Its not
>> >>totally clear to me the differences between these.  what about
>>increasing
>> >># of partitions?
>> >
>> >    I see this as an alter topic operation so it is at topic level and
>>the
>> >user must have alter permissions on topic.
>> >
>> >>6) groups are mentioned, are we supporting right away or is that a
>>follow
>> >>on item? (is there going to be a kafka.supergroups)
>> >
>> >    I think it can be a separate jira just for braking down the code
>> >review
>> >in smaller chunk. We will support it in first version but I think if we
>> >can not do it for any reason that should not block a release with all
>>the
>> >other authZ work. We made deliberate design choices (like introducing a
>> >principalType in KafkaPrinciapl) to allow supporting groups as an
>> >incremental change.
>> >
>> >>7) Are there config options for setting acls when I create my topic?
>>Or
>> >>do I have to create my topic and then run the kafka-acl.sh script to
>>set
>> >>them?  Although its very small, there would be possible race there
>>that
>> >>someone could start producing to topic before acls are set.
>> >
>> >    We discussed this yesterday and we agreed to go with kafka-acl.sh.
>>Yes
>> >there is a very very small window of vulnerability but I think that
>>really
>> >does not warrant to change the decision in this case.
>> >
>> >>8) are there configs for cluster level acl defaults?  Or does it
>>default
>> >>to superusers on bringing up new cluster and you have to modify with
>>cli.
>> >>thanks,Tom
>> >
>> >    No defaults, the default is superusers will have full access. I
>>don’t
>> >think making assumptions about ones security requirement should be our
>> >burden.
>> >
>> >
>> >>
>> >>    On Tuesday, April 21, 2015 7:10 PM, Parth Brahmbhatt
>> >><pb...@hortonworks.com>>
>>wrote:
>> >>
>> >>
>> >> I have added the notes to KIP-11 Open question sections.
>> >>
>> >>Thanks
>> >>Parth
>> >>
>> >>On 4/21/15, 4:49 PM, "Gwen Shapira"
>> >><gs...@cloudera.com>> wrote:
>> >>
>> >>>Adding my notes from today's call to the thread:
>> >>>
>> >>>** Deny or Allow all by default? We will add a configuration to
>> >>>control this. The configuration will default to “allow” for backward
>> >>>compatibility. Security admins can set it to "deny"
>> >>>
>> >>>** Storing ACLs for default authorizers: We'll store them in ZK.
>>We'll
>> >>>support pointing the authorizer to any ZK.
>> >>>The use of ZK will be internal to the default authorizer. Authorizer
>> >>>reads ACLs from cache every hour. We proposed having mechanism
>> >>>(possibly via new ZK node) to tell broker to refresh the cache
>> >>>immediately.
>> >>>
>> >>>** Support deny as permission type - we agreed to keep this.
>> >>>
>> >>>** Mapping operations to API: We may need to add Group as a resource,
>> >>>with JoinGroup and OffsetCommit require privilege on the consumer
>> >>>group.
>> >>>This can be something we pass now and authorizers can support in
>> >>>future. - Jay will write specifics to the mailing list discussion.
>> >>>
>> >>>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps
>> >>><ja...@gmail.com>> wrote:
>> >>>> Following up on the KIP discussion. Two options for authorizing
>> >>>>consumers
>> >>>> to read topic "t" as part of group "g":
>> >>>> 1. READ permission on resource /topic/t
>> >>>> 2. READ permission on resource /topic/t AND WRITE permission on
>> >>>>/group/g
>> >>>>
>> >>>> The advantage of (1) is that it is simpler. The disadvantage is
>>that
>> >>>>any
>> >>>> member of any group that reads from t can commit offsets as any
>>other
>> >>>> member of a different group. This doesn't effect data security (who
>> >>>>can
>> >>>> access what) but it is a bit of a management issue--a malicious
>>person
>> >>>>can
>> >>>> cause data loss or duplicates for another consumer by committing
>> >>>>offset.
>> >>>>
>> >>>> I think I favor (2) but it's worth it to think it through.
>> >>>>
>> >>>> -Jay
>> >>>>
>> >>>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
>> >>>> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
>> >>>>wrote:
>> >>>>
>> >>>>> Hey Jun,
>> >>>>>
>> >>>>> Yes and we support wild cards for all acl entities principal,
>>hosts
>> >>>>>and
>> >>>>> operation.
>> >>>>>
>> >>>>> Thanks
>> >>>>> Parth
>> >>>>>
>> >>>>> On 4/21/15, 9:06 AM, "Jun Rao"
>> >>>>><ju...@confluent.io>> wrote:
>> >>>>>
>> >>>>> >Harsha, Parth,
>> >>>>> >
>> >>>>> >Thanks for the clarification. This makes sense. Perhaps we can
>> >>>>>clarify the
>> >>>>> >meaning of those rules in the wiki.
>> >>>>> >
>> >>>>> >Related to this, it seems that we need to support wildcard in
>> >>>>>cli/request
>> >>>>> >protocol for topics?
>> >>>>> >
>> >>>>> >Jun
>> >>>>> >
>> >>>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>> >>>>> >pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
>> >>>>>wrote:
>> >>>>> >
>> >>>>> >> The iptables on unix supports the DENY operator, not that it
>> >>>>>should
>> >>>>> >> matter. The deny operator can also be used to specify ³allow
>>user1
>> >>>>>to
>> >>>>> >>READ
>> >>>>> >> from topic1 from all hosts but host1,host2². Again we could
>>add a
>> >>>>>host
>> >>>>> >> group semantic and extra complexity around that, not sure if
>>its
>> >>>>>worth
>> >>>>> >>it.
>> >>>>> >> In addition with DENY operator you are now not forced to
>>create a
>> >>>>> >>special
>> >>>>> >> group just to support the authorization use case. I am not
>> >>>>>convinced
>> >>>>> >>that
>> >>>>> >> the operator it self is really all that confusing. There are 3
>> >>>>>practical
>> >>>>> >> use cases:
>> >>>>> >> - Resource with no acl what so ever -> allow access to
>>everyone (
>> >>>>>just
>> >>>>> >>for
>> >>>>> >> backward compatibility, I would much rather fail close and
>>force
>> >>>>>users
>> >>>>> >>to
>> >>>>> >> explicitly grant acls that allows access to all users.)
>> >>>>> >> - Resource with some acl attached -> only users that have a
>> >>>>>matching
>> >>>>> >>allow
>> >>>>> >> acl are allowed (i.e. ³allow READ access to topic1 to user1
>>from
>> >>>>>all
>> >>>>> >> hosts², only user1 has READ access and no other user has
>>access of
>> >>>>>any
>> >>>>> >> kind)
>> >>>>> >> - Resource with some allow and some deny acl attached -> users
>>are
>> >>>>> >>allowed
>> >>>>> >> to perform operation only when they satisfy allow acl and do
>>not
>> >>>>>have
>> >>>>> >> conflicting deny acl. Users that have no acl(allow or deny)
>>will
>> >>>>>still
>> >>>>> >>not
>> >>>>> >> have any access. (i.e. ³allow READ access to topic1 to user1
>>from
>> >>>>>all
>> >>>>> >> hosts except host1 and host², only user1 has access but not
>>from
>> >>>>>host1
>> >>>>> >>an
>> >>>>> >> host2)
>> >>>>> >>
>> >>>>> >> I think we need to make a decision on deny primarily because
>>with
>> >>>>> >> introduction of acl management API, Acl is now a public class
>>that
>> >>>>>will
>> >>>>> >>be
>> >>>>> >> used by Ranger/Santry and other authroization providers. In
>> >>>>>Current
>> >>>>> >>design
>> >>>>> >> the acl has a permissionType enum field with possible values of
>> >>>>>Allow
>> >>>>> >>and
>> >>>>> >> Deny. If we chose to remove deny we can assume all acls to be
>>of
>> >>>>>allow
>> >>>>> >> type and remove the permissionType field completely.
>> >>>>> >>
>> >>>>> >> Thanks
>> >>>>> >> Parth
>> >>>>> >>
>> >>>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira"
>> >>>>><gs...@cloudera.com>> wrote:
>> >>>>> >>
>> >>>>> >> >I think thats how its done in pretty much any system I can
>>think
>> >>>>>of.
>> >>>>> >> >
>> >>>>> >>
>> >>>>> >>
>> >>>>>
>> >>>>>
>> >>
>> >>
>> >>
>> >>
>> >
>> >
>> >
>>
>>
>
>
>-- 
>Jeff Holoman
>Systems Engineer

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Jeff Holoman <jh...@cloudera.com>]

Parth,

This is a long thread, so trying to keep up here, sorry if this has been
covered before. First, great job on the KIP proposal and work so far.

Are we sure that we want to tie host level access to a given user? My
understanding is that the ACL will be (omitting some fields)

user_a, host1, host2, host3
user_b, host1, host2, host3

So there would potentially be a lot of redundancy in the configs. Does it
make sense to have hosts be at the same level as principal in the
hierarchy? This way you could just blanket the allowed / denied hosts and
only have to worry about the users. So if you follow this, then

we can wildcard the user so we can have a separate list of just host-based
access. What's the order that the perms would be evaluated if a there was
more than one match on a principal ?

Is the thought that there wouldn't usually be much overlap on hosts? I
guess I can imagine a scenario where I want to offline/online access to a
particular hosts or set of hosts and if there was overlap, I'm doing a
bunch of alter commands for just a single host. Maybe this is too contrived
an example?

I agree that having this level of granularity gives flexibility but I
wonder if people will actually use it and not just * the hosts for a given
user and create separate "global" list as i mentioned above?

The only other system I know of that ties users with hosts for access is
MySql and I don't love that model. Companies usually standardize on group
authorization anyway, are we complicating that issue with the inclusion of
hosts attached to users? Additionally I worry about the debt of big JSON
configs in the first place, most non-developers find them non-intuitive
already, so anything to ease this I think would be beneficial.


Thanks

Jeff

On Wed, Apr 22, 2015 at 2:22 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:

> Sorry I missed your last questions. I am +0 on adding ―host option for
> ―list, we could add it for symmetry. Again if this is only a CLI change it
> can be added later if you mean adding this in authorizer interface then we
> should make a decision now.
>
> Given a choice I would like to actually keep only one option which is
> resource based get (remove even the get based on principal). I see those
> (getAcl for principal or host) as special filtering case which can easily
> be achieved by a third party tool by doing "list all topics" and calling
> getAcls for each topic and applying filtering logic on that.  I really
> don’t see the need to make those first class citizens of the authorizer
> interface given these kind of queries will be issued outside of broker JVM
> so they will not benefit from the caching and because the storage will be
> indexed on resource both these options even as a first class API will just
> scan all topic acls and apply filtering logic.
>
> Thanks
> Parth
>
> On 4/22/15, 11:08 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>
> wrote:
>
> >Please see all the available options here
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+I
> >nterface#KIP-11-AuthorizationInterface-AclManagement(CLI) . I think it
> >covers both hosts and operations and allows to specify a list for both.
> >
> >Thanks
> >Parth
> >
> >From: Tom Graves <tg...@yahoo.com>>
> >Reply-To: Tom Graves <tg...@yahoo.com>>
> >Date: Wednesday, April 22, 2015 at 11:02 AM
> >To: Parth Brahmbhatt
> ><pb...@hortonworks.com>>,
> >"dev@kafka.apache.org<ma...@kafka.apache.org>"
> ><de...@kafka.apache.org>>
> >Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
> >
> >Thanks for the explanations Parth.
> >
> >On the configs questions, the way I see it is its more likely to
> >accidentally give everyone access, especially since you have to run a
> >separate command to change the acls. If there was some config for
> >defaults, a cluster admin could change that to be nobody or certain set
> >of users, then grant others permissions.  This would also remove the race
> >between commands.  This is something you can always add later though if
> >people request it.
> >
> >So in kafka-acl.sh how do I actually tell it what the operation is?
> >kafka-acl.sh --topic testtopic --add --grandprincipal user:joe,user:kate
> >
> >where does READ, WRITE, etc go?  Can specify as a list so I don't have to
> >run this a bunch of times for each.
> >
> >Do you want to have a --host option for --list so that admins could see
> >what acls apply to specific host(s)?
> >
> >Tom
> >
> >
> >
> >On Wednesday, April 22, 2015 11:38 AM, Parth Brahmbhatt
> ><pb...@hortonworks.com>> wrote:
> >
> >
> >
> >FYI, I have modified the KIP to include group as resource. In order to
> >access “joinGroup” and “commitOFfset” APIs the user will need a read
> >permission on topic and WRITE permission on group.
> >
> >I plan to open a VOTE thread by noon if there are no more concerns.
> >
> >Thanks
> >Parth
> >
> >On 4/22/15, 9:03 AM, "Tom Graves"
> ><tg...@yahoo.com.INVALID>>
> wrote:
> >
> >>Hey everyone,
> >>Sorry to jump in on the conversation so late. I'm new to Kafka. I'll
> >>apologize in advance if you have already covered some of my questions.  I
> >>read through the wiki and had some comments and questions.
> >>1) public enum Operation needs EDIT changed to ALTER
> >
> >>    Done.
> >
> >>2) Does the Authorizer class need a setAcls?  Rather then just add to be
> >>able to set to explicit list and overwrite what was there?  I see the
> >>kafka-acl.sh lists a removeall so I guess you could do removeall and then
> >>add.  I also don't see a removeall in the Authorizer class, is it going
> >>to loop through them all to remove each one?
> >
> >    There is an overloaded version of removeAcls in the interface that
> >takes
> >in resource as the only input and as described in the javadoc all the acls
> >attached to that resource will be deleted. To cover the setAcl use case
> >the caller can first call remove and then add.
> >
> >>3) Can someone tell me what the use case to do acls based on the hosts?
> >>I can see some possibilities just wondering if we can concrete ones where
> >>one user is allowed from one host but not another.
> >
> >    I am not sure if I understand the question given the use case you
> >described in your question is what we are trying to cover with use of
> >hosts in Acl. There are some additional use cases like “allow access to
> >any user from host1,host2” but I think primarily it gives the admins the
> >ability to define acls at a more granular level.
> >
> >>4) I'm a bit unclear how the "resource" works in the Authorizer class.
> >>From what I see we have 2 resources - topics and cluster.  If I want to
> >>add an acl to allow "joe" to CREATE for the cluster then I call addAcls
> >>with  Acl("user: joe", ALLOW, Set(*), Set(CREATE)) and "cluster"?  What
> >>if I want to call addAcls for DESCRIBE on a topic?  Is the resource then
> >>"topic" or is it the topic name?
> >
> >    We now have 3 resources(added group), please see the updated doc. The
> >CREATE acl that you described is correct. For any topic operation you
> >should use topic name as the resource name and for group the user will
> >provide groupId as resource name.
> >
> >>5) reassigning partitions is a CLUSTER_ACTION or superuser?  Its not
> >>totally clear to me the differences between these.  what about increasing
> >># of partitions?
> >
> >    I see this as an alter topic operation so it is at topic level and the
> >user must have alter permissions on topic.
> >
> >>6) groups are mentioned, are we supporting right away or is that a follow
> >>on item? (is there going to be a kafka.supergroups)
> >
> >    I think it can be a separate jira just for braking down the code
> >review
> >in smaller chunk. We will support it in first version but I think if we
> >can not do it for any reason that should not block a release with all the
> >other authZ work. We made deliberate design choices (like introducing a
> >principalType in KafkaPrinciapl) to allow supporting groups as an
> >incremental change.
> >
> >>7) Are there config options for setting acls when I create my topic?  Or
> >>do I have to create my topic and then run the kafka-acl.sh script to set
> >>them?  Although its very small, there would be possible race there that
> >>someone could start producing to topic before acls are set.
> >
> >    We discussed this yesterday and we agreed to go with kafka-acl.sh. Yes
> >there is a very very small window of vulnerability but I think that really
> >does not warrant to change the decision in this case.
> >
> >>8) are there configs for cluster level acl defaults?  Or does it default
> >>to superusers on bringing up new cluster and you have to modify with cli.
> >>thanks,Tom
> >
> >    No defaults, the default is superusers will have full access. I don’t
> >think making assumptions about ones security requirement should be our
> >burden.
> >
> >
> >>
> >>    On Tuesday, April 21, 2015 7:10 PM, Parth Brahmbhatt
> >><pb...@hortonworks.com>> wrote:
> >>
> >>
> >> I have added the notes to KIP-11 Open question sections.
> >>
> >>Thanks
> >>Parth
> >>
> >>On 4/21/15, 4:49 PM, "Gwen Shapira"
> >><gs...@cloudera.com>> wrote:
> >>
> >>>Adding my notes from today's call to the thread:
> >>>
> >>>** Deny or Allow all by default? We will add a configuration to
> >>>control this. The configuration will default to “allow” for backward
> >>>compatibility. Security admins can set it to "deny"
> >>>
> >>>** Storing ACLs for default authorizers: We'll store them in ZK. We'll
> >>>support pointing the authorizer to any ZK.
> >>>The use of ZK will be internal to the default authorizer. Authorizer
> >>>reads ACLs from cache every hour. We proposed having mechanism
> >>>(possibly via new ZK node) to tell broker to refresh the cache
> >>>immediately.
> >>>
> >>>** Support deny as permission type - we agreed to keep this.
> >>>
> >>>** Mapping operations to API: We may need to add Group as a resource,
> >>>with JoinGroup and OffsetCommit require privilege on the consumer
> >>>group.
> >>>This can be something we pass now and authorizers can support in
> >>>future. - Jay will write specifics to the mailing list discussion.
> >>>
> >>>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps
> >>><ja...@gmail.com>> wrote:
> >>>> Following up on the KIP discussion. Two options for authorizing
> >>>>consumers
> >>>> to read topic "t" as part of group "g":
> >>>> 1. READ permission on resource /topic/t
> >>>> 2. READ permission on resource /topic/t AND WRITE permission on
> >>>>/group/g
> >>>>
> >>>> The advantage of (1) is that it is simpler. The disadvantage is that
> >>>>any
> >>>> member of any group that reads from t can commit offsets as any other
> >>>> member of a different group. This doesn't effect data security (who
> >>>>can
> >>>> access what) but it is a bit of a management issue--a malicious person
> >>>>can
> >>>> cause data loss or duplicates for another consumer by committing
> >>>>offset.
> >>>>
> >>>> I think I favor (2) but it's worth it to think it through.
> >>>>
> >>>> -Jay
> >>>>
> >>>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
> >>>> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
> >>>>wrote:
> >>>>
> >>>>> Hey Jun,
> >>>>>
> >>>>> Yes and we support wild cards for all acl entities principal, hosts
> >>>>>and
> >>>>> operation.
> >>>>>
> >>>>> Thanks
> >>>>> Parth
> >>>>>
> >>>>> On 4/21/15, 9:06 AM, "Jun Rao"
> >>>>><ju...@confluent.io>> wrote:
> >>>>>
> >>>>> >Harsha, Parth,
> >>>>> >
> >>>>> >Thanks for the clarification. This makes sense. Perhaps we can
> >>>>>clarify the
> >>>>> >meaning of those rules in the wiki.
> >>>>> >
> >>>>> >Related to this, it seems that we need to support wildcard in
> >>>>>cli/request
> >>>>> >protocol for topics?
> >>>>> >
> >>>>> >Jun
> >>>>> >
> >>>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
> >>>>> >pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
> >>>>>wrote:
> >>>>> >
> >>>>> >> The iptables on unix supports the DENY operator, not that it
> >>>>>should
> >>>>> >> matter. The deny operator can also be used to specify ³allow user1
> >>>>>to
> >>>>> >>READ
> >>>>> >> from topic1 from all hosts but host1,host2². Again we could add a
> >>>>>host
> >>>>> >> group semantic and extra complexity around that, not sure if its
> >>>>>worth
> >>>>> >>it.
> >>>>> >> In addition with DENY operator you are now not forced to create a
> >>>>> >>special
> >>>>> >> group just to support the authorization use case. I am not
> >>>>>convinced
> >>>>> >>that
> >>>>> >> the operator it self is really all that confusing. There are 3
> >>>>>practical
> >>>>> >> use cases:
> >>>>> >> - Resource with no acl what so ever -> allow access to everyone (
> >>>>>just
> >>>>> >>for
> >>>>> >> backward compatibility, I would much rather fail close and force
> >>>>>users
> >>>>> >>to
> >>>>> >> explicitly grant acls that allows access to all users.)
> >>>>> >> - Resource with some acl attached -> only users that have a
> >>>>>matching
> >>>>> >>allow
> >>>>> >> acl are allowed (i.e. ³allow READ access to topic1 to user1 from
> >>>>>all
> >>>>> >> hosts², only user1 has READ access and no other user has access of
> >>>>>any
> >>>>> >> kind)
> >>>>> >> - Resource with some allow and some deny acl attached -> users are
> >>>>> >>allowed
> >>>>> >> to perform operation only when they satisfy allow acl and do not
> >>>>>have
> >>>>> >> conflicting deny acl. Users that have no acl(allow or deny) will
> >>>>>still
> >>>>> >>not
> >>>>> >> have any access. (i.e. ³allow READ access to topic1 to user1 from
> >>>>>all
> >>>>> >> hosts except host1 and host², only user1 has access but not from
> >>>>>host1
> >>>>> >>an
> >>>>> >> host2)
> >>>>> >>
> >>>>> >> I think we need to make a decision on deny primarily because with
> >>>>> >> introduction of acl management API, Acl is now a public class that
> >>>>>will
> >>>>> >>be
> >>>>> >> used by Ranger/Santry and other authroization providers. In
> >>>>>Current
> >>>>> >>design
> >>>>> >> the acl has a permissionType enum field with possible values of
> >>>>>Allow
> >>>>> >>and
> >>>>> >> Deny. If we chose to remove deny we can assume all acls to be of
> >>>>>allow
> >>>>> >> type and remove the permissionType field completely.
> >>>>> >>
> >>>>> >> Thanks
> >>>>> >> Parth
> >>>>> >>
> >>>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira"
> >>>>><gs...@cloudera.com>> wrote:
> >>>>> >>
> >>>>> >> >I think thats how its done in pretty much any system I can think
> >>>>>of.
> >>>>> >> >
> >>>>> >>
> >>>>> >>
> >>>>>
> >>>>>
> >>
> >>
> >>
> >>
> >
> >
> >
>
>


-- 
Jeff Holoman
Systems Engineer

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Tom Graves <tg...@yahoo.com.INVALID>]

Thanks!  Makes sense.
Tom 


     On Wednesday, April 22, 2015 1:25 PM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:
   

 You are right , I forgot to mention the ―operation option in CLI , I just
added it. Sorry for about the confusion.

Thanks
Parth

On 4/22/15, 11:22 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>
wrote:

>Sorry I missed your last questions. I am +0 on adding ―host option for
>―list, we could add it for symmetry. Again if this is only a CLI change it
>can be added later if you mean adding this in authorizer interface then we
>should make a decision now.
>
>Given a choice I would like to actually keep only one option which is
>resource based get (remove even the get based on principal). I see those
>(getAcl for principal or host) as special filtering case which can easily
>be achieved by a third party tool by doing "list all topics" and calling
>getAcls for each topic and applying filtering logic on that.  I really
>don’t see the need to make those first class citizens of the authorizer
>interface given these kind of queries will be issued outside of broker JVM
>so they will not benefit from the caching and because the storage will be
>indexed on resource both these options even as a first class API will just
>scan all topic acls and apply filtering logic.
>
>Thanks
>Parth
>
>On 4/22/15, 11:08 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>
>wrote:
>
>>Please see all the available options here
>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+
>>I
>>nterface#KIP-11-AuthorizationInterface-AclManagement(CLI) . I think it
>>covers both hosts and operations and allows to specify a list for both.
>>
>>Thanks
>>Parth
>>
>>From: Tom Graves <tg...@yahoo.com>>
>>Reply-To: Tom Graves <tg...@yahoo.com>>
>>Date: Wednesday, April 22, 2015 at 11:02 AM
>>To: Parth Brahmbhatt
>><pb...@hortonworks.com>>,
>>"dev@kafka.apache.org<ma...@kafka.apache.org>"
>><de...@kafka.apache.org>>
>>Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
>>
>>Thanks for the explanations Parth.
>>
>>On the configs questions, the way I see it is its more likely to
>>accidentally give everyone access, especially since you have to run a
>>separate command to change the acls. If there was some config for
>>defaults, a cluster admin could change that to be nobody or certain set
>>of users, then grant others permissions.  This would also remove the race
>>between commands.  This is something you can always add later though if
>>people request it.
>>
>>So in kafka-acl.sh how do I actually tell it what the operation is?
>>kafka-acl.sh --topic testtopic --add --grandprincipal user:joe,user:kate
>>
>>where does READ, WRITE, etc go?  Can specify as a list so I don't have to
>>run this a bunch of times for each.
>>
>>Do you want to have a --host option for --list so that admins could see
>>what acls apply to specific host(s)?
>>
>>Tom
>>
>>
>>
>>On Wednesday, April 22, 2015 11:38 AM, Parth Brahmbhatt
>><pb...@hortonworks.com>> wrote:
>>
>>
>>
>>FYI, I have modified the KIP to include group as resource. In order to
>>access “joinGroup” and “commitOFfset” APIs the user will need a read
>>permission on topic and WRITE permission on group.
>>
>>I plan to open a VOTE thread by noon if there are no more concerns.
>>
>>Thanks
>>Parth
>>
>>On 4/22/15, 9:03 AM, "Tom Graves"
>><tg...@yahoo.com.INVALID>>
>>wrote:
>>
>>>Hey everyone,
>>>Sorry to jump in on the conversation so late. I'm new to Kafka. I'll
>>>apologize in advance if you have already covered some of my questions.
>>>I
>>>read through the wiki and had some comments and questions.
>>>1) public enum Operation needs EDIT changed to ALTER
>>
>>>    Done.
>>
>>>2) Does the Authorizer class need a setAcls?  Rather then just add to be
>>>able to set to explicit list and overwrite what was there?  I see the
>>>kafka-acl.sh lists a removeall so I guess you could do removeall and
>>>then
>>>add.  I also don't see a removeall in the Authorizer class, is it going
>>>to loop through them all to remove each one?
>>
>>    There is an overloaded version of removeAcls in the interface that
>>takes
>>in resource as the only input and as described in the javadoc all the
>>acls
>>attached to that resource will be deleted. To cover the setAcl use case
>>the caller can first call remove and then add.
>>
>>>3) Can someone tell me what the use case to do acls based on the hosts?
>>>I can see some possibilities just wondering if we can concrete ones
>>>where
>>>one user is allowed from one host but not another.
>>
>>    I am not sure if I understand the question given the use case you
>>described in your question is what we are trying to cover with use of
>>hosts in Acl. There are some additional use cases like “allow access to
>>any user from host1,host2” but I think primarily it gives the admins the
>>ability to define acls at a more granular level.
>>
>>>4) I'm a bit unclear how the "resource" works in the Authorizer class.
>>>From what I see we have 2 resources - topics and cluster.  If I want to
>>>add an acl to allow "joe" to CREATE for the cluster then I call addAcls
>>>with  Acl("user: joe", ALLOW, Set(*), Set(CREATE)) and "cluster"?  What
>>>if I want to call addAcls for DESCRIBE on a topic?  Is the resource then
>>>"topic" or is it the topic name?
>>
>>    We now have 3 resources(added group), please see the updated doc. The
>>CREATE acl that you described is correct. For any topic operation you
>>should use topic name as the resource name and for group the user will
>>provide groupId as resource name.
>>
>>>5) reassigning partitions is a CLUSTER_ACTION or superuser?  Its not
>>>totally clear to me the differences between these.  what about
>>>increasing
>>># of partitions?
>>
>>    I see this as an alter topic operation so it is at topic level and
>>the
>>user must have alter permissions on topic.
>>
>>>6) groups are mentioned, are we supporting right away or is that a
>>>follow
>>>on item? (is there going to be a kafka.supergroups)
>>
>>    I think it can be a separate jira just for braking down the code
>>review
>>in smaller chunk. We will support it in first version but I think if we
>>can not do it for any reason that should not block a release with all the
>>other authZ work. We made deliberate design choices (like introducing a
>>principalType in KafkaPrinciapl) to allow supporting groups as an
>>incremental change.
>>
>>>7) Are there config options for setting acls when I create my topic?  Or
>>>do I have to create my topic and then run the kafka-acl.sh script to set
>>>them?  Although its very small, there would be possible race there that
>>>someone could start producing to topic before acls are set.
>>
>>    We discussed this yesterday and we agreed to go with kafka-acl.sh.
>>Yes
>>there is a very very small window of vulnerability but I think that
>>really
>>does not warrant to change the decision in this case.
>>
>>>8) are there configs for cluster level acl defaults?  Or does it default
>>>to superusers on bringing up new cluster and you have to modify with
>>>cli.
>>>thanks,Tom
>>
>>    No defaults, the default is superusers will have full access. I don’t
>>think making assumptions about ones security requirement should be our
>>burden.
>>
>>
>>>
>>>    On Tuesday, April 21, 2015 7:10 PM, Parth Brahmbhatt
>>><pb...@hortonworks.com>> wrote:
>>>
>>>
>>> I have added the notes to KIP-11 Open question sections.
>>>
>>>Thanks
>>>Parth
>>>
>>>On 4/21/15, 4:49 PM, "Gwen Shapira"
>>><gs...@cloudera.com>> wrote:
>>>
>>>>Adding my notes from today's call to the thread:
>>>>
>>>>** Deny or Allow all by default? We will add a configuration to
>>>>control this. The configuration will default to “allow” for backward
>>>>compatibility. Security admins can set it to "deny"
>>>>
>>>>** Storing ACLs for default authorizers: We'll store them in ZK. We'll
>>>>support pointing the authorizer to any ZK.
>>>>The use of ZK will be internal to the default authorizer. Authorizer
>>>>reads ACLs from cache every hour. We proposed having mechanism
>>>>(possibly via new ZK node) to tell broker to refresh the cache
>>>>immediately.
>>>>
>>>>** Support deny as permission type - we agreed to keep this.
>>>>
>>>>** Mapping operations to API: We may need to add Group as a resource,
>>>>with JoinGroup and OffsetCommit require privilege on the consumer
>>>>group.
>>>>This can be something we pass now and authorizers can support in
>>>>future. - Jay will write specifics to the mailing list discussion.
>>>>
>>>>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps
>>>><ja...@gmail.com>> wrote:
>>>>> Following up on the KIP discussion. Two options for authorizing
>>>>>consumers
>>>>> to read topic "t" as part of group "g":
>>>>> 1. READ permission on resource /topic/t
>>>>> 2. READ permission on resource /topic/t AND WRITE permission on
>>>>>/group/g
>>>>>
>>>>> The advantage of (1) is that it is simpler. The disadvantage is that
>>>>>any
>>>>> member of any group that reads from t can commit offsets as any other
>>>>> member of a different group. This doesn't effect data security (who
>>>>>can
>>>>> access what) but it is a bit of a management issue--a malicious
>>>>>person
>>>>>can
>>>>> cause data loss or duplicates for another consumer by committing
>>>>>offset.
>>>>>
>>>>> I think I favor (2) but it's worth it to think it through.
>>>>>
>>>>> -Jay
>>>>>
>>>>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
>>>>> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
>>>>>wrote:
>>>>>
>>>>>> Hey Jun,
>>>>>>
>>>>>> Yes and we support wild cards for all acl entities principal, hosts
>>>>>>and
>>>>>> operation.
>>>>>>
>>>>>> Thanks
>>>>>> Parth
>>>>>>
>>>>>> On 4/21/15, 9:06 AM, "Jun Rao"
>>>>>><ju...@confluent.io>> wrote:
>>>>>>
>>>>>> >Harsha, Parth,
>>>>>> >
>>>>>> >Thanks for the clarification. This makes sense. Perhaps we can
>>>>>>clarify the
>>>>>> >meaning of those rules in the wiki.
>>>>>> >
>>>>>> >Related to this, it seems that we need to support wildcard in
>>>>>>cli/request
>>>>>> >protocol for topics?
>>>>>> >
>>>>>> >Jun
>>>>>> >
>>>>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>>>>>> >pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
>>>>>>wrote:
>>>>>> >
>>>>>> >> The iptables on unix supports the DENY operator, not that it
>>>>>>should
>>>>>> >> matter. The deny operator can also be used to specify ³allow
>>>>>>user1
>>>>>>to
>>>>>> >>READ
>>>>>> >> from topic1 from all hosts but host1,host2². Again we could add a
>>>>>>host
>>>>>> >> group semantic and extra complexity around that, not sure if its
>>>>>>worth
>>>>>> >>it.
>>>>>> >> In addition with DENY operator you are now not forced to create a
>>>>>> >>special
>>>>>> >> group just to support the authorization use case. I am not
>>>>>>convinced
>>>>>> >>that
>>>>>> >> the operator it self is really all that confusing. There are 3
>>>>>>practical
>>>>>> >> use cases:
>>>>>> >> - Resource with no acl what so ever -> allow access to everyone (
>>>>>>just
>>>>>> >>for
>>>>>> >> backward compatibility, I would much rather fail close and force
>>>>>>users
>>>>>> >>to
>>>>>> >> explicitly grant acls that allows access to all users.)
>>>>>> >> - Resource with some acl attached -> only users that have a
>>>>>>matching
>>>>>> >>allow
>>>>>> >> acl are allowed (i.e. ³allow READ access to topic1 to user1 from
>>>>>>all
>>>>>> >> hosts², only user1 has READ access and no other user has access
>>>>>>of
>>>>>>any
>>>>>> >> kind)
>>>>>> >> - Resource with some allow and some deny acl attached -> users
>>>>>>are
>>>>>> >>allowed
>>>>>> >> to perform operation only when they satisfy allow acl and do not
>>>>>>have
>>>>>> >> conflicting deny acl. Users that have no acl(allow or deny) will
>>>>>>still
>>>>>> >>not
>>>>>> >> have any access. (i.e. ³allow READ access to topic1 to user1 from
>>>>>>all
>>>>>> >> hosts except host1 and host², only user1 has access but not from
>>>>>>host1
>>>>>> >>an
>>>>>> >> host2)
>>>>>> >>
>>>>>> >> I think we need to make a decision on deny primarily because with
>>>>>> >> introduction of acl management API, Acl is now a public class
>>>>>>that
>>>>>>will
>>>>>> >>be
>>>>>> >> used by Ranger/Santry and other authroization providers. In
>>>>>>Current
>>>>>> >>design
>>>>>> >> the acl has a permissionType enum field with possible values of
>>>>>>Allow
>>>>>> >>and
>>>>>> >> Deny. If we chose to remove deny we can assume all acls to be of
>>>>>>allow
>>>>>> >> type and remove the permissionType field completely.
>>>>>> >>
>>>>>> >> Thanks
>>>>>> >> Parth
>>>>>> >>
>>>>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira"
>>>>>><gs...@cloudera.com>> wrote:
>>>>>> >>
>>>>>> >> >I think thats how its done in pretty much any system I can think
>>>>>>of.
>>>>>> >> >
>>>>>> >>
>>>>>> >>
>>>>>>
>>>>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>



  

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

You are right , I forgot to mention the ―operation option in CLI , I just
added it. Sorry for about the confusion.

Thanks
Parth

On 4/22/15, 11:22 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>
wrote:

>Sorry I missed your last questions. I am +0 on adding ―host option for
>―list, we could add it for symmetry. Again if this is only a CLI change it
>can be added later if you mean adding this in authorizer interface then we
>should make a decision now.
>
>Given a choice I would like to actually keep only one option which is
>resource based get (remove even the get based on principal). I see those
>(getAcl for principal or host) as special filtering case which can easily
>be achieved by a third party tool by doing "list all topics" and calling
>getAcls for each topic and applying filtering logic on that.  I really
>don’t see the need to make those first class citizens of the authorizer
>interface given these kind of queries will be issued outside of broker JVM
>so they will not benefit from the caching and because the storage will be
>indexed on resource both these options even as a first class API will just
>scan all topic acls and apply filtering logic.
>
>Thanks
>Parth
>
>On 4/22/15, 11:08 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>
>wrote:
>
>>Please see all the available options here
>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+
>>I
>>nterface#KIP-11-AuthorizationInterface-AclManagement(CLI) . I think it
>>covers both hosts and operations and allows to specify a list for both.
>>
>>Thanks
>>Parth
>>
>>From: Tom Graves <tg...@yahoo.com>>
>>Reply-To: Tom Graves <tg...@yahoo.com>>
>>Date: Wednesday, April 22, 2015 at 11:02 AM
>>To: Parth Brahmbhatt
>><pb...@hortonworks.com>>,
>>"dev@kafka.apache.org<ma...@kafka.apache.org>"
>><de...@kafka.apache.org>>
>>Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
>>
>>Thanks for the explanations Parth.
>>
>>On the configs questions, the way I see it is its more likely to
>>accidentally give everyone access, especially since you have to run a
>>separate command to change the acls. If there was some config for
>>defaults, a cluster admin could change that to be nobody or certain set
>>of users, then grant others permissions.  This would also remove the race
>>between commands.  This is something you can always add later though if
>>people request it.
>>
>>So in kafka-acl.sh how do I actually tell it what the operation is?
>>kafka-acl.sh --topic testtopic --add --grandprincipal user:joe,user:kate
>>
>>where does READ, WRITE, etc go?  Can specify as a list so I don't have to
>>run this a bunch of times for each.
>>
>>Do you want to have a --host option for --list so that admins could see
>>what acls apply to specific host(s)?
>>
>>Tom
>>
>>
>>
>>On Wednesday, April 22, 2015 11:38 AM, Parth Brahmbhatt
>><pb...@hortonworks.com>> wrote:
>>
>>
>>
>>FYI, I have modified the KIP to include group as resource. In order to
>>access “joinGroup” and “commitOFfset” APIs the user will need a read
>>permission on topic and WRITE permission on group.
>>
>>I plan to open a VOTE thread by noon if there are no more concerns.
>>
>>Thanks
>>Parth
>>
>>On 4/22/15, 9:03 AM, "Tom Graves"
>><tg...@yahoo.com.INVALID>>
>>wrote:
>>
>>>Hey everyone,
>>>Sorry to jump in on the conversation so late. I'm new to Kafka. I'll
>>>apologize in advance if you have already covered some of my questions.
>>>I
>>>read through the wiki and had some comments and questions.
>>>1) public enum Operation needs EDIT changed to ALTER
>>
>>>    Done.
>>
>>>2) Does the Authorizer class need a setAcls?  Rather then just add to be
>>>able to set to explicit list and overwrite what was there?  I see the
>>>kafka-acl.sh lists a removeall so I guess you could do removeall and
>>>then
>>>add.  I also don't see a removeall in the Authorizer class, is it going
>>>to loop through them all to remove each one?
>>
>>    There is an overloaded version of removeAcls in the interface that
>>takes
>>in resource as the only input and as described in the javadoc all the
>>acls
>>attached to that resource will be deleted. To cover the setAcl use case
>>the caller can first call remove and then add.
>>
>>>3) Can someone tell me what the use case to do acls based on the hosts?
>>>I can see some possibilities just wondering if we can concrete ones
>>>where
>>>one user is allowed from one host but not another.
>>
>>    I am not sure if I understand the question given the use case you
>>described in your question is what we are trying to cover with use of
>>hosts in Acl. There are some additional use cases like “allow access to
>>any user from host1,host2” but I think primarily it gives the admins the
>>ability to define acls at a more granular level.
>>
>>>4) I'm a bit unclear how the "resource" works in the Authorizer class.
>>>From what I see we have 2 resources - topics and cluster.  If I want to
>>>add an acl to allow "joe" to CREATE for the cluster then I call addAcls
>>>with  Acl("user: joe", ALLOW, Set(*), Set(CREATE)) and "cluster"?  What
>>>if I want to call addAcls for DESCRIBE on a topic?  Is the resource then
>>>"topic" or is it the topic name?
>>
>>    We now have 3 resources(added group), please see the updated doc. The
>>CREATE acl that you described is correct. For any topic operation you
>>should use topic name as the resource name and for group the user will
>>provide groupId as resource name.
>>
>>>5) reassigning partitions is a CLUSTER_ACTION or superuser?  Its not
>>>totally clear to me the differences between these.  what about
>>>increasing
>>># of partitions?
>>
>>    I see this as an alter topic operation so it is at topic level and
>>the
>>user must have alter permissions on topic.
>>
>>>6) groups are mentioned, are we supporting right away or is that a
>>>follow
>>>on item? (is there going to be a kafka.supergroups)
>>
>>    I think it can be a separate jira just for braking down the code
>>review
>>in smaller chunk. We will support it in first version but I think if we
>>can not do it for any reason that should not block a release with all the
>>other authZ work. We made deliberate design choices (like introducing a
>>principalType in KafkaPrinciapl) to allow supporting groups as an
>>incremental change.
>>
>>>7) Are there config options for setting acls when I create my topic?  Or
>>>do I have to create my topic and then run the kafka-acl.sh script to set
>>>them?  Although its very small, there would be possible race there that
>>>someone could start producing to topic before acls are set.
>>
>>    We discussed this yesterday and we agreed to go with kafka-acl.sh.
>>Yes
>>there is a very very small window of vulnerability but I think that
>>really
>>does not warrant to change the decision in this case.
>>
>>>8) are there configs for cluster level acl defaults?  Or does it default
>>>to superusers on bringing up new cluster and you have to modify with
>>>cli.
>>>thanks,Tom
>>
>>    No defaults, the default is superusers will have full access. I don’t
>>think making assumptions about ones security requirement should be our
>>burden.
>>
>>
>>>
>>>    On Tuesday, April 21, 2015 7:10 PM, Parth Brahmbhatt
>>><pb...@hortonworks.com>> wrote:
>>>
>>>
>>> I have added the notes to KIP-11 Open question sections.
>>>
>>>Thanks
>>>Parth
>>>
>>>On 4/21/15, 4:49 PM, "Gwen Shapira"
>>><gs...@cloudera.com>> wrote:
>>>
>>>>Adding my notes from today's call to the thread:
>>>>
>>>>** Deny or Allow all by default? We will add a configuration to
>>>>control this. The configuration will default to “allow” for backward
>>>>compatibility. Security admins can set it to "deny"
>>>>
>>>>** Storing ACLs for default authorizers: We'll store them in ZK. We'll
>>>>support pointing the authorizer to any ZK.
>>>>The use of ZK will be internal to the default authorizer. Authorizer
>>>>reads ACLs from cache every hour. We proposed having mechanism
>>>>(possibly via new ZK node) to tell broker to refresh the cache
>>>>immediately.
>>>>
>>>>** Support deny as permission type - we agreed to keep this.
>>>>
>>>>** Mapping operations to API: We may need to add Group as a resource,
>>>>with JoinGroup and OffsetCommit require privilege on the consumer
>>>>group.
>>>>This can be something we pass now and authorizers can support in
>>>>future. - Jay will write specifics to the mailing list discussion.
>>>>
>>>>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps
>>>><ja...@gmail.com>> wrote:
>>>>> Following up on the KIP discussion. Two options for authorizing
>>>>>consumers
>>>>> to read topic "t" as part of group "g":
>>>>> 1. READ permission on resource /topic/t
>>>>> 2. READ permission on resource /topic/t AND WRITE permission on
>>>>>/group/g
>>>>>
>>>>> The advantage of (1) is that it is simpler. The disadvantage is that
>>>>>any
>>>>> member of any group that reads from t can commit offsets as any other
>>>>> member of a different group. This doesn't effect data security (who
>>>>>can
>>>>> access what) but it is a bit of a management issue--a malicious
>>>>>person
>>>>>can
>>>>> cause data loss or duplicates for another consumer by committing
>>>>>offset.
>>>>>
>>>>> I think I favor (2) but it's worth it to think it through.
>>>>>
>>>>> -Jay
>>>>>
>>>>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
>>>>> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
>>>>>wrote:
>>>>>
>>>>>> Hey Jun,
>>>>>>
>>>>>> Yes and we support wild cards for all acl entities principal, hosts
>>>>>>and
>>>>>> operation.
>>>>>>
>>>>>> Thanks
>>>>>> Parth
>>>>>>
>>>>>> On 4/21/15, 9:06 AM, "Jun Rao"
>>>>>><ju...@confluent.io>> wrote:
>>>>>>
>>>>>> >Harsha, Parth,
>>>>>> >
>>>>>> >Thanks for the clarification. This makes sense. Perhaps we can
>>>>>>clarify the
>>>>>> >meaning of those rules in the wiki.
>>>>>> >
>>>>>> >Related to this, it seems that we need to support wildcard in
>>>>>>cli/request
>>>>>> >protocol for topics?
>>>>>> >
>>>>>> >Jun
>>>>>> >
>>>>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>>>>>> >pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
>>>>>>wrote:
>>>>>> >
>>>>>> >> The iptables on unix supports the DENY operator, not that it
>>>>>>should
>>>>>> >> matter. The deny operator can also be used to specify ³allow
>>>>>>user1
>>>>>>to
>>>>>> >>READ
>>>>>> >> from topic1 from all hosts but host1,host2². Again we could add a
>>>>>>host
>>>>>> >> group semantic and extra complexity around that, not sure if its
>>>>>>worth
>>>>>> >>it.
>>>>>> >> In addition with DENY operator you are now not forced to create a
>>>>>> >>special
>>>>>> >> group just to support the authorization use case. I am not
>>>>>>convinced
>>>>>> >>that
>>>>>> >> the operator it self is really all that confusing. There are 3
>>>>>>practical
>>>>>> >> use cases:
>>>>>> >> - Resource with no acl what so ever -> allow access to everyone (
>>>>>>just
>>>>>> >>for
>>>>>> >> backward compatibility, I would much rather fail close and force
>>>>>>users
>>>>>> >>to
>>>>>> >> explicitly grant acls that allows access to all users.)
>>>>>> >> - Resource with some acl attached -> only users that have a
>>>>>>matching
>>>>>> >>allow
>>>>>> >> acl are allowed (i.e. ³allow READ access to topic1 to user1 from
>>>>>>all
>>>>>> >> hosts², only user1 has READ access and no other user has access
>>>>>>of
>>>>>>any
>>>>>> >> kind)
>>>>>> >> - Resource with some allow and some deny acl attached -> users
>>>>>>are
>>>>>> >>allowed
>>>>>> >> to perform operation only when they satisfy allow acl and do not
>>>>>>have
>>>>>> >> conflicting deny acl. Users that have no acl(allow or deny) will
>>>>>>still
>>>>>> >>not
>>>>>> >> have any access. (i.e. ³allow READ access to topic1 to user1 from
>>>>>>all
>>>>>> >> hosts except host1 and host², only user1 has access but not from
>>>>>>host1
>>>>>> >>an
>>>>>> >> host2)
>>>>>> >>
>>>>>> >> I think we need to make a decision on deny primarily because with
>>>>>> >> introduction of acl management API, Acl is now a public class
>>>>>>that
>>>>>>will
>>>>>> >>be
>>>>>> >> used by Ranger/Santry and other authroization providers. In
>>>>>>Current
>>>>>> >>design
>>>>>> >> the acl has a permissionType enum field with possible values of
>>>>>>Allow
>>>>>> >>and
>>>>>> >> Deny. If we chose to remove deny we can assume all acls to be of
>>>>>>allow
>>>>>> >> type and remove the permissionType field completely.
>>>>>> >>
>>>>>> >> Thanks
>>>>>> >> Parth
>>>>>> >>
>>>>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira"
>>>>>><gs...@cloudera.com>> wrote:
>>>>>> >>
>>>>>> >> >I think thats how its done in pretty much any system I can think
>>>>>>of.
>>>>>> >> >
>>>>>> >>
>>>>>> >>
>>>>>>
>>>>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Sorry I missed your last questions. I am +0 on adding ―host option for
―list, we could add it for symmetry. Again if this is only a CLI change it
can be added later if you mean adding this in authorizer interface then we
should make a decision now.

Given a choice I would like to actually keep only one option which is
resource based get (remove even the get based on principal). I see those
(getAcl for principal or host) as special filtering case which can easily
be achieved by a third party tool by doing "list all topics" and calling
getAcls for each topic and applying filtering logic on that.  I really
don’t see the need to make those first class citizens of the authorizer
interface given these kind of queries will be issued outside of broker JVM
so they will not benefit from the caching and because the storage will be
indexed on resource both these options even as a first class API will just
scan all topic acls and apply filtering logic.

Thanks
Parth

On 4/22/15, 11:08 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>
wrote:

>Please see all the available options here
>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+I
>nterface#KIP-11-AuthorizationInterface-AclManagement(CLI) . I think it
>covers both hosts and operations and allows to specify a list for both.
>
>Thanks
>Parth
>
>From: Tom Graves <tg...@yahoo.com>>
>Reply-To: Tom Graves <tg...@yahoo.com>>
>Date: Wednesday, April 22, 2015 at 11:02 AM
>To: Parth Brahmbhatt
><pb...@hortonworks.com>>,
>"dev@kafka.apache.org<ma...@kafka.apache.org>"
><de...@kafka.apache.org>>
>Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
>
>Thanks for the explanations Parth.
>
>On the configs questions, the way I see it is its more likely to
>accidentally give everyone access, especially since you have to run a
>separate command to change the acls. If there was some config for
>defaults, a cluster admin could change that to be nobody or certain set
>of users, then grant others permissions.  This would also remove the race
>between commands.  This is something you can always add later though if
>people request it.
>
>So in kafka-acl.sh how do I actually tell it what the operation is?
>kafka-acl.sh --topic testtopic --add --grandprincipal user:joe,user:kate
>
>where does READ, WRITE, etc go?  Can specify as a list so I don't have to
>run this a bunch of times for each.
>
>Do you want to have a --host option for --list so that admins could see
>what acls apply to specific host(s)?
>
>Tom
>
>
>
>On Wednesday, April 22, 2015 11:38 AM, Parth Brahmbhatt
><pb...@hortonworks.com>> wrote:
>
>
>
>FYI, I have modified the KIP to include group as resource. In order to
>access “joinGroup” and “commitOFfset” APIs the user will need a read
>permission on topic and WRITE permission on group.
>
>I plan to open a VOTE thread by noon if there are no more concerns.
>
>Thanks
>Parth
>
>On 4/22/15, 9:03 AM, "Tom Graves"
><tg...@yahoo.com.INVALID>> wrote:
>
>>Hey everyone,
>>Sorry to jump in on the conversation so late. I'm new to Kafka. I'll
>>apologize in advance if you have already covered some of my questions.  I
>>read through the wiki and had some comments and questions.
>>1) public enum Operation needs EDIT changed to ALTER
>
>>    Done.
>
>>2) Does the Authorizer class need a setAcls?  Rather then just add to be
>>able to set to explicit list and overwrite what was there?  I see the
>>kafka-acl.sh lists a removeall so I guess you could do removeall and then
>>add.  I also don't see a removeall in the Authorizer class, is it going
>>to loop through them all to remove each one?
>
>    There is an overloaded version of removeAcls in the interface that
>takes
>in resource as the only input and as described in the javadoc all the acls
>attached to that resource will be deleted. To cover the setAcl use case
>the caller can first call remove and then add.
>
>>3) Can someone tell me what the use case to do acls based on the hosts?
>>I can see some possibilities just wondering if we can concrete ones where
>>one user is allowed from one host but not another.
>
>    I am not sure if I understand the question given the use case you
>described in your question is what we are trying to cover with use of
>hosts in Acl. There are some additional use cases like “allow access to
>any user from host1,host2” but I think primarily it gives the admins the
>ability to define acls at a more granular level.
>
>>4) I'm a bit unclear how the "resource" works in the Authorizer class.
>>From what I see we have 2 resources - topics and cluster.  If I want to
>>add an acl to allow "joe" to CREATE for the cluster then I call addAcls
>>with  Acl("user: joe", ALLOW, Set(*), Set(CREATE)) and "cluster"?  What
>>if I want to call addAcls for DESCRIBE on a topic?  Is the resource then
>>"topic" or is it the topic name?
>
>    We now have 3 resources(added group), please see the updated doc. The
>CREATE acl that you described is correct. For any topic operation you
>should use topic name as the resource name and for group the user will
>provide groupId as resource name.
>
>>5) reassigning partitions is a CLUSTER_ACTION or superuser?  Its not
>>totally clear to me the differences between these.  what about increasing
>># of partitions?
>
>    I see this as an alter topic operation so it is at topic level and the
>user must have alter permissions on topic.
>
>>6) groups are mentioned, are we supporting right away or is that a follow
>>on item? (is there going to be a kafka.supergroups)
>
>    I think it can be a separate jira just for braking down the code
>review
>in smaller chunk. We will support it in first version but I think if we
>can not do it for any reason that should not block a release with all the
>other authZ work. We made deliberate design choices (like introducing a
>principalType in KafkaPrinciapl) to allow supporting groups as an
>incremental change.
>
>>7) Are there config options for setting acls when I create my topic?  Or
>>do I have to create my topic and then run the kafka-acl.sh script to set
>>them?  Although its very small, there would be possible race there that
>>someone could start producing to topic before acls are set.
>
>    We discussed this yesterday and we agreed to go with kafka-acl.sh. Yes
>there is a very very small window of vulnerability but I think that really
>does not warrant to change the decision in this case.
>
>>8) are there configs for cluster level acl defaults?  Or does it default
>>to superusers on bringing up new cluster and you have to modify with cli.
>>thanks,Tom
>
>    No defaults, the default is superusers will have full access. I don’t
>think making assumptions about ones security requirement should be our
>burden.
>
>
>>
>>    On Tuesday, April 21, 2015 7:10 PM, Parth Brahmbhatt
>><pb...@hortonworks.com>> wrote:
>>
>>
>> I have added the notes to KIP-11 Open question sections.
>>
>>Thanks
>>Parth
>>
>>On 4/21/15, 4:49 PM, "Gwen Shapira"
>><gs...@cloudera.com>> wrote:
>>
>>>Adding my notes from today's call to the thread:
>>>
>>>** Deny or Allow all by default? We will add a configuration to
>>>control this. The configuration will default to “allow” for backward
>>>compatibility. Security admins can set it to "deny"
>>>
>>>** Storing ACLs for default authorizers: We'll store them in ZK. We'll
>>>support pointing the authorizer to any ZK.
>>>The use of ZK will be internal to the default authorizer. Authorizer
>>>reads ACLs from cache every hour. We proposed having mechanism
>>>(possibly via new ZK node) to tell broker to refresh the cache
>>>immediately.
>>>
>>>** Support deny as permission type - we agreed to keep this.
>>>
>>>** Mapping operations to API: We may need to add Group as a resource,
>>>with JoinGroup and OffsetCommit require privilege on the consumer
>>>group.
>>>This can be something we pass now and authorizers can support in
>>>future. - Jay will write specifics to the mailing list discussion.
>>>
>>>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps
>>><ja...@gmail.com>> wrote:
>>>> Following up on the KIP discussion. Two options for authorizing
>>>>consumers
>>>> to read topic "t" as part of group "g":
>>>> 1. READ permission on resource /topic/t
>>>> 2. READ permission on resource /topic/t AND WRITE permission on
>>>>/group/g
>>>>
>>>> The advantage of (1) is that it is simpler. The disadvantage is that
>>>>any
>>>> member of any group that reads from t can commit offsets as any other
>>>> member of a different group. This doesn't effect data security (who
>>>>can
>>>> access what) but it is a bit of a management issue--a malicious person
>>>>can
>>>> cause data loss or duplicates for another consumer by committing
>>>>offset.
>>>>
>>>> I think I favor (2) but it's worth it to think it through.
>>>>
>>>> -Jay
>>>>
>>>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
>>>> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
>>>>wrote:
>>>>
>>>>> Hey Jun,
>>>>>
>>>>> Yes and we support wild cards for all acl entities principal, hosts
>>>>>and
>>>>> operation.
>>>>>
>>>>> Thanks
>>>>> Parth
>>>>>
>>>>> On 4/21/15, 9:06 AM, "Jun Rao"
>>>>><ju...@confluent.io>> wrote:
>>>>>
>>>>> >Harsha, Parth,
>>>>> >
>>>>> >Thanks for the clarification. This makes sense. Perhaps we can
>>>>>clarify the
>>>>> >meaning of those rules in the wiki.
>>>>> >
>>>>> >Related to this, it seems that we need to support wildcard in
>>>>>cli/request
>>>>> >protocol for topics?
>>>>> >
>>>>> >Jun
>>>>> >
>>>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>>>>> >pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
>>>>>wrote:
>>>>> >
>>>>> >> The iptables on unix supports the DENY operator, not that it
>>>>>should
>>>>> >> matter. The deny operator can also be used to specify ³allow user1
>>>>>to
>>>>> >>READ
>>>>> >> from topic1 from all hosts but host1,host2². Again we could add a
>>>>>host
>>>>> >> group semantic and extra complexity around that, not sure if its
>>>>>worth
>>>>> >>it.
>>>>> >> In addition with DENY operator you are now not forced to create a
>>>>> >>special
>>>>> >> group just to support the authorization use case. I am not
>>>>>convinced
>>>>> >>that
>>>>> >> the operator it self is really all that confusing. There are 3
>>>>>practical
>>>>> >> use cases:
>>>>> >> - Resource with no acl what so ever -> allow access to everyone (
>>>>>just
>>>>> >>for
>>>>> >> backward compatibility, I would much rather fail close and force
>>>>>users
>>>>> >>to
>>>>> >> explicitly grant acls that allows access to all users.)
>>>>> >> - Resource with some acl attached -> only users that have a
>>>>>matching
>>>>> >>allow
>>>>> >> acl are allowed (i.e. ³allow READ access to topic1 to user1 from
>>>>>all
>>>>> >> hosts², only user1 has READ access and no other user has access of
>>>>>any
>>>>> >> kind)
>>>>> >> - Resource with some allow and some deny acl attached -> users are
>>>>> >>allowed
>>>>> >> to perform operation only when they satisfy allow acl and do not
>>>>>have
>>>>> >> conflicting deny acl. Users that have no acl(allow or deny) will
>>>>>still
>>>>> >>not
>>>>> >> have any access. (i.e. ³allow READ access to topic1 to user1 from
>>>>>all
>>>>> >> hosts except host1 and host², only user1 has access but not from
>>>>>host1
>>>>> >>an
>>>>> >> host2)
>>>>> >>
>>>>> >> I think we need to make a decision on deny primarily because with
>>>>> >> introduction of acl management API, Acl is now a public class that
>>>>>will
>>>>> >>be
>>>>> >> used by Ranger/Santry and other authroization providers. In
>>>>>Current
>>>>> >>design
>>>>> >> the acl has a permissionType enum field with possible values of
>>>>>Allow
>>>>> >>and
>>>>> >> Deny. If we chose to remove deny we can assume all acls to be of
>>>>>allow
>>>>> >> type and remove the permissionType field completely.
>>>>> >>
>>>>> >> Thanks
>>>>> >> Parth
>>>>> >>
>>>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira"
>>>>><gs...@cloudera.com>> wrote:
>>>>> >>
>>>>> >> >I think thats how its done in pretty much any system I can think
>>>>>of.
>>>>> >> >
>>>>> >>
>>>>> >>
>>>>>
>>>>>
>>
>>
>>
>>
>
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Please see all the available options here https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface#KIP-11-AuthorizationInterface-AclManagement(CLI) . I think it covers both hosts and operations and allows to specify a list for both.

Thanks
Parth

From: Tom Graves <tg...@yahoo.com>>
Reply-To: Tom Graves <tg...@yahoo.com>>
Date: Wednesday, April 22, 2015 at 11:02 AM
To: Parth Brahmbhatt <pb...@hortonworks.com>>, "dev@kafka.apache.org<ma...@kafka.apache.org>" <de...@kafka.apache.org>>
Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security

Thanks for the explanations Parth.

On the configs questions, the way I see it is its more likely to accidentally give everyone access, especially since you have to run a separate command to change the acls. If there was some config for defaults, a cluster admin could change that to be nobody or certain set of users, then grant others permissions.  This would also remove the race between commands.  This is something you can always add later though if people request it.

So in kafka-acl.sh how do I actually tell it what the operation is?
kafka-acl.sh --topic testtopic --add --grandprincipal user:joe,user:kate

where does READ, WRITE, etc go?  Can specify as a list so I don't have to run this a bunch of times for each.

Do you want to have a --host option for --list so that admins could see what acls apply to specific host(s)?

Tom



On Wednesday, April 22, 2015 11:38 AM, Parth Brahmbhatt <pb...@hortonworks.com>> wrote:



FYI, I have modified the KIP to include group as resource. In order to
access “joinGroup” and “commitOFfset” APIs the user will need a read
permission on topic and WRITE permission on group.

I plan to open a VOTE thread by noon if there are no more concerns.

Thanks
Parth

On 4/22/15, 9:03 AM, "Tom Graves" <tg...@yahoo.com.INVALID>> wrote:

>Hey everyone,
>Sorry to jump in on the conversation so late. I'm new to Kafka. I'll
>apologize in advance if you have already covered some of my questions.  I
>read through the wiki and had some comments and questions.
>1) public enum Operation needs EDIT changed to ALTER

>    Done.

>2) Does the Authorizer class need a setAcls?  Rather then just add to be
>able to set to explicit list and overwrite what was there?  I see the
>kafka-acl.sh lists a removeall so I guess you could do removeall and then
>add.  I also don't see a removeall in the Authorizer class, is it going
>to loop through them all to remove each one?

    There is an overloaded version of removeAcls in the interface that takes
in resource as the only input and as described in the javadoc all the acls
attached to that resource will be deleted. To cover the setAcl use case
the caller can first call remove and then add.

>3) Can someone tell me what the use case to do acls based on the hosts?
>I can see some possibilities just wondering if we can concrete ones where
>one user is allowed from one host but not another.

    I am not sure if I understand the question given the use case you
described in your question is what we are trying to cover with use of
hosts in Acl. There are some additional use cases like “allow access to
any user from host1,host2” but I think primarily it gives the admins the
ability to define acls at a more granular level.

>4) I'm a bit unclear how the "resource" works in the Authorizer class.
>From what I see we have 2 resources - topics and cluster.  If I want to
>add an acl to allow "joe" to CREATE for the cluster then I call addAcls
>with  Acl("user: joe", ALLOW, Set(*), Set(CREATE)) and "cluster"?  What
>if I want to call addAcls for DESCRIBE on a topic?  Is the resource then
>"topic" or is it the topic name?

    We now have 3 resources(added group), please see the updated doc. The
CREATE acl that you described is correct. For any topic operation you
should use topic name as the resource name and for group the user will
provide groupId as resource name.

>5) reassigning partitions is a CLUSTER_ACTION or superuser?  Its not
>totally clear to me the differences between these.  what about increasing
># of partitions?

    I see this as an alter topic operation so it is at topic level and the
user must have alter permissions on topic.

>6) groups are mentioned, are we supporting right away or is that a follow
>on item? (is there going to be a kafka.supergroups)

    I think it can be a separate jira just for braking down the code review
in smaller chunk. We will support it in first version but I think if we
can not do it for any reason that should not block a release with all the
other authZ work. We made deliberate design choices (like introducing a
principalType in KafkaPrinciapl) to allow supporting groups as an
incremental change.

>7) Are there config options for setting acls when I create my topic?  Or
>do I have to create my topic and then run the kafka-acl.sh script to set
>them?  Although its very small, there would be possible race there that
>someone could start producing to topic before acls are set.

    We discussed this yesterday and we agreed to go with kafka-acl.sh. Yes
there is a very very small window of vulnerability but I think that really
does not warrant to change the decision in this case.

>8) are there configs for cluster level acl defaults?  Or does it default
>to superusers on bringing up new cluster and you have to modify with cli.
>thanks,Tom

    No defaults, the default is superusers will have full access. I don’t
think making assumptions about ones security requirement should be our
burden.


>
>    On Tuesday, April 21, 2015 7:10 PM, Parth Brahmbhatt
><pb...@hortonworks.com>> wrote:
>
>
> I have added the notes to KIP-11 Open question sections.
>
>Thanks
>Parth
>
>On 4/21/15, 4:49 PM, "Gwen Shapira" <gs...@cloudera.com>> wrote:
>
>>Adding my notes from today's call to the thread:
>>
>>** Deny or Allow all by default? We will add a configuration to
>>control this. The configuration will default to “allow” for backward
>>compatibility. Security admins can set it to "deny"
>>
>>** Storing ACLs for default authorizers: We'll store them in ZK. We'll
>>support pointing the authorizer to any ZK.
>>The use of ZK will be internal to the default authorizer. Authorizer
>>reads ACLs from cache every hour. We proposed having mechanism
>>(possibly via new ZK node) to tell broker to refresh the cache
>>immediately.
>>
>>** Support deny as permission type - we agreed to keep this.
>>
>>** Mapping operations to API: We may need to add Group as a resource,
>>with JoinGroup and OffsetCommit require privilege on the consumer
>>group.
>>This can be something we pass now and authorizers can support in
>>future. - Jay will write specifics to the mailing list discussion.
>>
>>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps <ja...@gmail.com>> wrote:
>>> Following up on the KIP discussion. Two options for authorizing
>>>consumers
>>> to read topic "t" as part of group "g":
>>> 1. READ permission on resource /topic/t
>>> 2. READ permission on resource /topic/t AND WRITE permission on
>>>/group/g
>>>
>>> The advantage of (1) is that it is simpler. The disadvantage is that
>>>any
>>> member of any group that reads from t can commit offsets as any other
>>> member of a different group. This doesn't effect data security (who can
>>> access what) but it is a bit of a management issue--a malicious person
>>>can
>>> cause data loss or duplicates for another consumer by committing
>>>offset.
>>>
>>> I think I favor (2) but it's worth it to think it through.
>>>
>>> -Jay
>>>
>>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
>>> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>> wrote:
>>>
>>>> Hey Jun,
>>>>
>>>> Yes and we support wild cards for all acl entities principal, hosts
>>>>and
>>>> operation.
>>>>
>>>> Thanks
>>>> Parth
>>>>
>>>> On 4/21/15, 9:06 AM, "Jun Rao" <ju...@confluent.io>> wrote:
>>>>
>>>> >Harsha, Parth,
>>>> >
>>>> >Thanks for the clarification. This makes sense. Perhaps we can
>>>>clarify the
>>>> >meaning of those rules in the wiki.
>>>> >
>>>> >Related to this, it seems that we need to support wildcard in
>>>>cli/request
>>>> >protocol for topics?
>>>> >
>>>> >Jun
>>>> >
>>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>>>> >pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>> wrote:
>>>> >
>>>> >> The iptables on unix supports the DENY operator, not that it should
>>>> >> matter. The deny operator can also be used to specify ³allow user1
>>>>to
>>>> >>READ
>>>> >> from topic1 from all hosts but host1,host2². Again we could add a
>>>>host
>>>> >> group semantic and extra complexity around that, not sure if its
>>>>worth
>>>> >>it.
>>>> >> In addition with DENY operator you are now not forced to create a
>>>> >>special
>>>> >> group just to support the authorization use case. I am not
>>>>convinced
>>>> >>that
>>>> >> the operator it self is really all that confusing. There are 3
>>>>practical
>>>> >> use cases:
>>>> >> - Resource with no acl what so ever -> allow access to everyone (
>>>>just
>>>> >>for
>>>> >> backward compatibility, I would much rather fail close and force
>>>>users
>>>> >>to
>>>> >> explicitly grant acls that allows access to all users.)
>>>> >> - Resource with some acl attached -> only users that have a
>>>>matching
>>>> >>allow
>>>> >> acl are allowed (i.e. ³allow READ access to topic1 to user1 from
>>>>all
>>>> >> hosts², only user1 has READ access and no other user has access of
>>>>any
>>>> >> kind)
>>>> >> - Resource with some allow and some deny acl attached -> users are
>>>> >>allowed
>>>> >> to perform operation only when they satisfy allow acl and do not
>>>>have
>>>> >> conflicting deny acl. Users that have no acl(allow or deny) will
>>>>still
>>>> >>not
>>>> >> have any access. (i.e. ³allow READ access to topic1 to user1 from
>>>>all
>>>> >> hosts except host1 and host², only user1 has access but not from
>>>>host1
>>>> >>an
>>>> >> host2)
>>>> >>
>>>> >> I think we need to make a decision on deny primarily because with
>>>> >> introduction of acl management API, Acl is now a public class that
>>>>will
>>>> >>be
>>>> >> used by Ranger/Santry and other authroization providers. In Current
>>>> >>design
>>>> >> the acl has a permissionType enum field with possible values of
>>>>Allow
>>>> >>and
>>>> >> Deny. If we chose to remove deny we can assume all acls to be of
>>>>allow
>>>> >> type and remove the permissionType field completely.
>>>> >>
>>>> >> Thanks
>>>> >> Parth
>>>> >>
>>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com>> wrote:
>>>> >>
>>>> >> >I think thats how its done in pretty much any system I can think
>>>>of.
>>>> >> >
>>>> >>
>>>> >>
>>>>
>>>>
>
>
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Tom Graves <tg...@yahoo.com.INVALID>]

Thanks for the explanations Parth.
On the configs questions, the way I see it is its more likely to accidentally give everyone access, especially since you have to run a separate command to change the acls. If there was some config for defaults, a cluster admin could change that to be nobody or certain set of users, then grant others permissions.  This would also remove the race between commands.  This is something you can always add later though if people request it. 
So in kafka-acl.sh how do I actually tell it what the operation is?kafka-acl.sh --topic testtopic --add --grandprincipal user:joe,user:kate 
where does READ, WRITE, etc go?  Can specify as a list so I don't have to run this a bunch of times for each.
Do you want to have a --host option for --list so that admins could see what acls apply to specific host(s)?
Tom 


     On Wednesday, April 22, 2015 11:38 AM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:
   

 
FYI, I have modified the KIP to include group as resource. In order to
access “joinGroup” and “commitOFfset” APIs the user will need a read
permission on topic and WRITE permission on group.

I plan to open a VOTE thread by noon if there are no more concerns.

Thanks
Parth

On 4/22/15, 9:03 AM, "Tom Graves" <tg...@yahoo.com.INVALID> wrote:

>Hey everyone,
>Sorry to jump in on the conversation so late. I'm new to Kafka. I'll
>apologize in advance if you have already covered some of my questions.  I
>read through the wiki and had some comments and questions.
>1) public enum Operation needs EDIT changed to ALTER

>    Done.

>2) Does the Authorizer class need a setAcls?  Rather then just add to be
>able to set to explicit list and overwrite what was there?  I see the
>kafka-acl.sh lists a removeall so I guess you could do removeall and then
>add.  I also don't see a removeall in the Authorizer class, is it going
>to loop through them all to remove each one?

    There is an overloaded version of removeAcls in the interface that takes
in resource as the only input and as described in the javadoc all the acls
attached to that resource will be deleted. To cover the setAcl use case
the caller can first call remove and then add.

>3) Can someone tell me what the use case to do acls based on the hosts?
>I can see some possibilities just wondering if we can concrete ones where
>one user is allowed from one host but not another.

    I am not sure if I understand the question given the use case you
described in your question is what we are trying to cover with use of
hosts in Acl. There are some additional use cases like “allow access to
any user from host1,host2” but I think primarily it gives the admins the
ability to define acls at a more granular level.

>4) I'm a bit unclear how the "resource" works in the Authorizer class.
>From what I see we have 2 resources - topics and cluster.  If I want to
>add an acl to allow "joe" to CREATE for the cluster then I call addAcls
>with  Acl("user: joe", ALLOW, Set(*), Set(CREATE)) and "cluster"?  What
>if I want to call addAcls for DESCRIBE on a topic?  Is the resource then
>"topic" or is it the topic name?

    We now have 3 resources(added group), please see the updated doc. The
CREATE acl that you described is correct. For any topic operation you
should use topic name as the resource name and for group the user will
provide groupId as resource name.

>5) reassigning partitions is a CLUSTER_ACTION or superuser?  Its not
>totally clear to me the differences between these.  what about increasing
># of partitions?

    I see this as an alter topic operation so it is at topic level and the
user must have alter permissions on topic.    

>6) groups are mentioned, are we supporting right away or is that a follow
>on item? (is there going to be a kafka.supergroups)

    I think it can be a separate jira just for braking down the code review
in smaller chunk. We will support it in first version but I think if we
can not do it for any reason that should not block a release with all the
other authZ work. We made deliberate design choices (like introducing a
principalType in KafkaPrinciapl) to allow supporting groups as an
incremental change.

>7) Are there config options for setting acls when I create my topic?  Or
>do I have to create my topic and then run the kafka-acl.sh script to set
>them?  Although its very small, there would be possible race there that
>someone could start producing to topic before acls are set.

    We discussed this yesterday and we agreed to go with kafka-acl.sh. Yes
there is a very very small window of vulnerability but I think that really
does not warrant to change the decision in this case.

>8) are there configs for cluster level acl defaults?  Or does it default
>to superusers on bringing up new cluster and you have to modify with cli.
>thanks,Tom

    No defaults, the default is superusers will have full access. I don’t
think making assumptions about ones security requirement should be our
burden.

>
>    On Tuesday, April 21, 2015 7:10 PM, Parth Brahmbhatt
><pb...@hortonworks.com> wrote:
>  
>
> I have added the notes to KIP-11 Open question sections.
>
>Thanks
>Parth
>
>On 4/21/15, 4:49 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>
>>Adding my notes from today's call to the thread:
>>
>>** Deny or Allow all by default? We will add a configuration to
>>control this. The configuration will default to “allow” for backward
>>compatibility. Security admins can set it to "deny"
>>
>>** Storing ACLs for default authorizers: We'll store them in ZK. We'll
>>support pointing the authorizer to any ZK.
>>The use of ZK will be internal to the default authorizer. Authorizer
>>reads ACLs from cache every hour. We proposed having mechanism
>>(possibly via new ZK node) to tell broker to refresh the cache
>>immediately.
>>
>>** Support deny as permission type - we agreed to keep this.
>>
>>** Mapping operations to API: We may need to add Group as a resource,
>>with JoinGroup and OffsetCommit require privilege on the consumer
>>group.
>>This can be something we pass now and authorizers can support in
>>future. - Jay will write specifics to the mailing list discussion.
>>
>>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps <ja...@gmail.com> wrote:
>>> Following up on the KIP discussion. Two options for authorizing
>>>consumers
>>> to read topic "t" as part of group "g":
>>> 1. READ permission on resource /topic/t
>>> 2. READ permission on resource /topic/t AND WRITE permission on
>>>/group/g
>>>
>>> The advantage of (1) is that it is simpler. The disadvantage is that
>>>any
>>> member of any group that reads from t can commit offsets as any other
>>> member of a different group. This doesn't effect data security (who can
>>> access what) but it is a bit of a management issue--a malicious person
>>>can
>>> cause data loss or duplicates for another consumer by committing
>>>offset.
>>>
>>> I think I favor (2) but it's worth it to think it through.
>>>
>>> -Jay
>>>
>>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
>>> pbrahmbhatt@hortonworks.com> wrote:
>>>
>>>> Hey Jun,
>>>>
>>>> Yes and we support wild cards for all acl entities principal, hosts
>>>>and
>>>> operation.
>>>>
>>>> Thanks
>>>> Parth
>>>>
>>>> On 4/21/15, 9:06 AM, "Jun Rao" <ju...@confluent.io> wrote:
>>>>
>>>> >Harsha, Parth,
>>>> >
>>>> >Thanks for the clarification. This makes sense. Perhaps we can
>>>>clarify the
>>>> >meaning of those rules in the wiki.
>>>> >
>>>> >Related to this, it seems that we need to support wildcard in
>>>>cli/request
>>>> >protocol for topics?
>>>> >
>>>> >Jun
>>>> >
>>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>>>> >pbrahmbhatt@hortonworks.com> wrote:
>>>> >
>>>> >> The iptables on unix supports the DENY operator, not that it should
>>>> >> matter. The deny operator can also be used to specify ³allow user1
>>>>to
>>>> >>READ
>>>> >> from topic1 from all hosts but host1,host2². Again we could add a
>>>>host
>>>> >> group semantic and extra complexity around that, not sure if its
>>>>worth
>>>> >>it.
>>>> >> In addition with DENY operator you are now not forced to create a
>>>> >>special
>>>> >> group just to support the authorization use case. I am not
>>>>convinced
>>>> >>that
>>>> >> the operator it self is really all that confusing. There are 3
>>>>practical
>>>> >> use cases:
>>>> >> - Resource with no acl what so ever -> allow access to everyone (
>>>>just
>>>> >>for
>>>> >> backward compatibility, I would much rather fail close and force
>>>>users
>>>> >>to
>>>> >> explicitly grant acls that allows access to all users.)
>>>> >> - Resource with some acl attached -> only users that have a
>>>>matching
>>>> >>allow
>>>> >> acl are allowed (i.e. ³allow READ access to topic1 to user1 from
>>>>all
>>>> >> hosts², only user1 has READ access and no other user has access of
>>>>any
>>>> >> kind)
>>>> >> - Resource with some allow and some deny acl attached -> users are
>>>> >>allowed
>>>> >> to perform operation only when they satisfy allow acl and do not
>>>>have
>>>> >> conflicting deny acl. Users that have no acl(allow or deny) will
>>>>still
>>>> >>not
>>>> >> have any access. (i.e. ³allow READ access to topic1 to user1 from
>>>>all
>>>> >> hosts except host1 and host², only user1 has access but not from
>>>>host1
>>>> >>an
>>>> >> host2)
>>>> >>
>>>> >> I think we need to make a decision on deny primarily because with
>>>> >> introduction of acl management API, Acl is now a public class that
>>>>will
>>>> >>be
>>>> >> used by Ranger/Santry and other authroization providers. In Current
>>>> >>design
>>>> >> the acl has a permissionType enum field with possible values of
>>>>Allow
>>>> >>and
>>>> >> Deny. If we chose to remove deny we can assume all acls to be of
>>>>allow
>>>> >> type and remove the permissionType field completely.
>>>> >>
>>>> >> Thanks
>>>> >> Parth
>>>> >>
>>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>>>> >>
>>>> >> >I think thats how its done in pretty much any system I can think
>>>>of.
>>>> >> >
>>>> >>
>>>> >>
>>>>
>>>>
>
>
>
>  



  

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

FYI, I have modified the KIP to include group as resource. In order to
access “joinGroup” and “commitOFfset” APIs the user will need a read
permission on topic and WRITE permission on group.

I plan to open a VOTE thread by noon if there are no more concerns.

Thanks
Parth

On 4/22/15, 9:03 AM, "Tom Graves" <tg...@yahoo.com.INVALID> wrote:

>Hey everyone,
>Sorry to jump in on the conversation so late. I'm new to Kafka. I'll
>apologize in advance if you have already covered some of my questions.  I
>read through the wiki and had some comments and questions.
>1) public enum Operation needs EDIT changed to ALTER

>	Done.

>2) Does the Authorizer class need a setAcls?  Rather then just add to be
>able to set to explicit list and overwrite what was there?  I see the
>kafka-acl.sh lists a removeall so I guess you could do removeall and then
>add.  I also don't see a removeall in the Authorizer class, is it going
>to loop through them all to remove each one?

	There is an overloaded version of removeAcls in the interface that takes
in resource as the only input and as described in the javadoc all the acls
attached to that resource will be deleted. To cover the setAcl use case
the caller can first call remove and then add.

>3) Can someone tell me what the use case to do acls based on the hosts?
>I can see some possibilities just wondering if we can concrete ones where
>one user is allowed from one host but not another.

	I am not sure if I understand the question given the use case you
described in your question is what we are trying to cover with use of
hosts in Acl. There are some additional use cases like “allow access to
any user from host1,host2” but I think primarily it gives the admins the
ability to define acls at a more granular level.

>4) I'm a bit unclear how the "resource" works in the Authorizer class.
>From what I see we have 2 resources - topics and cluster.  If I want to
>add an acl to allow "joe" to CREATE for the cluster then I call addAcls
>with  Acl("user: joe", ALLOW, Set(*), Set(CREATE)) and "cluster"?  What
>if I want to call addAcls for DESCRIBE on a topic?  Is the resource then
>"topic" or is it the topic name?

	We now have 3 resources(added group), please see the updated doc. The
CREATE acl that you described is correct. For any topic operation you
should use topic name as the resource name and for group the user will
provide groupId as resource name.

>5) reassigning partitions is a CLUSTER_ACTION or superuser?  Its not
>totally clear to me the differences between these.  what about increasing
># of partitions?

	I see this as an alter topic operation so it is at topic level and the
user must have alter permissions on topic.	

>6) groups are mentioned, are we supporting right away or is that a follow
>on item? (is there going to be a kafka.supergroups)

	I think it can be a separate jira just for braking down the code review
in smaller chunk. We will support it in first version but I think if we
can not do it for any reason that should not block a release with all the
other authZ work. We made deliberate design choices (like introducing a
principalType in KafkaPrinciapl) to allow supporting groups as an
incremental change.

>7) Are there config options for setting acls when I create my topic?  Or
>do I have to create my topic and then run the kafka-acl.sh script to set
>them?  Although its very small, there would be possible race there that
>someone could start producing to topic before acls are set.

	We discussed this yesterday and we agreed to go with kafka-acl.sh. Yes
there is a very very small window of vulnerability but I think that really
does not warrant to change the decision in this case.

>8) are there configs for cluster level acl defaults?  Or does it default
>to superusers on bringing up new cluster and you have to modify with cli.
>thanks,Tom

	No defaults, the default is superusers will have full access. I don’t
think making assumptions about ones security requirement should be our
burden.

>
>     On Tuesday, April 21, 2015 7:10 PM, Parth Brahmbhatt
><pb...@hortonworks.com> wrote:
>   
>
> I have added the notes to KIP-11 Open question sections.
>
>Thanks
>Parth
>
>On 4/21/15, 4:49 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>
>>Adding my notes from today's call to the thread:
>>
>>** Deny or Allow all by default? We will add a configuration to
>>control this. The configuration will default to “allow” for backward
>>compatibility. Security admins can set it to "deny"
>>
>>** Storing ACLs for default authorizers: We'll store them in ZK. We'll
>>support pointing the authorizer to any ZK.
>>The use of ZK will be internal to the default authorizer. Authorizer
>>reads ACLs from cache every hour. We proposed having mechanism
>>(possibly via new ZK node) to tell broker to refresh the cache
>>immediately.
>>
>>** Support deny as permission type - we agreed to keep this.
>>
>>** Mapping operations to API: We may need to add Group as a resource,
>>with JoinGroup and OffsetCommit require privilege on the consumer
>>group.
>>This can be something we pass now and authorizers can support in
>>future. - Jay will write specifics to the mailing list discussion.
>>
>>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps <ja...@gmail.com> wrote:
>>> Following up on the KIP discussion. Two options for authorizing
>>>consumers
>>> to read topic "t" as part of group "g":
>>> 1. READ permission on resource /topic/t
>>> 2. READ permission on resource /topic/t AND WRITE permission on
>>>/group/g
>>>
>>> The advantage of (1) is that it is simpler. The disadvantage is that
>>>any
>>> member of any group that reads from t can commit offsets as any other
>>> member of a different group. This doesn't effect data security (who can
>>> access what) but it is a bit of a management issue--a malicious person
>>>can
>>> cause data loss or duplicates for another consumer by committing
>>>offset.
>>>
>>> I think I favor (2) but it's worth it to think it through.
>>>
>>> -Jay
>>>
>>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
>>> pbrahmbhatt@hortonworks.com> wrote:
>>>
>>>> Hey Jun,
>>>>
>>>> Yes and we support wild cards for all acl entities principal, hosts
>>>>and
>>>> operation.
>>>>
>>>> Thanks
>>>> Parth
>>>>
>>>> On 4/21/15, 9:06 AM, "Jun Rao" <ju...@confluent.io> wrote:
>>>>
>>>> >Harsha, Parth,
>>>> >
>>>> >Thanks for the clarification. This makes sense. Perhaps we can
>>>>clarify the
>>>> >meaning of those rules in the wiki.
>>>> >
>>>> >Related to this, it seems that we need to support wildcard in
>>>>cli/request
>>>> >protocol for topics?
>>>> >
>>>> >Jun
>>>> >
>>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>>>> >pbrahmbhatt@hortonworks.com> wrote:
>>>> >
>>>> >> The iptables on unix supports the DENY operator, not that it should
>>>> >> matter. The deny operator can also be used to specify ³allow user1
>>>>to
>>>> >>READ
>>>> >> from topic1 from all hosts but host1,host2². Again we could add a
>>>>host
>>>> >> group semantic and extra complexity around that, not sure if its
>>>>worth
>>>> >>it.
>>>> >> In addition with DENY operator you are now not forced to create a
>>>> >>special
>>>> >> group just to support the authorization use case. I am not
>>>>convinced
>>>> >>that
>>>> >> the operator it self is really all that confusing. There are 3
>>>>practical
>>>> >> use cases:
>>>> >> - Resource with no acl what so ever -> allow access to everyone (
>>>>just
>>>> >>for
>>>> >> backward compatibility, I would much rather fail close and force
>>>>users
>>>> >>to
>>>> >> explicitly grant acls that allows access to all users.)
>>>> >> - Resource with some acl attached -> only users that have a
>>>>matching
>>>> >>allow
>>>> >> acl are allowed (i.e. ³allow READ access to topic1 to user1 from
>>>>all
>>>> >> hosts², only user1 has READ access and no other user has access of
>>>>any
>>>> >> kind)
>>>> >> - Resource with some allow and some deny acl attached -> users are
>>>> >>allowed
>>>> >> to perform operation only when they satisfy allow acl and do not
>>>>have
>>>> >> conflicting deny acl. Users that have no acl(allow or deny) will
>>>>still
>>>> >>not
>>>> >> have any access. (i.e. ³allow READ access to topic1 to user1 from
>>>>all
>>>> >> hosts except host1 and host², only user1 has access but not from
>>>>host1
>>>> >>an
>>>> >> host2)
>>>> >>
>>>> >> I think we need to make a decision on deny primarily because with
>>>> >> introduction of acl management API, Acl is now a public class that
>>>>will
>>>> >>be
>>>> >> used by Ranger/Santry and other authroization providers. In Current
>>>> >>design
>>>> >> the acl has a permissionType enum field with possible values of
>>>>Allow
>>>> >>and
>>>> >> Deny. If we chose to remove deny we can assume all acls to be of
>>>>allow
>>>> >> type and remove the permissionType field completely.
>>>> >>
>>>> >> Thanks
>>>> >> Parth
>>>> >>
>>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>>>> >>
>>>> >> >I think thats how its done in pretty much any system I can think
>>>>of.
>>>> >> >
>>>> >>
>>>> >>
>>>>
>>>>
>
>
>
>  

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Tom Graves <tg...@yahoo.com.INVALID>]

Hey everyone,
Sorry to jump in on the conversation so late. I'm new to Kafka. I'll apologize in advance if you have already covered some of my questions.  I read through the wiki and had some comments and questions.
1) public enum Operation needs EDIT changed to ALTER

2) Does the Authorizer class need a setAcls?  Rather then just add to be able to set to explicit list and overwrite what was there?  I see the kafka-acl.sh lists a removeall so I guess you could do removeall and then add.  I also don't see a removeall in the Authorizer class, is it going to loop through them all to remove each one?
3) Can someone tell me what the use case to do acls based on the hosts?  I can see some possibilities just wondering if we can concrete ones where one user is allowed from one host but not another.
4) I'm a bit unclear how the "resource" works in the Authorizer class.  From what I see we have 2 resources - topics and cluster.  If I want to add an acl to allow "joe" to CREATE for the cluster then I call addAcls with  Acl("user: joe", ALLOW, Set(*), Set(CREATE)) and "cluster"?  What if I want to call addAcls for DESCRIBE on a topic?  Is the resource then "topic" or is it the topic name?
5) reassigning partitions is a CLUSTER_ACTION or superuser?  Its not totally clear to me the differences between these.  what about increasing # of partitions?
6) groups are mentioned, are we supporting right away or is that a follow on item? (is there going to be a kafka.supergroups)
7) Are there config options for setting acls when I create my topic?  Or do I have to create my topic and then run the kafka-acl.sh script to set them?  Although its very small, there would be possible race there that someone could start producing to topic before acls are set.
8) are there configs for cluster level acl defaults?  Or does it default to superusers on bringing up new cluster and you have to modify with cli.
thanks,Tom

     On Tuesday, April 21, 2015 7:10 PM, Parth Brahmbhatt <pb...@hortonworks.com> wrote:
   

 I have added the notes to KIP-11 Open question sections.

Thanks
Parth

On 4/21/15, 4:49 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:

>Adding my notes from today's call to the thread:
>
>** Deny or Allow all by default? We will add a configuration to
>control this. The configuration will default to “allow” for backward
>compatibility. Security admins can set it to "deny"
>
>** Storing ACLs for default authorizers: We'll store them in ZK. We'll
>support pointing the authorizer to any ZK.
>The use of ZK will be internal to the default authorizer. Authorizer
>reads ACLs from cache every hour. We proposed having mechanism
>(possibly via new ZK node) to tell broker to refresh the cache
>immediately.
>
>** Support deny as permission type - we agreed to keep this.
>
>** Mapping operations to API: We may need to add Group as a resource,
>with JoinGroup and OffsetCommit require privilege on the consumer
>group.
>This can be something we pass now and authorizers can support in
>future. - Jay will write specifics to the mailing list discussion.
>
>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps <ja...@gmail.com> wrote:
>> Following up on the KIP discussion. Two options for authorizing
>>consumers
>> to read topic "t" as part of group "g":
>> 1. READ permission on resource /topic/t
>> 2. READ permission on resource /topic/t AND WRITE permission on /group/g
>>
>> The advantage of (1) is that it is simpler. The disadvantage is that any
>> member of any group that reads from t can commit offsets as any other
>> member of a different group. This doesn't effect data security (who can
>> access what) but it is a bit of a management issue--a malicious person
>>can
>> cause data loss or duplicates for another consumer by committing offset.
>>
>> I think I favor (2) but it's worth it to think it through.
>>
>> -Jay
>>
>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
>> pbrahmbhatt@hortonworks.com> wrote:
>>
>>> Hey Jun,
>>>
>>> Yes and we support wild cards for all acl entities principal, hosts and
>>> operation.
>>>
>>> Thanks
>>> Parth
>>>
>>> On 4/21/15, 9:06 AM, "Jun Rao" <ju...@confluent.io> wrote:
>>>
>>> >Harsha, Parth,
>>> >
>>> >Thanks for the clarification. This makes sense. Perhaps we can
>>>clarify the
>>> >meaning of those rules in the wiki.
>>> >
>>> >Related to this, it seems that we need to support wildcard in
>>>cli/request
>>> >protocol for topics?
>>> >
>>> >Jun
>>> >
>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>>> >pbrahmbhatt@hortonworks.com> wrote:
>>> >
>>> >> The iptables on unix supports the DENY operator, not that it should
>>> >> matter. The deny operator can also be used to specify ³allow user1
>>>to
>>> >>READ
>>> >> from topic1 from all hosts but host1,host2². Again we could add a
>>>host
>>> >> group semantic and extra complexity around that, not sure if its
>>>worth
>>> >>it.
>>> >> In addition with DENY operator you are now not forced to create a
>>> >>special
>>> >> group just to support the authorization use case. I am not convinced
>>> >>that
>>> >> the operator it self is really all that confusing. There are 3
>>>practical
>>> >> use cases:
>>> >> - Resource with no acl what so ever -> allow access to everyone (
>>>just
>>> >>for
>>> >> backward compatibility, I would much rather fail close and force
>>>users
>>> >>to
>>> >> explicitly grant acls that allows access to all users.)
>>> >> - Resource with some acl attached -> only users that have a matching
>>> >>allow
>>> >> acl are allowed (i.e. ³allow READ access to topic1 to user1 from all
>>> >> hosts², only user1 has READ access and no other user has access of
>>>any
>>> >> kind)
>>> >> - Resource with some allow and some deny acl attached -> users are
>>> >>allowed
>>> >> to perform operation only when they satisfy allow acl and do not
>>>have
>>> >> conflicting deny acl. Users that have no acl(allow or deny) will
>>>still
>>> >>not
>>> >> have any access. (i.e. ³allow READ access to topic1 to user1 from
>>>all
>>> >> hosts except host1 and host², only user1 has access but not from
>>>host1
>>> >>an
>>> >> host2)
>>> >>
>>> >> I think we need to make a decision on deny primarily because with
>>> >> introduction of acl management API, Acl is now a public class that
>>>will
>>> >>be
>>> >> used by Ranger/Santry and other authroization providers. In Current
>>> >>design
>>> >> the acl has a permissionType enum field with possible values of
>>>Allow
>>> >>and
>>> >> Deny. If we chose to remove deny we can assume all acls to be of
>>>allow
>>> >> type and remove the permissionType field completely.
>>> >>
>>> >> Thanks
>>> >> Parth
>>> >>
>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>>> >>
>>> >> >I think thats how its done in pretty much any system I can think
>>>of.
>>> >> >
>>> >>
>>> >>
>>>
>>>



  

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

I have added the notes to KIP-11 Open question sections.

Thanks
Parth

On 4/21/15, 4:49 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:

>Adding my notes from today's call to the thread:
>
>** Deny or Allow all by default? We will add a configuration to
>control this. The configuration will default to “allow” for backward
>compatibility. Security admins can set it to "deny"
>
>** Storing ACLs for default authorizers: We'll store them in ZK. We'll
>support pointing the authorizer to any ZK.
>The use of ZK will be internal to the default authorizer. Authorizer
>reads ACLs from cache every hour. We proposed having mechanism
>(possibly via new ZK node) to tell broker to refresh the cache
>immediately.
>
>** Support deny as permission type - we agreed to keep this.
>
>** Mapping operations to API: We may need to add Group as a resource,
>with JoinGroup and OffsetCommit require privilege on the consumer
>group.
>This can be something we pass now and authorizers can support in
>future. - Jay will write specifics to the mailing list discussion.
>
>On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps <ja...@gmail.com> wrote:
>> Following up on the KIP discussion. Two options for authorizing
>>consumers
>> to read topic "t" as part of group "g":
>> 1. READ permission on resource /topic/t
>> 2. READ permission on resource /topic/t AND WRITE permission on /group/g
>>
>> The advantage of (1) is that it is simpler. The disadvantage is that any
>> member of any group that reads from t can commit offsets as any other
>> member of a different group. This doesn't effect data security (who can
>> access what) but it is a bit of a management issue--a malicious person
>>can
>> cause data loss or duplicates for another consumer by committing offset.
>>
>> I think I favor (2) but it's worth it to think it through.
>>
>> -Jay
>>
>> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
>> pbrahmbhatt@hortonworks.com> wrote:
>>
>>> Hey Jun,
>>>
>>> Yes and we support wild cards for all acl entities principal, hosts and
>>> operation.
>>>
>>> Thanks
>>> Parth
>>>
>>> On 4/21/15, 9:06 AM, "Jun Rao" <ju...@confluent.io> wrote:
>>>
>>> >Harsha, Parth,
>>> >
>>> >Thanks for the clarification. This makes sense. Perhaps we can
>>>clarify the
>>> >meaning of those rules in the wiki.
>>> >
>>> >Related to this, it seems that we need to support wildcard in
>>>cli/request
>>> >protocol for topics?
>>> >
>>> >Jun
>>> >
>>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>>> >pbrahmbhatt@hortonworks.com> wrote:
>>> >
>>> >> The iptables on unix supports the DENY operator, not that it should
>>> >> matter. The deny operator can also be used to specify ³allow user1
>>>to
>>> >>READ
>>> >> from topic1 from all hosts but host1,host2². Again we could add a
>>>host
>>> >> group semantic and extra complexity around that, not sure if its
>>>worth
>>> >>it.
>>> >> In addition with DENY operator you are now not forced to create a
>>> >>special
>>> >> group just to support the authorization use case. I am not convinced
>>> >>that
>>> >> the operator it self is really all that confusing. There are 3
>>>practical
>>> >> use cases:
>>> >> - Resource with no acl what so ever -> allow access to everyone (
>>>just
>>> >>for
>>> >> backward compatibility, I would much rather fail close and force
>>>users
>>> >>to
>>> >> explicitly grant acls that allows access to all users.)
>>> >> - Resource with some acl attached -> only users that have a matching
>>> >>allow
>>> >> acl are allowed (i.e. ³allow READ access to topic1 to user1 from all
>>> >> hosts², only user1 has READ access and no other user has access of
>>>any
>>> >> kind)
>>> >> - Resource with some allow and some deny acl attached -> users are
>>> >>allowed
>>> >> to perform operation only when they satisfy allow acl and do not
>>>have
>>> >> conflicting deny acl. Users that have no acl(allow or deny) will
>>>still
>>> >>not
>>> >> have any access. (i.e. ³allow READ access to topic1 to user1 from
>>>all
>>> >> hosts except host1 and host², only user1 has access but not from
>>>host1
>>> >>an
>>> >> host2)
>>> >>
>>> >> I think we need to make a decision on deny primarily because with
>>> >> introduction of acl management API, Acl is now a public class that
>>>will
>>> >>be
>>> >> used by Ranger/Santry and other authroization providers. In Current
>>> >>design
>>> >> the acl has a permissionType enum field with possible values of
>>>Allow
>>> >>and
>>> >> Deny. If we chose to remove deny we can assume all acls to be of
>>>allow
>>> >> type and remove the permissionType field completely.
>>> >>
>>> >> Thanks
>>> >> Parth
>>> >>
>>> >> On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>>> >>
>>> >> >I think thats how its done in pretty much any system I can think
>>>of.
>>> >> >
>>> >>
>>> >>
>>>
>>>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Gwen Shapira <gs...@cloudera.com>]

Adding my notes from today's call to the thread:

** Deny or Allow all by default? We will add a configuration to
control this. The configuration will default to “allow” for backward
compatibility. Security admins can set it to "deny"

** Storing ACLs for default authorizers: We'll store them in ZK. We'll
support pointing the authorizer to any ZK.
The use of ZK will be internal to the default authorizer. Authorizer
reads ACLs from cache every hour. We proposed having mechanism
(possibly via new ZK node) to tell broker to refresh the cache
immediately.

** Support deny as permission type - we agreed to keep this.

** Mapping operations to API: We may need to add Group as a resource,
with JoinGroup and OffsetCommit require privilege on the consumer
group.
This can be something we pass now and authorizers can support in
future. - Jay will write specifics to the mailing list discussion.

On Tue, Apr 21, 2015 at 4:32 PM, Jay Kreps <ja...@gmail.com> wrote:
> Following up on the KIP discussion. Two options for authorizing consumers
> to read topic "t" as part of group "g":
> 1. READ permission on resource /topic/t
> 2. READ permission on resource /topic/t AND WRITE permission on /group/g
>
> The advantage of (1) is that it is simpler. The disadvantage is that any
> member of any group that reads from t can commit offsets as any other
> member of a different group. This doesn't effect data security (who can
> access what) but it is a bit of a management issue--a malicious person can
> cause data loss or duplicates for another consumer by committing offset.
>
> I think I favor (2) but it's worth it to think it through.
>
> -Jay
>
> On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
> pbrahmbhatt@hortonworks.com> wrote:
>
>> Hey Jun,
>>
>> Yes and we support wild cards for all acl entities principal, hosts and
>> operation.
>>
>> Thanks
>> Parth
>>
>> On 4/21/15, 9:06 AM, "Jun Rao" <ju...@confluent.io> wrote:
>>
>> >Harsha, Parth,
>> >
>> >Thanks for the clarification. This makes sense. Perhaps we can clarify the
>> >meaning of those rules in the wiki.
>> >
>> >Related to this, it seems that we need to support wildcard in cli/request
>> >protocol for topics?
>> >
>> >Jun
>> >
>> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>> >pbrahmbhatt@hortonworks.com> wrote:
>> >
>> >> The iptables on unix supports the DENY operator, not that it should
>> >> matter. The deny operator can also be used to specify ³allow user1 to
>> >>READ
>> >> from topic1 from all hosts but host1,host2². Again we could add a host
>> >> group semantic and extra complexity around that, not sure if its worth
>> >>it.
>> >> In addition with DENY operator you are now not forced to create a
>> >>special
>> >> group just to support the authorization use case. I am not convinced
>> >>that
>> >> the operator it self is really all that confusing. There are 3 practical
>> >> use cases:
>> >> - Resource with no acl what so ever -> allow access to everyone ( just
>> >>for
>> >> backward compatibility, I would much rather fail close and force users
>> >>to
>> >> explicitly grant acls that allows access to all users.)
>> >> - Resource with some acl attached -> only users that have a matching
>> >>allow
>> >> acl are allowed (i.e. ³allow READ access to topic1 to user1 from all
>> >> hosts², only user1 has READ access and no other user has access of any
>> >> kind)
>> >> - Resource with some allow and some deny acl attached -> users are
>> >>allowed
>> >> to perform operation only when they satisfy allow acl and do not have
>> >> conflicting deny acl. Users that have no acl(allow or deny) will still
>> >>not
>> >> have any access. (i.e. ³allow READ access to topic1 to user1 from all
>> >> hosts except host1 and host², only user1 has access but not from host1
>> >>an
>> >> host2)
>> >>
>> >> I think we need to make a decision on deny primarily because with
>> >> introduction of acl management API, Acl is now a public class that will
>> >>be
>> >> used by Ranger/Santry and other authroization providers. In Current
>> >>design
>> >> the acl has a permissionType enum field with possible values of Allow
>> >>and
>> >> Deny. If we chose to remove deny we can assume all acls to be of allow
>> >> type and remove the permissionType field completely.
>> >>
>> >> Thanks
>> >> Parth
>> >>
>> >> On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>> >>
>> >> >I think thats how its done in pretty much any system I can think of.
>> >> >
>> >>
>> >>
>>
>>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Jay Kreps <ja...@gmail.com>]

Following up on the KIP discussion. Two options for authorizing consumers
to read topic "t" as part of group "g":
1. READ permission on resource /topic/t
2. READ permission on resource /topic/t AND WRITE permission on /group/g

The advantage of (1) is that it is simpler. The disadvantage is that any
member of any group that reads from t can commit offsets as any other
member of a different group. This doesn't effect data security (who can
access what) but it is a bit of a management issue--a malicious person can
cause data loss or duplicates for another consumer by committing offset.

I think I favor (2) but it's worth it to think it through.

-Jay

On Tue, Apr 21, 2015 at 2:43 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:

> Hey Jun,
>
> Yes and we support wild cards for all acl entities principal, hosts and
> operation.
>
> Thanks
> Parth
>
> On 4/21/15, 9:06 AM, "Jun Rao" <ju...@confluent.io> wrote:
>
> >Harsha, Parth,
> >
> >Thanks for the clarification. This makes sense. Perhaps we can clarify the
> >meaning of those rules in the wiki.
> >
> >Related to this, it seems that we need to support wildcard in cli/request
> >protocol for topics?
> >
> >Jun
> >
> >On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
> >pbrahmbhatt@hortonworks.com> wrote:
> >
> >> The iptables on unix supports the DENY operator, not that it should
> >> matter. The deny operator can also be used to specify ³allow user1 to
> >>READ
> >> from topic1 from all hosts but host1,host2². Again we could add a host
> >> group semantic and extra complexity around that, not sure if its worth
> >>it.
> >> In addition with DENY operator you are now not forced to create a
> >>special
> >> group just to support the authorization use case. I am not convinced
> >>that
> >> the operator it self is really all that confusing. There are 3 practical
> >> use cases:
> >> - Resource with no acl what so ever -> allow access to everyone ( just
> >>for
> >> backward compatibility, I would much rather fail close and force users
> >>to
> >> explicitly grant acls that allows access to all users.)
> >> - Resource with some acl attached -> only users that have a matching
> >>allow
> >> acl are allowed (i.e. ³allow READ access to topic1 to user1 from all
> >> hosts², only user1 has READ access and no other user has access of any
> >> kind)
> >> - Resource with some allow and some deny acl attached -> users are
> >>allowed
> >> to perform operation only when they satisfy allow acl and do not have
> >> conflicting deny acl. Users that have no acl(allow or deny) will still
> >>not
> >> have any access. (i.e. ³allow READ access to topic1 to user1 from all
> >> hosts except host1 and host², only user1 has access but not from host1
> >>an
> >> host2)
> >>
> >> I think we need to make a decision on deny primarily because with
> >> introduction of acl management API, Acl is now a public class that will
> >>be
> >> used by Ranger/Santry and other authroization providers. In Current
> >>design
> >> the acl has a permissionType enum field with possible values of Allow
> >>and
> >> Deny. If we chose to remove deny we can assume all acls to be of allow
> >> type and remove the permissionType field completely.
> >>
> >> Thanks
> >> Parth
> >>
> >> On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
> >>
> >> >I think thats how its done in pretty much any system I can think of.
> >> >
> >>
> >>
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Hey Jun,

Yes and we support wild cards for all acl entities principal, hosts and
operation.

Thanks
Parth

On 4/21/15, 9:06 AM, "Jun Rao" <ju...@confluent.io> wrote:

>Harsha, Parth,
>
>Thanks for the clarification. This makes sense. Perhaps we can clarify the
>meaning of those rules in the wiki.
>
>Related to this, it seems that we need to support wildcard in cli/request
>protocol for topics?
>
>Jun
>
>On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
>pbrahmbhatt@hortonworks.com> wrote:
>
>> The iptables on unix supports the DENY operator, not that it should
>> matter. The deny operator can also be used to specify ³allow user1 to
>>READ
>> from topic1 from all hosts but host1,host2². Again we could add a host
>> group semantic and extra complexity around that, not sure if its worth
>>it.
>> In addition with DENY operator you are now not forced to create a
>>special
>> group just to support the authorization use case. I am not convinced
>>that
>> the operator it self is really all that confusing. There are 3 practical
>> use cases:
>> - Resource with no acl what so ever -> allow access to everyone ( just
>>for
>> backward compatibility, I would much rather fail close and force users
>>to
>> explicitly grant acls that allows access to all users.)
>> - Resource with some acl attached -> only users that have a matching
>>allow
>> acl are allowed (i.e. ³allow READ access to topic1 to user1 from all
>> hosts², only user1 has READ access and no other user has access of any
>> kind)
>> - Resource with some allow and some deny acl attached -> users are
>>allowed
>> to perform operation only when they satisfy allow acl and do not have
>> conflicting deny acl. Users that have no acl(allow or deny) will still
>>not
>> have any access. (i.e. ³allow READ access to topic1 to user1 from all
>> hosts except host1 and host², only user1 has access but not from host1
>>an
>> host2)
>>
>> I think we need to make a decision on deny primarily because with
>> introduction of acl management API, Acl is now a public class that will
>>be
>> used by Ranger/Santry and other authroization providers. In Current
>>design
>> the acl has a permissionType enum field with possible values of Allow
>>and
>> Deny. If we chose to remove deny we can assume all acls to be of allow
>> type and remove the permissionType field completely.
>>
>> Thanks
>> Parth
>>
>> On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>>
>> >I think thats how its done in pretty much any system I can think of.
>> >
>>
>>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Jun Rao <ju...@confluent.io>]

Harsha, Parth,

Thanks for the clarification. This makes sense. Perhaps we can clarify the
meaning of those rules in the wiki.

Related to this, it seems that we need to support wildcard in cli/request
protocol for topics?

Jun

On Mon, Apr 20, 2015 at 9:07 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:

> The iptables on unix supports the DENY operator, not that it should
> matter. The deny operator can also be used to specify ³allow user1 to READ
> from topic1 from all hosts but host1,host2². Again we could add a host
> group semantic and extra complexity around that, not sure if its worth it.
> In addition with DENY operator you are now not forced to create a special
> group just to support the authorization use case. I am not convinced that
> the operator it self is really all that confusing. There are 3 practical
> use cases:
> - Resource with no acl what so ever -> allow access to everyone ( just for
> backward compatibility, I would much rather fail close and force users to
> explicitly grant acls that allows access to all users.)
> - Resource with some acl attached -> only users that have a matching allow
> acl are allowed (i.e. ³allow READ access to topic1 to user1 from all
> hosts², only user1 has READ access and no other user has access of any
> kind)
> - Resource with some allow and some deny acl attached -> users are allowed
> to perform operation only when they satisfy allow acl and do not have
> conflicting deny acl. Users that have no acl(allow or deny) will still not
> have any access. (i.e. ³allow READ access to topic1 to user1 from all
> hosts except host1 and host², only user1 has access but not from host1 an
> host2)
>
> I think we need to make a decision on deny primarily because with
> introduction of acl management API, Acl is now a public class that will be
> used by Ranger/Santry and other authroization providers. In Current design
> the acl has a permissionType enum field with possible values of Allow and
> Deny. If we chose to remove deny we can assume all acls to be of allow
> type and remove the permissionType field completely.
>
> Thanks
> Parth
>
> On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>
> >I think thats how its done in pretty much any system I can think of.
> >
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Todd Palino <tp...@gmail.com>]

I tend to agree with Parth's point here. Most ACL systems I run into have deny and allow. In general, you have a default policy of allow, then you follow your rules stopping at the first line that matches. If you would like a default deny policy, you have a bunch of allow rules and your last rule is "deny all". This says "everyone listed is allowed. Everyone else is denied." If you instead want a default allow, you have a list of deny rules and the last rule is "allow all". This says "everyone listed is denied. Everyone else is allowed".

I think leaving out a full rule set would be a mistake, as it makes the assumption that you know what all the use cases are. I think all it will really mean is that we will see a KIP before long to fix it.

-Todd

> On Apr 20, 2015, at 7:13 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> 
> Thanks for clarifying the logic.
> 
> I'm +0 on the deny thing.
> IMO, its not really needed, but if you think its important, I don't
> object to having it in.
> 
> Gwen
> 
> On Mon, Apr 20, 2015 at 7:07 PM, Parth Brahmbhatt
> <pb...@hortonworks.com> wrote:
>> The iptables on unix supports the DENY operator, not that it should
>> matter. The deny operator can also be used to specify ³allow user1 to READ
>> from topic1 from all hosts but host1,host2². Again we could add a host
>> group semantic and extra complexity around that, not sure if its worth it.
>> In addition with DENY operator you are now not forced to create a special
>> group just to support the authorization use case. I am not convinced that
>> the operator it self is really all that confusing. There are 3 practical
>> use cases:
>> - Resource with no acl what so ever -> allow access to everyone ( just for
>> backward compatibility, I would much rather fail close and force users to
>> explicitly grant acls that allows access to all users.)
>> - Resource with some acl attached -> only users that have a matching allow
>> acl are allowed (i.e. ³allow READ access to topic1 to user1 from all
>> hosts², only user1 has READ access and no other user has access of any
>> kind)
>> - Resource with some allow and some deny acl attached -> users are allowed
>> to perform operation only when they satisfy allow acl and do not have
>> conflicting deny acl. Users that have no acl(allow or deny) will still not
>> have any access. (i.e. ³allow READ access to topic1 to user1 from all
>> hosts except host1 and host², only user1 has access but not from host1 an
>> host2)
>> 
>> I think we need to make a decision on deny primarily because with
>> introduction of acl management API, Acl is now a public class that will be
>> used by Ranger/Santry and other authroization providers. In Current design
>> the acl has a permissionType enum field with possible values of Allow and
>> Deny. If we chose to remove deny we can assume all acls to be of allow
>> type and remove the permissionType field completely.
>> 
>> Thanks
>> Parth
>> 
>>> On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>>> 
>>> I think thats how its done in pretty much any system I can think of.
>> 

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Gwen Shapira <gs...@cloudera.com>]

Thanks for clarifying the logic.

I'm +0 on the deny thing.
IMO, its not really needed, but if you think its important, I don't
object to having it in.

Gwen

On Mon, Apr 20, 2015 at 7:07 PM, Parth Brahmbhatt
<pb...@hortonworks.com> wrote:
> The iptables on unix supports the DENY operator, not that it should
> matter. The deny operator can also be used to specify ³allow user1 to READ
> from topic1 from all hosts but host1,host2². Again we could add a host
> group semantic and extra complexity around that, not sure if its worth it.
> In addition with DENY operator you are now not forced to create a special
> group just to support the authorization use case. I am not convinced that
> the operator it self is really all that confusing. There are 3 practical
> use cases:
> - Resource with no acl what so ever -> allow access to everyone ( just for
> backward compatibility, I would much rather fail close and force users to
> explicitly grant acls that allows access to all users.)
> - Resource with some acl attached -> only users that have a matching allow
> acl are allowed (i.e. ³allow READ access to topic1 to user1 from all
> hosts², only user1 has READ access and no other user has access of any
> kind)
> - Resource with some allow and some deny acl attached -> users are allowed
> to perform operation only when they satisfy allow acl and do not have
> conflicting deny acl. Users that have no acl(allow or deny) will still not
> have any access. (i.e. ³allow READ access to topic1 to user1 from all
> hosts except host1 and host², only user1 has access but not from host1 an
> host2)
>
> I think we need to make a decision on deny primarily because with
> introduction of acl management API, Acl is now a public class that will be
> used by Ranger/Santry and other authroization providers. In Current design
> the acl has a permissionType enum field with possible values of Allow and
> Deny. If we chose to remove deny we can assume all acls to be of allow
> type and remove the permissionType field completely.
>
> Thanks
> Parth
>
> On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:
>
>>I think thats how its done in pretty much any system I can think of.
>>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

The iptables on unix supports the DENY operator, not that it should
matter. The deny operator can also be used to specify ³allow user1 to READ
from topic1 from all hosts but host1,host2². Again we could add a host
group semantic and extra complexity around that, not sure if its worth it.
In addition with DENY operator you are now not forced to create a special
group just to support the authorization use case. I am not convinced that
the operator it self is really all that confusing. There are 3 practical
use cases:
- Resource with no acl what so ever -> allow access to everyone ( just for
backward compatibility, I would much rather fail close and force users to
explicitly grant acls that allows access to all users.)
- Resource with some acl attached -> only users that have a matching allow
acl are allowed (i.e. ³allow READ access to topic1 to user1 from all
hosts², only user1 has READ access and no other user has access of any
kind)
- Resource with some allow and some deny acl attached -> users are allowed
to perform operation only when they satisfy allow acl and do not have
conflicting deny acl. Users that have no acl(allow or deny) will still not
have any access. (i.e. ³allow READ access to topic1 to user1 from all
hosts except host1 and host², only user1 has access but not from host1 an
host2)

I think we need to make a decision on deny primarily because with
introduction of acl management API, Acl is now a public class that will be
used by Ranger/Santry and other authroization providers. In Current design
the acl has a permissionType enum field with possible values of Allow and
Deny. If we chose to remove deny we can assume all acls to be of allow
type and remove the permissionType field completely.

Thanks
Parth

On 4/20/15, 6:12 PM, "Gwen Shapira" <gs...@cloudera.com> wrote:

>I think thats how its done in pretty much any system I can think of.
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Gwen Shapira <gs...@cloudera.com>]

Wouldn't the "everyone except few users" be easier to manage through
groups / roles?
I think thats how its done in pretty much any system I can think of.
Do you know any system with "deny" option?

I don't feel strongly about this (i.e. its an extra feature, people
can use it or not. maybe it will be useful for IP blacklists).
Just thinking that if it causes confusion, perhaps this is something
we can leave out of scope for now and add later.



On Mon, Apr 20, 2015 at 5:57 PM, Sriharsha Chintalapani <ka...@harsha.io> wrote:
>  According to the pseudo code, if you have a rule "deny user1", then it
>> essentially denies all users?
>
> Gwen,
>
>       I see “deny” being useful. I.e you want to allow everyone except few
> users. This is probably a common case otherwise without deny you have to add
> long list of users who should be in allow list.
>
> Jun,
>
>     As per the pseudo code, It acts per  User for an operation they are
> requesting for .
>
> If user1 connects to Kafka and sends write request to “test-topic" it will
> be granted if the acls for “test-topic” has  "allow user1”  and denies the
> request if the acls for the “test-topic” has “deny user1”.
>
> lets say if the topic acls contains “deny user1” and "allow *”
>
> user2 , user3 (all the other users except user1) can write to topic but only
> user1 be denied.
>
>
> " Suppose there are two rules. Rule1 allows
>
>>> >>user1
>>> >> >and rule2 denies user2. Does user3 have access? If not, does removing
>>> >> >rule1
>>> >> >enable user3 access? "
>
> Rule1: User1, allow
>
> Rule2: User2, deny
>
> In this user3 won’t have access as there is no allow rule for that user so
> user3 will be denied.
>
> Thanks,
>
> Harsha
>
>
>
> --
> Harsha
> Sent with Airmail
>
> On April 20, 2015 at 5:34:43 PM, Gwen Shapira (gshapira@cloudera.com) wrote:
>
> I admit that I'm also not sure what we gain by having "deny" rules.
> A user is either in the "allow" list (one way or another) or it will
> be denied by default.
>
> I think this is something we can skip for now and add later if needed?
>
> On Mon, Apr 20, 2015 at 5:24 PM, Jun Rao <ju...@confluent.io> wrote:
>> According to the pseudo code, if you have a rule "deny user1", then it
>> essentially denies all users?
>>
>> Thanks,
>>
>> Jun
>>
>> On Mon, Apr 20, 2015 at 5:16 PM, Parth Brahmbhatt <
>> pbrahmbhatt@hortonworks.com> wrote:
>>
>>>
>>> Here is a pseudo code that explains my current approach:
>>>
>>> acls = authorizer.getAcl(resource)
>>> if(acls == null || acls.isEmpty) {
>>> allow all requests for backward compatibility. (any topics that
>>> were
>>> created prior to security support will not have acls) This is debatable ,
>>> generally we should block everyone which is what I would prefer but that
>>> means anyone moving to authorizer must go to all of his existing topics
>>> and add acl to allow all. If we are fine with imposing this requirement I
>>> can start returning deny when no acls are found.
>>> } else {
>>> //So the user has set some acls explicitly, this means they have
>>> knowingly enabled authorizer. Let’t first check if they have set an Acl
>>> to
>>> deny this user/host/operation combination.
>>> if some acl denies this request for this principal/host/operation
>>> combination , return deny
>>>
>>> //this principal/host/operation does not have any explicit deny
>>> acl,
>>> check if there is some explicit acl that allows the operation
>>> if at least one acl allows this request for this
>>> principal/host/operation
>>> combination , return allow
>>>
>>> // no acl was found for this principal/host/operation combination
>>> to
>>> allow this operation, so we will deny the request
>>> return deny
>>> }
>>>
>>>
>>> Thanks
>>> Parth
>>>
>>>
>>> On 4/20/15, 2:21 PM, "Jun Rao" <ju...@confluent.io> wrote:
>>>
>>> >Hmm, I thought the semantics is that if you only have rule "deny user2",
>>> >it
>>> >means that everyone except user2 has access?
>>> >
>>> >Thanks,
>>> >
>>> >Jun
>>> >
>>> >On Mon, Apr 20, 2015 at 3:25 PM, Parth Brahmbhatt <
>>> >pbrahmbhatt@hortonworks.com> wrote:
>>> >
>>> >> user3 does not have access and removing the deny rule does not grant
>>> >> him
>>> >> or user2 access. user2 even without the deny rule will not have
>>> >> access.
>>> >>
>>> >> Thanks
>>> >> Parth
>>> >>
>>> >> On 4/20/15, 12:03 PM, "Jun Rao" <ju...@confluent.io> wrote:
>>> >>
>>> >> >Just a followup question. Suppose there are two rules. Rule1 allows
>>> >>user1
>>> >> >and rule2 denies user2. Does user3 have access? If not, does removing
>>> >> >rule1
>>> >> >enable user3 access?
>>> >> >
>>> >> >Thanks,
>>> >> >
>>> >> >Jun
>>> >> >
>>> >> >On Mon, Apr 20, 2015 at 1:34 PM, Parth Brahmbhatt <
>>> >> >pbrahmbhatt@hortonworks.com> wrote:
>>> >> >
>>> >> >>
>>> >> >> Hi Joel,
>>> >> >>
>>> >> >> Thanks for the review and I plan to update the KIP today with all
>>> >> >> the
>>> >> >> updated info. My comments in line below.
>>> >> >>
>>> >> >> Thanks
>>> >> >> Parth
>>> >> >>
>>> >> >>
>>> >> >> On 4/20/15, 10:07 AM, "Joel Koshy" <jjkoshy.w@gmail.com<mailto:
>>> >> >> jjkoshy.w@gmail.com>> wrote:
>>> >> >>
>>> >> >> Hi Parth,
>>> >> >>
>>> >> >> Nice work on this KIP. I did another read through and had a few
>>> >> >> more
>>> >> >> comments (with edits after I went through the thread). Many of
>>> >> >> these
>>> >> >> comments were brought up by others as well, so it appears that the
>>> >>KIP
>>> >> >> would benefit from an update at this point to incorporate comments
>>> >> >> from the thread and last hangout.
>>> >> >>
>>> >> >> - The operation enum is mostly self-explanatory, but it would help
>>> >> >> (for the sake of clarity and completeness if nothing else) to
>>> >> >> document exactly what each of the enums are. E.g., I think this
>>> >>came
>>> >> >> up in our hangout - SEND_CONTROL_MESSAGE is unclear and I don't
>>> >> >> remember what was said about it. <Edit>: After going through the
>>> >> >> thread it seems the conclusion was to categorize operations. E.g.,
>>> >> >> WRITE could apply to multiple requests. Again, this is unclear, so
>>> >> >> if it would be great if you could update the KIP to clarify what
>>> >>you
>>> >> >> intend.
>>> >> >>
>>> >> >> Will add to document. SEND_CONTROL_MESSAGE Probably a very bad name
>>> >>but
>>> >> >> these are intra borker API calls like controller notifying other
>>> >> >>brokers to
>>> >> >> update metadata or heartbeats. Any better naming suggestions?
>>> >> >>
>>> >> >> - When you update the KIP to categorize the requests it would also
>>> >> >> help to have a column for what the resource is for each.
>>> >> >>
>>> >> >> Will add to the KIP.
>>> >> >>
>>> >> >> - FWIW I prefer a 1-1 mapping between requests and operations. I
>>> >>think
>>> >> >> categorizing requests into these can be confusing because:
>>> >> >> - The resource being protected for different requests will be
>>> >> >> different. We are mostly thinking about topics (read/write) but
>>> >> >> there are requests for which topic is not the right resource.
>>> >> >> E.g., for topic creation, the resource as you suggested would be
>>> >> >> something global/common such as “cluster”. For
>>> >> >> OffsetCommit/FetchRequest, the resource may be the consumer
>>> >>group,
>>> >> >> or maybe a tuple of <consumer group, topic>. So this can be
>>> >> >> confusing - i.e., different resources and request types in the
>>> >> >> same category. It may be simpler and clearer to just have a 1-1
>>> >> >> mapping between the operation enum and requests.
>>> >> >>
>>> >> >> I only see 2 resource categories right now cluster and topic. I
>>> >>don’t
>>> >> >> really care one way or another so we can probably make a quick
>>> >>decision
>>> >> >>in
>>> >> >> tomorrow’s meeting to either to 1-1 mapping or have categorization?
>>> >> >>
>>> >> >> - Some requests that are intuitively READ have WRITE side-effects.
>>> >> >> E.g., (currently) TopicMetadataRequest with auto-create, although
>>> >> >> that will eventually go away. ConsumerMetadataRequest still
>>> >> >> auto-creates the offsets topic. Likewise, ADMIN-type requests may
>>> >> >> be interpreted as having side-effects (depending on who you ask).
>>> >> >>
>>> >> >> Yes and what I am doing right now is checking authorization for all
>>> >> >> possible actions i.e. for auto-create it checks if the config has
>>> >> >> it
>>> >> >> enabled and if yes, check for read + create authorization. Its not
>>> >>very
>>> >> >> meaningful right now as there is no CREATE authorization but I
>>> >> >> think
>>> >> >>this
>>> >> >> is implementation detail, we need to ensure we call authorize with
>>> >>all
>>> >> >> possible operations from KafkaAPI.
>>> >> >> - <quote>When an ACL is missing - fail open</quote>. What does
>>> >>missing
>>> >> >> mean? i.e., no explicit ACL for a principal? I'm confused by this
>>> >> >> especially in relation to the precedence of DENY over ALLOW. So per
>>> >> >> the description:
>>> >> >> - If no ACLs exist for topic A then ALLOW all operations on it by
>>> >> >> anyone.
>>> >> >> - If I now add an ACL for a certain principal P to ALLOW (say)
>>> >>WRITE
>>> >> >> to the topic then either:
>>> >> >> - This has the effect of DENYing WRITE to all other principals
>>> >> >> - Or, this ACL serves no purpose
>>> >> >> - If the effect is to DENY WRITE to all other principals, what
>>> >>about
>>> >> >> READ. Do all principals (including P) have READ permissions to
>>> >> >> topic A?
>>> >> >> - In other words, it seems for a specific ACL to be meaningful then
>>> >> >> fail close is necessary for an absent ACL.
>>> >> >> - <edit>After through the thread: it appears that the DENY override
>>> >> >> only applies to the given principal. i.e., in the above case it
>>> >> >> appears that the other principals will in fact be granted access.
>>> >> >> Then this makes the ACL that was added pointless right?
>>> >> >>
>>> >> >> The rule I was going with is
>>> >> >> - If there is no ACL I.e. This might be a topic that was created in
>>> >>non
>>> >> >> secure mode or was created before we supported ACLs. We assume you
>>> >> >> do
>>> >> >>not
>>> >> >> want authorization and let all requests go through.
>>> >> >> - once you add any ACL, we assume you want authorization on the
>>> >> >> topic
>>> >> >>and
>>> >> >> all the general authorization rules now start to apply, I.e we fail
>>> >> >>close
>>> >> >> if we don’t find an ACL that allows access or if we find an ACL
>>> >> >> that
>>> >> >>denies
>>> >> >> access. It does not matter if you added a READACL or WRITEACL or
>>> >> >>ALLOWACL
>>> >> >> or DENY ACL. If you add any ACL, now every user gets checked
>>> >> >> against
>>> >> >>that
>>> >> >> and if it does not satisfy the ACL, request fails. I.e. If you add
>>> >> >> an
>>> >> >>ACL
>>> >> >> “Allow write to topic-1 form user1 from all hosts” , user-1 has
>>> >> >> write
>>> >> >> access from all hosts and no other user has any access(except for
>>> >> >> superusers who have all the access).
>>> >> >> - Deny ACLS are suppose to be used to restrict access authorized by
>>> >>some
>>> >> >> allow ACL, they are not suppose to be required. Implicitly anyone
>>> >> >> who
>>> >> >>does
>>> >> >> not have an allow acl, gets denied. The Deny ACLs are only added to
>>> >>give
>>> >> >> more control to administrators who wants more granular control with
>>> >> >>lesser
>>> >> >> config. The scenario described in mailing list was “Allow user X
>>> >>access
>>> >> >> from all hosts but Host1,Host2”. in absence of DENY operator you
>>> >> >> will
>>> >> >>have
>>> >> >> to exhaustively list all possible hosts in your ACL which is what
>>> >> >> we
>>> >>are
>>> >> >> trying to avoid.
>>> >> >>
>>> >> >> - On ZK ACLs: I think ZK will be closed to everyone except Kafka
>>> >> >> brokers. This is a dependency on KIP-4 though. i.e., eventually all
>>> >> >> clients should talk to brokers only via RPC.
>>> >> >>
>>> >> >> Yes.
>>> >> >>
>>> >> >> - Topic owner: list vs single entry - both have issues off the bat
>>> >> >> (although list is more intuitive at least to me), but perhaps you
>>> >> >> could write up some example workflows to clarify the current
>>> >> >> proposal. I was thinking that anyone in the owner list should be
>>> >> >> considered a super-user of the topic and can grant/revoke
>>> >> >> permissions. They should also be allowed to add other principals as
>>> >> >> owners. Even with this it is unclear who should be allowed to
>>> >>remove
>>> >> >> owners.
>>> >> >>
>>> >> >> As you pointed out in the last KIP meeting owners/creators have use
>>> >>out
>>> >> >> side of security context (plain simple auditing). I don’t think the
>>> >> >> authorizer work depends on this, it was my bad to even mention it
>>> >> >> in
>>> >> >>first
>>> >> >> place. I think we can have this discussion outside of
>>> >> >>authorizer/security
>>> >> >> context and once we have a way to get topic owners the default
>>> >> >>Authorizer
>>> >> >> can start using it. It makes sense to treat all owners as super
>>> >> >> users
>>> >> >>and I
>>> >> >> think it is safe to assume superusers can also modify ownership but
>>> >> >> I
>>> >> >>think
>>> >> >> this should not be treated as blocking work for authorization.
>>> >> >>
>>> >> >> - What is the effect of deleting a topic - should all associated
>>> >> >> ACLs
>>> >> >> be deleted as well?
>>> >> >> They should be and with acls being stored as part of TopicConfig
>>> >> >> this
>>> >> >>was
>>> >> >> taken care of automatically. With the new ACL management API the
>>> >>users
>>> >> >>will
>>> >> >> have to call remove ACLs explicitly to perform the cleanup. If
>>> >>everyone
>>> >> >> thinks this should be automated , with the new APIs we will need a
>>> >> >>hook(or
>>> >> >> poll) to be notified when a topic is deleted to perform cleanup.
>>> >> >> - TopicConfigCache to store topic-ACLs. As mentioned above, not all
>>> >> >> requests will be tied to topics. We may want to have an entirely
>>> >> >> separate ZK directory for ACLs. We have a similar issue with
>>> >>quotas.
>>> >> >> This ties in with dynamic config management. We can certainly
>>> >> >> leverage the dynamic config management part of topic configs but I
>>> >> >> think we need to have a story for non-topic resources.
>>> >> >>
>>> >> >> In the first proposal I was going with a topic-Acl and cluster-Acl
>>> >>where
>>> >> >> cluster-Acls were json acl local files on all brokers. With the new
>>> >>ACL
>>> >> >> management APIs we are planning to have /kafka-acl node under which
>>> >>all
>>> >> >> acls will be stored in /kakfa-acls/resource-name -> {acl json
>>> >> >> data}.
>>> >> >> Cluster acls will just have resource name kafka-cluster.
>>> >> >>
>>> >> >> Thanks,
>>> >> >>
>>> >> >> Joel
>>> >> >>
>>> >> >> On Thu, Apr 16, 2015 at 12:15:37AM +0000, Parth Brahmbhatt wrote:
>>> >> >> Kafka currently stores logConfig overrides specified during topic
>>> >> >>creation
>>> >> >> in zookeeper, its just an instance of java.util.Properties
>>> >> >> converted
>>> >>to
>>> >> >> json. I am proposing in addition to that we store acls and owner as
>>> >>well
>>> >> >> as part of same Properties map.
>>> >> >> There is some infrastructure around reading this config, converting
>>> >>it
>>> >> >> back to Properties map and most importantly propagating any changes
>>> >> >> efficiently which we will be able to leverage. As this
>>> >>infrastructure is
>>> >> >> common to the cluster the reading (not interpreting) of config
>>> >>happens
>>> >> >> outside of any authorization code.
>>> >> >> If the TopicConfigCache just kept the json representation and left
>>> >>it to
>>> >> >> authorizer to parse it, the authorizer will have to either parse
>>> >> >> the
>>> >> >>json
>>> >> >> for each request(not acceptable) or it will have to keep one more
>>> >>layer
>>> >> >>of
>>> >> >> parsed ACL instance cache. Assuming authorizer will keep an
>>> >>additional
>>> >> >> caching layer we will now have to implement some way to invalidate
>>> >>the
>>> >> >> cache which means the TopicConfigCache will have to be an
>>> >> >> observable
>>> >> >>which
>>> >> >> the Authorizer observes and invalidates its cache entries when
>>> >> >> topicConfigCache gets updated. Seemed like unnecessary complexity
>>> >>with
>>> >> >>not
>>> >> >> lot to gain so I went with TopicConfigCache interpreting the json
>>> >> >> and
>>> >> >> caching a higher level modeled object.
>>> >> >> In summary, the interpretation is done for both optimization and
>>> >> >> simplicity. If you think it is important to allow custom ACL format
>>> >> >> support we can add one more pluggable config(acl.parser) and
>>> >> >> interface(AclParser) or it could just be another method in
>>> >>Authorizer.
>>> >> >> One thing to note the current ACL json is versioned so it is easy
>>> >> >> to
>>> >> >>make
>>> >> >> changes to it however it won’t be possible to support custom ACL
>>> >>formats
>>> >> >> with the current design.
>>> >> >> Thanks
>>> >> >> Parth
>>> >> >> On 4/15/15, 4:29 PM, "Michael Herstine"
>>> >><mherstine@linkedin.com.INVALID
>>> >> >> <ma...@linkedin.com.INVALID>>
>>> >> >> wrote:
>>> >> >> >Hi Parth,
>>> >> >> >
>>> >> >> >I’m a little confused: why would Kafka need to interpret the JSON?
>>> >> >>IIRC
>>> >> >> >KIP-11 even says that the TopicConfigData will just store the
>>> >> >> > JSON.
>>> >>I’m
>>> >> >> >not really making a design recommendation here, just trying to
>>> >> >>understand
>>> >> >> >what you’re proposing.
>>> >> >> >
>>> >> >> >On 4/15/15, 11:20 AM, "Parth Brahmbhatt"
>>> >><pbrahmbhatt@hortonworks.com
>>> >> >> <ma...@hortonworks.com>>
>>> >> >> >wrote:
>>> >> >> >
>>> >> >> >>Hi Michael,
>>> >> >> >>
>>> >> >> >>There is code in kafka codebase that reads and interprets the
>>> >> >> >> topic
>>> >> >> >>config JSON which has acls, owner and logconfig properties. There
>>> >>are
>>> >> >>3
>>> >> >> >>use cases that we are supporting with current proposal:
>>> >> >> >>
>>> >> >> >> * You use out of box simpleAcl authorizer which is tied to the
>>> >>acl
>>> >> >> >>stored in topic config and the format is locked down.
>>> >> >> >> * You have a custom authorizer and a custom ACL store.
>>> >> >>Ranger/Argus
>>> >> >> >>falls under this as they have their own acl store and ui that
>>> >> >> >> users
>>> >> >>use
>>> >> >> >>to configure acls on the cluster and cluster resources like
>>> >> >> >> topic.
>>> >> >>It is
>>> >> >> >>upto the custom authorizer to leverage the kafka acl configs or
>>> >> >> >>completely ignore them as they have set a user expectation that
>>> >>only
>>> >> >>acls
>>> >> >> >>configured via their ui/system will be effective.
>>> >> >> >> * You have a custom authorizer but no custom Acl store. You are
>>> >> >> >>completely tied to Acl structure that we have provided in out of
>>> >>box
>>> >> >> >>implementation.
>>> >> >> >>
>>> >> >> >>Thanks
>>> >> >> >>Parth
>>> >> >> >>
>>> >> >> >>On 4/15/15, 10:31 AM, "Michael Herstine"
>>> >> >>
>>> >>>><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
>>> >> >> ><ma...@linkedin.com.INVALID>>
>>> >> >> >>wrote:
>>> >> >> >>
>>> >> >> >>Hi Parth,
>>> >> >> >>
>>> >> >> >>One question that occurred to me at the end of today’s hangout:
>>> >> >> >> how
>>> >> >>tied
>>> >> >> >>are we to a particular ACL representation under your proposal? I
>>> >>know
>>> >> >> >>that
>>> >> >> >>TopicConfigCache will just contain JSON— if a particular site
>>> >>decides
>>> >> >> >>they
>>> >> >> >>want to represent their ACLs differently, and swap out the
>>> >>authorizer
>>> >> >> >>implementation, will that work? I guess what I’m asking is
>>> >> >> >> whether
>>> >> >> >>there’s any code in the Kafka codebase that will interpret that
>>> >>JSON,
>>> >> >>or
>>> >> >> >>does that logic live exclusively in the authorizer?
>>> >> >> >>
>>> >> >> >>On 4/14/15, 10:56 PM, "Don Bosco Durai"
>>> >> >>
>>> >>>><bo...@apache.org>>
>>> >> >> wrote:
>>> >> >> >>
>>> >> >> >>I also feel, having just IP would be more appropriate. Host
>>> >> >> >> lookup
>>> >> >>will
>>> >> >> >>unnecessary slow things down and would be insecure as you pointed
>>> >>out.
>>> >> >> >>
>>> >> >> >>With IP, it will be also able to setup policies (in future if
>>> >>needed)
>>> >> >> >>with
>>> >> >> >>ranges or netmasks and it would be more scalable.
>>> >> >> >>
>>> >> >> >>Bosco
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>On 4/14/15, 1:40 PM, "Michael Herstine"
>>> >> >>
>>> >>>><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
>>> >> >> ><ma...@linkedin.com.INVALID>>
>>> >> >> >>wrote:
>>> >> >> >>
>>> >> >> >>Hi Parth,
>>> >> >> >>
>>> >> >> >>Sorry to chime in so late, but I’ve got a minor question on the
>>> >>KIP.
>>> >> >> >>
>>> >> >> >>Several methods take a parameter named “host” of type String. Is
>>> >>that
>>> >> >> >>intended to be a hostname, or an IP address? If the former, I’m
>>> >> >>curious
>>> >> >> >>as
>>> >> >> >>to how that’s found (in my experience, when accepting an incoming
>>> >> >>socket
>>> >> >> >>connection, you only know the IP address, and there isn’t a way
>>> >> >> >> to
>>> >>map
>>> >> >> >>that to a hostname without a round trip to a DNS server, which is
>>> >> >> >>insecure
>>> >> >> >>anyway).
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
>>> >> >>
>>> >> >>>><pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
>>> >> ><mailto
>>> >> >>>>:
>>> >> >> pbrahmbhatt@hortonworks.com>>
>>> >> >> >>wrote:
>>> >> >> >>
>>> >> >> >>Hi all,
>>> >> >> >>
>>> >> >> >>I have modified the KIP to reflect the recent change request from
>>> >>the
>>> >> >> >>reviewers. I have been working on the code and I have the server
>>> >>side
>>> >> >> >>code
>>> >> >> >>for authorization ready. I am now modifying the command line
>>> >> >>utilities.
>>> >> >> >>I
>>> >> >> >>would really appreciate if some of the committers can spend
>>> >>sometime
>>> >> >>to
>>> >> >> >>review the KIP so we can make progress on this.
>>> >> >> >>
>>> >> >> >>Thanks
>>> >> >> >>Parth
>>> >> >> >>
>>> >> >> >>On 3/18/15, 2:20 PM, "Michael Herstine"
>>> >> >>
>>> >>>><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
>>> >> >> ><ma...@linkedin.com.INVALID>>
>>> >> >> >>wrote:
>>> >> >> >>
>>> >> >> >>Hi Parth,
>>> >> >> >>
>>> >> >> >>Thanks! A few questions:
>>> >> >> >>
>>> >> >> >>1. Do you want to permit rules in your ACLs that DENY access as
>>> >>well
>>> >> >>as
>>> >> >> >>ALLOW? This can be handy setting up rules that have exceptions.
>>> >>E.g.
>>> >> >> >>“Allow principal P to READ resource R from all hosts” with “Deny
>>> >> >> >>principal
>>> >> >> >>P READ access to resource R from host H1” in combination would
>>> >>allow P
>>> >> >> >>to
>>> >> >> >>READ R from all hosts *except* H1.
>>> >> >> >>
>>> >> >> >>2. When a topic is newly created, will there be an ACL created
>>> >> >> >> for
>>> >>it?
>>> >> >> >>If
>>> >> >> >>not, would that not deny subsequent access to it?
>>> >> >> >>
>>> >> >> >>(nit) Maybe use Principal instead of String to represent
>>> >>principals?
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>On 3/9/15, 11:48 AM, "Don Bosco Durai"
>>> >> >>
>>> >>>><bo...@apache.org>>
>>> >> >> wrote:
>>> >> >> >>
>>> >> >> >>Parth
>>> >> >> >>
>>> >> >> >>Overall it is looking good. Couple of questionsŠ
>>> >> >> >>
>>> >> >> >>- Can you give an example how the policies will look like in the
>>> >> >> >>default
>>> >> >> >>implementation?
>>> >> >> >>- In the operations, can we support ³CONNECT² also? This can be
>>> >>used
>>> >> >> >>during Session connection
>>> >> >> >>- Regarding access control for ³Topic Creation², since we can¹t
>>> >> >> >> do
>>> >>it
>>> >> >> >>on
>>> >> >> >>the server side, can we de-scope it for? And plan it as a future
>>> >> >> >>feature
>>> >> >> >>request?
>>> >> >> >>
>>> >> >> >>Thanks
>>> >> >> >>
>>> >> >> >>Bosco
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>On 3/6/15, 8:10 AM, "Harsha"
>>> >><kafka@harsha.io<mailto:kafka@harsha.io
>>> >> >> ><ma...@harsha.io>>
>>> >> >> >>wrote:
>>> >> >> >>
>>> >> >> >>Hi Parth,
>>> >> >> >> Thanks for putting this together. Overall it looks good
>>> >> >> >>to
>>> >> >> >> me. Although AdminUtils is a concern KIP-4 can
>>> >>probably
>>> >> >> >>fix
>>> >> >> >> that part.
>>> >> >> >>Thanks,
>>> >> >> >>Harsha
>>> >> >> >>
>>> >> >> >>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>> >> >> >>Forgot to add links to wiki and jira.
>>> >> >> >>Link to wiki:
>>> >> >>
>>> >>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
>>> >> >> >>t
>>> >> >> >>i
>>> >> >> >>o
>>> >> >> >>n
>>> >> >> >>+
>>> >> >> >>Interface
>>> >> >> >>Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>> >> >> >>Thanks
>>> >> >> >>Parth
>>> >> >> >>From: Parth Brahmbhatt
>>> >> >>
>>> >> >>>><pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
>>> >> ><mailto
>>> >> >>>>:
>>> >> >> pbrahmbhatt@hortonworks.com><mailto:p
>>> >> >> >>b
>>> >> >> >>rahmbhatt@hortonworks.com<ma...@hortonworks.com>>>
>>> >> >> >>Date: Thursday, March 5, 2015 at 10:33 AM
>>> >> >> >>To:
>>> >> >> >>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:
>>> >> >> dev@kafka.apache.org><mailto:dev@kafka.apach
>>> >> >> >>e
>>> >> >> >>.org>"
>>> >> >> >><de...@kafka.apache.org><mailto:
>>> >> >> dev@kafka.apache.org><mailto:dev@kafka.apach
>>> >> >> >>e
>>> >> >> >>.org>>
>>> >> >> >>Subject: [DISCUSS] KIP-11- Authorization design for kafka
>>> >> >> >> security
>>> >> >> >>Hi,
>>> >> >> >>KIP-11 is open for discussion , I have updated the wiki with the
>>> >> >> >>design
>>> >> >> >>and open questions.
>>> >> >> >>Thanks
>>> >> >> >>Parth
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >>
>>> >> >> >
>>> >> >>
>>> >> >>
>>> >> >>
>>> >>
>>> >>
>>>
>>>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Sriharsha Chintalapani <ka...@harsha.io>]

 According to the pseudo code, if you have a rule "deny user1", then it 
> essentially denies all users? 
Gwen,

      I see “deny” being useful. I.e you want to allow everyone except few users. This is probably a common case otherwise without deny you have to add long list of users who should be in allow list.

Jun,

    As per the pseudo code, It acts per  User for an operation they are requesting for . 

If user1 connects to Kafka and sends write request to “test-topic" it will be granted if the acls for “test-topic” has  "allow user1”  and denies the request if the acls for the “test-topic” has “deny user1”.

lets say if the topic acls contains “deny user1” and "allow *”  

user2 , user3 (all the other users except user1) can write to topic but only user1 be denied.



" Suppose there are two rules. Rule1 allows 

>> >>user1 
>> >> >and rule2 denies user2. Does user3 have access? If not, does removing 
>> >> >rule1 
>> >> >enable user3 access? "
Rule1: User1, allow

Rule2: User2, deny

In this user3 won’t have access as there is no allow rule for that user so user3 will be denied.

Thanks,

Harsha





-- 
Harsha
Sent with Airmail

On April 20, 2015 at 5:34:43 PM, Gwen Shapira (gshapira@cloudera.com) wrote:

I admit that I'm also not sure what we gain by having "deny" rules.  
A user is either in the "allow" list (one way or another) or it will  
be denied by default.  

I think this is something we can skip for now and add later if needed?  

On Mon, Apr 20, 2015 at 5:24 PM, Jun Rao <ju...@confluent.io> wrote:  
> According to the pseudo code, if you have a rule "deny user1", then it  
> essentially denies all users?  
>  
> Thanks,  
>  
> Jun  
>  
> On Mon, Apr 20, 2015 at 5:16 PM, Parth Brahmbhatt <  
> pbrahmbhatt@hortonworks.com> wrote:  
>  
>>  
>> Here is a pseudo code that explains my current approach:  
>>  
>> acls = authorizer.getAcl(resource)  
>> if(acls == null || acls.isEmpty) {  
>> allow all requests for backward compatibility. (any topics that  
>> were  
>> created prior to security support will not have acls) This is debatable ,  
>> generally we should block everyone which is what I would prefer but that  
>> means anyone moving to authorizer must go to all of his existing topics  
>> and add acl to allow all. If we are fine with imposing this requirement I  
>> can start returning deny when no acls are found.  
>> } else {  
>> //So the user has set some acls explicitly, this means they have  
>> knowingly enabled authorizer. Let’t first check if they have set an Acl to  
>> deny this user/host/operation combination.  
>> if some acl denies this request for this principal/host/operation  
>> combination , return deny  
>>  
>> //this principal/host/operation does not have any explicit deny  
>> acl,  
>> check if there is some explicit acl that allows the operation  
>> if at least one acl allows this request for this  
>> principal/host/operation  
>> combination , return allow  
>>  
>> // no acl was found for this principal/host/operation combination  
>> to  
>> allow this operation, so we will deny the request  
>> return deny  
>> }  
>>  
>>  
>> Thanks  
>> Parth  
>>  
>>  
>> On 4/20/15, 2:21 PM, "Jun Rao" <ju...@confluent.io> wrote:  
>>  
>> >Hmm, I thought the semantics is that if you only have rule "deny user2",  
>> >it  
>> >means that everyone except user2 has access?  
>> >  
>> >Thanks,  
>> >  
>> >Jun  
>> >  
>> >On Mon, Apr 20, 2015 at 3:25 PM, Parth Brahmbhatt <  
>> >pbrahmbhatt@hortonworks.com> wrote:  
>> >  
>> >> user3 does not have access and removing the deny rule does not grant him  
>> >> or user2 access. user2 even without the deny rule will not have access.  
>> >>  
>> >> Thanks  
>> >> Parth  
>> >>  
>> >> On 4/20/15, 12:03 PM, "Jun Rao" <ju...@confluent.io> wrote:  
>> >>  
>> >> >Just a followup question. Suppose there are two rules. Rule1 allows  
>> >>user1  
>> >> >and rule2 denies user2. Does user3 have access? If not, does removing  
>> >> >rule1  
>> >> >enable user3 access?  
>> >> >  
>> >> >Thanks,  
>> >> >  
>> >> >Jun  
>> >> >  
>> >> >On Mon, Apr 20, 2015 at 1:34 PM, Parth Brahmbhatt <  
>> >> >pbrahmbhatt@hortonworks.com> wrote:  
>> >> >  
>> >> >>  
>> >> >> Hi Joel,  
>> >> >>  
>> >> >> Thanks for the review and I plan to update the KIP today with all the  
>> >> >> updated info. My comments in line below.  
>> >> >>  
>> >> >> Thanks  
>> >> >> Parth  
>> >> >>  
>> >> >>  
>> >> >> On 4/20/15, 10:07 AM, "Joel Koshy" <jjkoshy.w@gmail.com<mailto:  
>> >> >> jjkoshy.w@gmail.com>> wrote:  
>> >> >>  
>> >> >> Hi Parth,  
>> >> >>  
>> >> >> Nice work on this KIP. I did another read through and had a few more  
>> >> >> comments (with edits after I went through the thread). Many of these  
>> >> >> comments were brought up by others as well, so it appears that the  
>> >>KIP  
>> >> >> would benefit from an update at this point to incorporate comments  
>> >> >> from the thread and last hangout.  
>> >> >>  
>> >> >> - The operation enum is mostly self-explanatory, but it would help  
>> >> >> (for the sake of clarity and completeness if nothing else) to  
>> >> >> document exactly what each of the enums are. E.g., I think this  
>> >>came  
>> >> >> up in our hangout - SEND_CONTROL_MESSAGE is unclear and I don't  
>> >> >> remember what was said about it. <Edit>: After going through the  
>> >> >> thread it seems the conclusion was to categorize operations. E.g.,  
>> >> >> WRITE could apply to multiple requests. Again, this is unclear, so  
>> >> >> if it would be great if you could update the KIP to clarify what  
>> >>you  
>> >> >> intend.  
>> >> >>  
>> >> >> Will add to document. SEND_CONTROL_MESSAGE Probably a very bad name  
>> >>but  
>> >> >> these are intra borker API calls like controller notifying other  
>> >> >>brokers to  
>> >> >> update metadata or heartbeats. Any better naming suggestions?  
>> >> >>  
>> >> >> - When you update the KIP to categorize the requests it would also  
>> >> >> help to have a column for what the resource is for each.  
>> >> >>  
>> >> >> Will add to the KIP.  
>> >> >>  
>> >> >> - FWIW I prefer a 1-1 mapping between requests and operations. I  
>> >>think  
>> >> >> categorizing requests into these can be confusing because:  
>> >> >> - The resource being protected for different requests will be  
>> >> >> different. We are mostly thinking about topics (read/write) but  
>> >> >> there are requests for which topic is not the right resource.  
>> >> >> E.g., for topic creation, the resource as you suggested would be  
>> >> >> something global/common such as “cluster”. For  
>> >> >> OffsetCommit/FetchRequest, the resource may be the consumer  
>> >>group,  
>> >> >> or maybe a tuple of <consumer group, topic>. So this can be  
>> >> >> confusing - i.e., different resources and request types in the  
>> >> >> same category. It may be simpler and clearer to just have a 1-1  
>> >> >> mapping between the operation enum and requests.  
>> >> >>  
>> >> >> I only see 2 resource categories right now cluster and topic. I  
>> >>don’t  
>> >> >> really care one way or another so we can probably make a quick  
>> >>decision  
>> >> >>in  
>> >> >> tomorrow’s meeting to either to 1-1 mapping or have categorization?  
>> >> >>  
>> >> >> - Some requests that are intuitively READ have WRITE side-effects.  
>> >> >> E.g., (currently) TopicMetadataRequest with auto-create, although  
>> >> >> that will eventually go away. ConsumerMetadataRequest still  
>> >> >> auto-creates the offsets topic. Likewise, ADMIN-type requests may  
>> >> >> be interpreted as having side-effects (depending on who you ask).  
>> >> >>  
>> >> >> Yes and what I am doing right now is checking authorization for all  
>> >> >> possible actions i.e. for auto-create it checks if the config has it  
>> >> >> enabled and if yes, check for read + create authorization. Its not  
>> >>very  
>> >> >> meaningful right now as there is no CREATE authorization but I think  
>> >> >>this  
>> >> >> is implementation detail, we need to ensure we call authorize with  
>> >>all  
>> >> >> possible operations from KafkaAPI.  
>> >> >> - <quote>When an ACL is missing - fail open</quote>. What does  
>> >>missing  
>> >> >> mean? i.e., no explicit ACL for a principal? I'm confused by this  
>> >> >> especially in relation to the precedence of DENY over ALLOW. So per  
>> >> >> the description:  
>> >> >> - If no ACLs exist for topic A then ALLOW all operations on it by  
>> >> >> anyone.  
>> >> >> - If I now add an ACL for a certain principal P to ALLOW (say)  
>> >>WRITE  
>> >> >> to the topic then either:  
>> >> >> - This has the effect of DENYing WRITE to all other principals  
>> >> >> - Or, this ACL serves no purpose  
>> >> >> - If the effect is to DENY WRITE to all other principals, what  
>> >>about  
>> >> >> READ. Do all principals (including P) have READ permissions to  
>> >> >> topic A?  
>> >> >> - In other words, it seems for a specific ACL to be meaningful then  
>> >> >> fail close is necessary for an absent ACL.  
>> >> >> - <edit>After through the thread: it appears that the DENY override  
>> >> >> only applies to the given principal. i.e., in the above case it  
>> >> >> appears that the other principals will in fact be granted access.  
>> >> >> Then this makes the ACL that was added pointless right?  
>> >> >>  
>> >> >> The rule I was going with is  
>> >> >> - If there is no ACL I.e. This might be a topic that was created in  
>> >>non  
>> >> >> secure mode or was created before we supported ACLs. We assume you do  
>> >> >>not  
>> >> >> want authorization and let all requests go through.  
>> >> >> - once you add any ACL, we assume you want authorization on the topic  
>> >> >>and  
>> >> >> all the general authorization rules now start to apply, I.e we fail  
>> >> >>close  
>> >> >> if we don’t find an ACL that allows access or if we find an ACL that  
>> >> >>denies  
>> >> >> access. It does not matter if you added a READACL or WRITEACL or  
>> >> >>ALLOWACL  
>> >> >> or DENY ACL. If you add any ACL, now every user gets checked against  
>> >> >>that  
>> >> >> and if it does not satisfy the ACL, request fails. I.e. If you add an  
>> >> >>ACL  
>> >> >> “Allow write to topic-1 form user1 from all hosts” , user-1 has write  
>> >> >> access from all hosts and no other user has any access(except for  
>> >> >> superusers who have all the access).  
>> >> >> - Deny ACLS are suppose to be used to restrict access authorized by  
>> >>some  
>> >> >> allow ACL, they are not suppose to be required. Implicitly anyone who  
>> >> >>does  
>> >> >> not have an allow acl, gets denied. The Deny ACLs are only added to  
>> >>give  
>> >> >> more control to administrators who wants more granular control with  
>> >> >>lesser  
>> >> >> config. The scenario described in mailing list was “Allow user X  
>> >>access  
>> >> >> from all hosts but Host1,Host2”. in absence of DENY operator you will  
>> >> >>have  
>> >> >> to exhaustively list all possible hosts in your ACL which is what we  
>> >>are  
>> >> >> trying to avoid.  
>> >> >>  
>> >> >> - On ZK ACLs: I think ZK will be closed to everyone except Kafka  
>> >> >> brokers. This is a dependency on KIP-4 though. i.e., eventually all  
>> >> >> clients should talk to brokers only via RPC.  
>> >> >>  
>> >> >> Yes.  
>> >> >>  
>> >> >> - Topic owner: list vs single entry - both have issues off the bat  
>> >> >> (although list is more intuitive at least to me), but perhaps you  
>> >> >> could write up some example workflows to clarify the current  
>> >> >> proposal. I was thinking that anyone in the owner list should be  
>> >> >> considered a super-user of the topic and can grant/revoke  
>> >> >> permissions. They should also be allowed to add other principals as  
>> >> >> owners. Even with this it is unclear who should be allowed to  
>> >>remove  
>> >> >> owners.  
>> >> >>  
>> >> >> As you pointed out in the last KIP meeting owners/creators have use  
>> >>out  
>> >> >> side of security context (plain simple auditing). I don’t think the  
>> >> >> authorizer work depends on this, it was my bad to even mention it in  
>> >> >>first  
>> >> >> place. I think we can have this discussion outside of  
>> >> >>authorizer/security  
>> >> >> context and once we have a way to get topic owners the default  
>> >> >>Authorizer  
>> >> >> can start using it. It makes sense to treat all owners as super users  
>> >> >>and I  
>> >> >> think it is safe to assume superusers can also modify ownership but I  
>> >> >>think  
>> >> >> this should not be treated as blocking work for authorization.  
>> >> >>  
>> >> >> - What is the effect of deleting a topic - should all associated ACLs  
>> >> >> be deleted as well?  
>> >> >> They should be and with acls being stored as part of TopicConfig this  
>> >> >>was  
>> >> >> taken care of automatically. With the new ACL management API the  
>> >>users  
>> >> >>will  
>> >> >> have to call remove ACLs explicitly to perform the cleanup. If  
>> >>everyone  
>> >> >> thinks this should be automated , with the new APIs we will need a  
>> >> >>hook(or  
>> >> >> poll) to be notified when a topic is deleted to perform cleanup.  
>> >> >> - TopicConfigCache to store topic-ACLs. As mentioned above, not all  
>> >> >> requests will be tied to topics. We may want to have an entirely  
>> >> >> separate ZK directory for ACLs. We have a similar issue with  
>> >>quotas.  
>> >> >> This ties in with dynamic config management. We can certainly  
>> >> >> leverage the dynamic config management part of topic configs but I  
>> >> >> think we need to have a story for non-topic resources.  
>> >> >>  
>> >> >> In the first proposal I was going with a topic-Acl and cluster-Acl  
>> >>where  
>> >> >> cluster-Acls were json acl local files on all brokers. With the new  
>> >>ACL  
>> >> >> management APIs we are planning to have /kafka-acl node under which  
>> >>all  
>> >> >> acls will be stored in /kakfa-acls/resource-name -> {acl json data}.  
>> >> >> Cluster acls will just have resource name kafka-cluster.  
>> >> >>  
>> >> >> Thanks,  
>> >> >>  
>> >> >> Joel  
>> >> >>  
>> >> >> On Thu, Apr 16, 2015 at 12:15:37AM +0000, Parth Brahmbhatt wrote:  
>> >> >> Kafka currently stores logConfig overrides specified during topic  
>> >> >>creation  
>> >> >> in zookeeper, its just an instance of java.util.Properties converted  
>> >>to  
>> >> >> json. I am proposing in addition to that we store acls and owner as  
>> >>well  
>> >> >> as part of same Properties map.  
>> >> >> There is some infrastructure around reading this config, converting  
>> >>it  
>> >> >> back to Properties map and most importantly propagating any changes  
>> >> >> efficiently which we will be able to leverage. As this  
>> >>infrastructure is  
>> >> >> common to the cluster the reading (not interpreting) of config  
>> >>happens  
>> >> >> outside of any authorization code.  
>> >> >> If the TopicConfigCache just kept the json representation and left  
>> >>it to  
>> >> >> authorizer to parse it, the authorizer will have to either parse the  
>> >> >>json  
>> >> >> for each request(not acceptable) or it will have to keep one more  
>> >>layer  
>> >> >>of  
>> >> >> parsed ACL instance cache. Assuming authorizer will keep an  
>> >>additional  
>> >> >> caching layer we will now have to implement some way to invalidate  
>> >>the  
>> >> >> cache which means the TopicConfigCache will have to be an observable  
>> >> >>which  
>> >> >> the Authorizer observes and invalidates its cache entries when  
>> >> >> topicConfigCache gets updated. Seemed like unnecessary complexity  
>> >>with  
>> >> >>not  
>> >> >> lot to gain so I went with TopicConfigCache interpreting the json and  
>> >> >> caching a higher level modeled object.  
>> >> >> In summary, the interpretation is done for both optimization and  
>> >> >> simplicity. If you think it is important to allow custom ACL format  
>> >> >> support we can add one more pluggable config(acl.parser) and  
>> >> >> interface(AclParser) or it could just be another method in  
>> >>Authorizer.  
>> >> >> One thing to note the current ACL json is versioned so it is easy to  
>> >> >>make  
>> >> >> changes to it however it won’t be possible to support custom ACL  
>> >>formats  
>> >> >> with the current design.  
>> >> >> Thanks  
>> >> >> Parth  
>> >> >> On 4/15/15, 4:29 PM, "Michael Herstine"  
>> >><mherstine@linkedin.com.INVALID  
>> >> >> <ma...@linkedin.com.INVALID>>  
>> >> >> wrote:  
>> >> >> >Hi Parth,  
>> >> >> >  
>> >> >> >I’m a little confused: why would Kafka need to interpret the JSON?  
>> >> >>IIRC  
>> >> >> >KIP-11 even says that the TopicConfigData will just store the JSON.  
>> >>I’m  
>> >> >> >not really making a design recommendation here, just trying to  
>> >> >>understand  
>> >> >> >what you’re proposing.  
>> >> >> >  
>> >> >> >On 4/15/15, 11:20 AM, "Parth Brahmbhatt"  
>> >><pbrahmbhatt@hortonworks.com  
>> >> >> <ma...@hortonworks.com>>  
>> >> >> >wrote:  
>> >> >> >  
>> >> >> >>Hi Michael,  
>> >> >> >>  
>> >> >> >>There is code in kafka codebase that reads and interprets the topic  
>> >> >> >>config JSON which has acls, owner and logconfig properties. There  
>> >>are  
>> >> >>3  
>> >> >> >>use cases that we are supporting with current proposal:  
>> >> >> >>  
>> >> >> >> * You use out of box simpleAcl authorizer which is tied to the  
>> >>acl  
>> >> >> >>stored in topic config and the format is locked down.  
>> >> >> >> * You have a custom authorizer and a custom ACL store.  
>> >> >>Ranger/Argus  
>> >> >> >>falls under this as they have their own acl store and ui that users  
>> >> >>use  
>> >> >> >>to configure acls on the cluster and cluster resources like topic.  
>> >> >>It is  
>> >> >> >>upto the custom authorizer to leverage the kafka acl configs or  
>> >> >> >>completely ignore them as they have set a user expectation that  
>> >>only  
>> >> >>acls  
>> >> >> >>configured via their ui/system will be effective.  
>> >> >> >> * You have a custom authorizer but no custom Acl store. You are  
>> >> >> >>completely tied to Acl structure that we have provided in out of  
>> >>box  
>> >> >> >>implementation.  
>> >> >> >>  
>> >> >> >>Thanks  
>> >> >> >>Parth  
>> >> >> >>  
>> >> >> >>On 4/15/15, 10:31 AM, "Michael Herstine"  
>> >> >>  
>> >>>><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID  
>> >> >> ><ma...@linkedin.com.INVALID>>  
>> >> >> >>wrote:  
>> >> >> >>  
>> >> >> >>Hi Parth,  
>> >> >> >>  
>> >> >> >>One question that occurred to me at the end of today’s hangout: how  
>> >> >>tied  
>> >> >> >>are we to a particular ACL representation under your proposal? I  
>> >>know  
>> >> >> >>that  
>> >> >> >>TopicConfigCache will just contain JSON— if a particular site  
>> >>decides  
>> >> >> >>they  
>> >> >> >>want to represent their ACLs differently, and swap out the  
>> >>authorizer  
>> >> >> >>implementation, will that work? I guess what I’m asking is whether  
>> >> >> >>there’s any code in the Kafka codebase that will interpret that  
>> >>JSON,  
>> >> >>or  
>> >> >> >>does that logic live exclusively in the authorizer?  
>> >> >> >>  
>> >> >> >>On 4/14/15, 10:56 PM, "Don Bosco Durai"  
>> >> >>  
>> >>>><bo...@apache.org>>  
>> >> >> wrote:  
>> >> >> >>  
>> >> >> >>I also feel, having just IP would be more appropriate. Host lookup  
>> >> >>will  
>> >> >> >>unnecessary slow things down and would be insecure as you pointed  
>> >>out.  
>> >> >> >>  
>> >> >> >>With IP, it will be also able to setup policies (in future if  
>> >>needed)  
>> >> >> >>with  
>> >> >> >>ranges or netmasks and it would be more scalable.  
>> >> >> >>  
>> >> >> >>Bosco  
>> >> >> >>  
>> >> >> >>  
>> >> >> >>On 4/14/15, 1:40 PM, "Michael Herstine"  
>> >> >>  
>> >>>><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID  
>> >> >> ><ma...@linkedin.com.INVALID>>  
>> >> >> >>wrote:  
>> >> >> >>  
>> >> >> >>Hi Parth,  
>> >> >> >>  
>> >> >> >>Sorry to chime in so late, but I’ve got a minor question on the  
>> >>KIP.  
>> >> >> >>  
>> >> >> >>Several methods take a parameter named “host” of type String. Is  
>> >>that  
>> >> >> >>intended to be a hostname, or an IP address? If the former, I’m  
>> >> >>curious  
>> >> >> >>as  
>> >> >> >>to how that’s found (in my experience, when accepting an incoming  
>> >> >>socket  
>> >> >> >>connection, you only know the IP address, and there isn’t a way to  
>> >>map  
>> >> >> >>that to a hostname without a round trip to a DNS server, which is  
>> >> >> >>insecure  
>> >> >> >>anyway).  
>> >> >> >>  
>> >> >> >>  
>> >> >> >>On 3/25/15, 1:07 PM, "Parth Brahmbhatt"  
>> >> >>  
>> >> >>>><pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com  
>> >> ><mailto  
>> >> >>>>:  
>> >> >> pbrahmbhatt@hortonworks.com>>  
>> >> >> >>wrote:  
>> >> >> >>  
>> >> >> >>Hi all,  
>> >> >> >>  
>> >> >> >>I have modified the KIP to reflect the recent change request from  
>> >>the  
>> >> >> >>reviewers. I have been working on the code and I have the server  
>> >>side  
>> >> >> >>code  
>> >> >> >>for authorization ready. I am now modifying the command line  
>> >> >>utilities.  
>> >> >> >>I  
>> >> >> >>would really appreciate if some of the committers can spend  
>> >>sometime  
>> >> >>to  
>> >> >> >>review the KIP so we can make progress on this.  
>> >> >> >>  
>> >> >> >>Thanks  
>> >> >> >>Parth  
>> >> >> >>  
>> >> >> >>On 3/18/15, 2:20 PM, "Michael Herstine"  
>> >> >>  
>> >>>><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID  
>> >> >> ><ma...@linkedin.com.INVALID>>  
>> >> >> >>wrote:  
>> >> >> >>  
>> >> >> >>Hi Parth,  
>> >> >> >>  
>> >> >> >>Thanks! A few questions:  
>> >> >> >>  
>> >> >> >>1. Do you want to permit rules in your ACLs that DENY access as  
>> >>well  
>> >> >>as  
>> >> >> >>ALLOW? This can be handy setting up rules that have exceptions.  
>> >>E.g.  
>> >> >> >>“Allow principal P to READ resource R from all hosts” with “Deny  
>> >> >> >>principal  
>> >> >> >>P READ access to resource R from host H1” in combination would  
>> >>allow P  
>> >> >> >>to  
>> >> >> >>READ R from all hosts *except* H1.  
>> >> >> >>  
>> >> >> >>2. When a topic is newly created, will there be an ACL created for  
>> >>it?  
>> >> >> >>If  
>> >> >> >>not, would that not deny subsequent access to it?  
>> >> >> >>  
>> >> >> >>(nit) Maybe use Principal instead of String to represent  
>> >>principals?  
>> >> >> >>  
>> >> >> >>  
>> >> >> >>On 3/9/15, 11:48 AM, "Don Bosco Durai"  
>> >> >>  
>> >>>><bo...@apache.org>>  
>> >> >> wrote:  
>> >> >> >>  
>> >> >> >>Parth  
>> >> >> >>  
>> >> >> >>Overall it is looking good. Couple of questionsŠ  
>> >> >> >>  
>> >> >> >>- Can you give an example how the policies will look like in the  
>> >> >> >>default  
>> >> >> >>implementation?  
>> >> >> >>- In the operations, can we support ³CONNECT² also? This can be  
>> >>used  
>> >> >> >>during Session connection  
>> >> >> >>- Regarding access control for ³Topic Creation², since we can¹t do  
>> >>it  
>> >> >> >>on  
>> >> >> >>the server side, can we de-scope it for? And plan it as a future  
>> >> >> >>feature  
>> >> >> >>request?  
>> >> >> >>  
>> >> >> >>Thanks  
>> >> >> >>  
>> >> >> >>Bosco  
>> >> >> >>  
>> >> >> >>  
>> >> >> >>On 3/6/15, 8:10 AM, "Harsha"  
>> >><kafka@harsha.io<mailto:kafka@harsha.io  
>> >> >> ><ma...@harsha.io>>  
>> >> >> >>wrote:  
>> >> >> >>  
>> >> >> >>Hi Parth,  
>> >> >> >> Thanks for putting this together. Overall it looks good  
>> >> >> >>to  
>> >> >> >> me. Although AdminUtils is a concern KIP-4 can  
>> >>probably  
>> >> >> >>fix  
>> >> >> >> that part.  
>> >> >> >>Thanks,  
>> >> >> >>Harsha  
>> >> >> >>  
>> >> >> >>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:  
>> >> >> >>Forgot to add links to wiki and jira.  
>> >> >> >>Link to wiki:  
>> >> >>  
>> >>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza  
>> >> >> >>t  
>> >> >> >>i  
>> >> >> >>o  
>> >> >> >>n  
>> >> >> >>+  
>> >> >> >>Interface  
>> >> >> >>Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688  
>> >> >> >>Thanks  
>> >> >> >>Parth  
>> >> >> >>From: Parth Brahmbhatt  
>> >> >>  
>> >> >>>><pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com  
>> >> ><mailto  
>> >> >>>>:  
>> >> >> pbrahmbhatt@hortonworks.com><mailto:p  
>> >> >> >>b  
>> >> >> >>rahmbhatt@hortonworks.com<ma...@hortonworks.com>>>  
>> >> >> >>Date: Thursday, March 5, 2015 at 10:33 AM  
>> >> >> >>To:  
>> >> >> >>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:  
>> >> >> dev@kafka.apache.org><mailto:dev@kafka.apach  
>> >> >> >>e  
>> >> >> >>.org>"  
>> >> >> >><de...@kafka.apache.org><mailto:  
>> >> >> dev@kafka.apache.org><mailto:dev@kafka.apach  
>> >> >> >>e  
>> >> >> >>.org>>  
>> >> >> >>Subject: [DISCUSS] KIP-11- Authorization design for kafka security  
>> >> >> >>Hi,  
>> >> >> >>KIP-11 is open for discussion , I have updated the wiki with the  
>> >> >> >>design  
>> >> >> >>and open questions.  
>> >> >> >>Thanks  
>> >> >> >>Parth  
>> >> >> >>  
>> >> >> >>  
>> >> >> >>  
>> >> >> >>  
>> >> >> >>  
>> >> >> >>  
>> >> >> >>  
>> >> >> >>  
>> >> >> >>  
>> >> >> >  
>> >> >>  
>> >> >>  
>> >> >>  
>> >>  
>> >>  
>>  
>>  

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Gwen Shapira <gs...@cloudera.com>]

I admit that I'm also not sure what we gain by having "deny" rules.
A user is either in the "allow" list (one way or another) or it will
be denied by default.

I think this is something we can skip for now and add later if needed?

On Mon, Apr 20, 2015 at 5:24 PM, Jun Rao <ju...@confluent.io> wrote:
> According to the pseudo code, if you have a rule "deny user1", then it
> essentially denies all users?
>
> Thanks,
>
> Jun
>
> On Mon, Apr 20, 2015 at 5:16 PM, Parth Brahmbhatt <
> pbrahmbhatt@hortonworks.com> wrote:
>
>>
>> Here is a pseudo code that explains my current approach:
>>
>> acls = authorizer.getAcl(resource)
>> if(acls == null || acls.isEmpty) {
>>         allow all requests for backward compatibility. (any topics that
>> were
>> created prior to security support will not have acls) This is debatable ,
>> generally we should block everyone which is what I would prefer but that
>> means anyone moving to authorizer must go to all of his existing topics
>> and add acl to allow all. If we are fine with imposing this requirement I
>> can start returning deny when no acls are found.
>> } else {
>>         //So the user has set some acls explicitly, this means they have
>> knowingly enabled authorizer. Let’t first check if they have set an Acl to
>> deny this user/host/operation combination.
>>         if some acl denies this request for this principal/host/operation
>> combination , return deny
>>
>>         //this principal/host/operation does not have any explicit deny
>> acl,
>> check if there is some explicit acl that allows the operation
>>         if at least one acl allows this request for this
>> principal/host/operation
>> combination , return allow
>>
>>         // no acl was found for this principal/host/operation combination
>> to
>> allow this operation, so we will deny the request
>>         return deny
>> }
>>
>>
>> Thanks
>> Parth
>>
>>
>> On 4/20/15, 2:21 PM, "Jun Rao" <ju...@confluent.io> wrote:
>>
>> >Hmm, I thought the semantics is that if you only have rule "deny user2",
>> >it
>> >means that everyone except user2 has access?
>> >
>> >Thanks,
>> >
>> >Jun
>> >
>> >On Mon, Apr 20, 2015 at 3:25 PM, Parth Brahmbhatt <
>> >pbrahmbhatt@hortonworks.com> wrote:
>> >
>> >> user3 does not have access and removing the deny rule does not grant him
>> >> or user2 access. user2 even without the deny rule will not have access.
>> >>
>> >> Thanks
>> >> Parth
>> >>
>> >> On 4/20/15, 12:03 PM, "Jun Rao" <ju...@confluent.io> wrote:
>> >>
>> >> >Just a followup question. Suppose there are two rules. Rule1 allows
>> >>user1
>> >> >and rule2 denies user2. Does user3 have access? If not, does removing
>> >> >rule1
>> >> >enable user3 access?
>> >> >
>> >> >Thanks,
>> >> >
>> >> >Jun
>> >> >
>> >> >On Mon, Apr 20, 2015 at 1:34 PM, Parth Brahmbhatt <
>> >> >pbrahmbhatt@hortonworks.com> wrote:
>> >> >
>> >> >>
>> >> >> Hi Joel,
>> >> >>
>> >> >> Thanks for the review and I plan to update the KIP today with all the
>> >> >> updated info. My comments in line below.
>> >> >>
>> >> >> Thanks
>> >> >> Parth
>> >> >>
>> >> >>
>> >> >> On 4/20/15, 10:07 AM, "Joel Koshy" <jjkoshy.w@gmail.com<mailto:
>> >> >> jjkoshy.w@gmail.com>> wrote:
>> >> >>
>> >> >> Hi Parth,
>> >> >>
>> >> >> Nice work on this KIP.  I did another read through and had a few more
>> >> >> comments (with edits after I went through the thread). Many of these
>> >> >> comments were brought up by others as well, so it appears that the
>> >>KIP
>> >> >> would benefit from an update at this point to incorporate comments
>> >> >> from the thread and last hangout.
>> >> >>
>> >> >> - The operation enum is mostly self-explanatory, but it would help
>> >> >>   (for the sake of clarity and completeness if nothing else) to
>> >> >>   document exactly what each of the enums are. E.g., I think this
>> >>came
>> >> >>   up in our hangout - SEND_CONTROL_MESSAGE is unclear and I don't
>> >> >>   remember what was said about it. <Edit>: After going through the
>> >> >>   thread it seems the conclusion was to categorize operations. E.g.,
>> >> >>   WRITE could apply to multiple requests. Again, this is unclear, so
>> >> >>   if it would be great if you could update the KIP to clarify what
>> >>you
>> >> >>   intend.
>> >> >>
>> >> >> Will add to document. SEND_CONTROL_MESSAGE Probably a very bad name
>> >>but
>> >> >> these are intra borker API calls like controller notifying other
>> >> >>brokers to
>> >> >> update metadata or heartbeats. Any better naming suggestions?
>> >> >>
>> >> >> - When you update the KIP to categorize the requests it would also
>> >> >>   help to have a column for what the resource is for each.
>> >> >>
>> >> >> Will add to the KIP.
>> >> >>
>> >> >> - FWIW I prefer a 1-1 mapping between requests and operations. I
>> >>think
>> >> >>   categorizing requests into these can be confusing because:
>> >> >>   - The resource being protected for different requests will be
>> >> >>     different. We are mostly thinking about topics (read/write) but
>> >> >>     there are requests for which topic is not the right resource.
>> >> >>     E.g., for topic creation, the resource as you suggested would be
>> >> >>     something global/common such as “cluster”. For
>> >> >>     OffsetCommit/FetchRequest, the resource may be the consumer
>> >>group,
>> >> >>     or maybe a tuple of <consumer group, topic>. So this can be
>> >> >>     confusing - i.e., different resources and request types in the
>> >> >>     same category. It may be simpler and clearer to just have a 1-1
>> >> >>     mapping between the operation enum and requests.
>> >> >>
>> >> >> I only see 2 resource categories right now cluster and topic.  I
>> >>don’t
>> >> >> really care one way or another so we can probably make a quick
>> >>decision
>> >> >>in
>> >> >> tomorrow’s meeting to either to 1-1 mapping or have categorization?
>> >> >>
>> >> >>   - Some requests that are intuitively READ have WRITE side-effects.
>> >> >>     E.g., (currently) TopicMetadataRequest with auto-create, although
>> >> >>     that will eventually go away. ConsumerMetadataRequest still
>> >> >>     auto-creates the offsets topic. Likewise, ADMIN-type requests may
>> >> >>     be interpreted as having side-effects (depending on who you ask).
>> >> >>
>> >> >> Yes and what I am doing right now is checking authorization for all
>> >> >> possible actions i.e. for auto-create it checks if the config has it
>> >> >> enabled and if yes, check for read + create authorization. Its not
>> >>very
>> >> >> meaningful right now as there is no CREATE authorization but I think
>> >> >>this
>> >> >> is implementation detail, we need to ensure we call authorize with
>> >>all
>> >> >> possible operations from KafkaAPI.
>> >> >> - <quote>When an ACL is missing - fail open</quote>. What does
>> >>missing
>> >> >>   mean? i.e., no explicit ACL for a principal? I'm confused by this
>> >> >>   especially in relation to the precedence of DENY over ALLOW. So per
>> >> >>   the description:
>> >> >>   - If no ACLs exist for topic A then ALLOW all operations on it by
>> >> >>     anyone.
>> >> >>   - If I now add an ACL for a certain principal P to ALLOW (say)
>> >>WRITE
>> >> >>     to the topic then either:
>> >> >>   - This has the effect of DENYing WRITE to all other principals
>> >> >>   - Or, this ACL serves no purpose
>> >> >>   - If the effect is to DENY WRITE to all other principals, what
>> >>about
>> >> >>     READ. Do all principals (including P) have READ permissions to
>> >> >>     topic A?
>> >> >>   - In other words, it seems for a specific ACL to be meaningful then
>> >> >>     fail close is necessary for an absent ACL.
>> >> >>   - <edit>After through the thread: it appears that the DENY override
>> >> >>     only applies to the given principal. i.e., in the above case it
>> >> >>     appears that the other principals will in fact be granted access.
>> >> >>     Then this makes the ACL that was added pointless right?
>> >> >>
>> >> >> The rule I was going with is
>> >> >> - If there is no ACL I.e. This might be a topic that was created in
>> >>non
>> >> >> secure mode or was created before we supported ACLs. We assume you do
>> >> >>not
>> >> >> want authorization and let all requests go through.
>> >> >> - once you add any ACL, we assume you want authorization on the topic
>> >> >>and
>> >> >> all the general authorization rules now start to apply, I.e we fail
>> >> >>close
>> >> >> if we don’t find an ACL that allows access or if we find an ACL that
>> >> >>denies
>> >> >> access. It does not matter if you added a READACL or WRITEACL or
>> >> >>ALLOWACL
>> >> >> or DENY ACL. If you add any ACL, now every user gets checked against
>> >> >>that
>> >> >> and if it does not satisfy the ACL, request fails. I.e. If you add an
>> >> >>ACL
>> >> >> “Allow write to topic-1 form user1 from all hosts” , user-1 has write
>> >> >> access from all hosts and no other user has any access(except for
>> >> >> superusers who have all the access).
>> >> >> - Deny ACLS are suppose to be used to restrict access authorized by
>> >>some
>> >> >> allow ACL, they are not suppose to be required. Implicitly anyone who
>> >> >>does
>> >> >> not have an allow acl, gets denied. The Deny ACLs are only added to
>> >>give
>> >> >> more control to administrators who wants more granular control with
>> >> >>lesser
>> >> >> config. The scenario described in mailing list was “Allow user X
>> >>access
>> >> >> from all hosts but Host1,Host2”. in absence of DENY operator you will
>> >> >>have
>> >> >> to exhaustively list all possible hosts in your ACL which is what we
>> >>are
>> >> >> trying to avoid.
>> >> >>
>> >> >> - On ZK ACLs: I think ZK will be closed to everyone except Kafka
>> >> >>   brokers. This is a dependency on KIP-4 though. i.e., eventually all
>> >> >>   clients should talk to brokers only via RPC.
>> >> >>
>> >> >> Yes.
>> >> >>
>> >> >> - Topic owner: list vs single entry - both have issues off the bat
>> >> >>   (although list is more intuitive at least to me), but perhaps you
>> >> >>   could write up some example workflows to clarify the current
>> >> >>   proposal. I was thinking that anyone in the owner list should be
>> >> >>   considered a super-user of the topic and can grant/revoke
>> >> >>   permissions. They should also be allowed to add other principals as
>> >> >>   owners. Even with this it is unclear who should be allowed to
>> >>remove
>> >> >>   owners.
>> >> >>
>> >> >> As you pointed out in the last KIP meeting owners/creators have use
>> >>out
>> >> >> side of security context (plain simple auditing). I don’t think the
>> >> >> authorizer work depends on this, it was my bad to even mention it in
>> >> >>first
>> >> >> place. I think we can have this discussion outside of
>> >> >>authorizer/security
>> >> >> context and once we have a way to get topic owners the default
>> >> >>Authorizer
>> >> >> can start using it. It makes sense to treat all owners as super users
>> >> >>and I
>> >> >> think it is safe to assume superusers can also modify ownership but I
>> >> >>think
>> >> >> this should not be treated as blocking work for authorization.
>> >> >>
>> >> >> - What is the effect of deleting a topic - should all associated ACLs
>> >> >>   be deleted as well?
>> >> >> They should be and with acls being stored as part of TopicConfig this
>> >> >>was
>> >> >> taken care of automatically. With the new ACL management API the
>> >>users
>> >> >>will
>> >> >> have to call remove ACLs explicitly to perform the cleanup. If
>> >>everyone
>> >> >> thinks this should be automated , with the new APIs we will need a
>> >> >>hook(or
>> >> >> poll) to be notified when a topic is deleted to perform cleanup.
>> >> >> - TopicConfigCache to store topic-ACLs. As mentioned above, not all
>> >> >>   requests will be tied to topics. We may want to have an entirely
>> >> >>   separate ZK directory for ACLs. We have a similar issue with
>> >>quotas.
>> >> >>   This ties in with dynamic config management. We can certainly
>> >> >>   leverage the dynamic config management part of topic configs but I
>> >> >>   think we need to have a story for non-topic resources.
>> >> >>
>> >> >> In the first proposal I was going with a topic-Acl and cluster-Acl
>> >>where
>> >> >> cluster-Acls were json acl local files on all brokers. With the new
>> >>ACL
>> >> >> management APIs we are planning to have /kafka-acl node under which
>> >>all
>> >> >> acls will be stored in /kakfa-acls/resource-name -> {acl json data}.
>> >> >> Cluster acls will just have resource name kafka-cluster.
>> >> >>
>> >> >> Thanks,
>> >> >>
>> >> >> Joel
>> >> >>
>> >> >> On Thu, Apr 16, 2015 at 12:15:37AM +0000, Parth Brahmbhatt wrote:
>> >> >> Kafka currently stores logConfig overrides specified during topic
>> >> >>creation
>> >> >> in zookeeper, its just an instance of java.util.Properties converted
>> >>to
>> >> >> json. I am proposing in addition to that we store acls and owner as
>> >>well
>> >> >> as part of same Properties map.
>> >> >> There is some infrastructure around reading this config, converting
>> >>it
>> >> >> back to Properties map and most importantly propagating any changes
>> >> >> efficiently which we will be able to leverage. As this
>> >>infrastructure is
>> >> >> common to the cluster the reading (not interpreting) of config
>> >>happens
>> >> >> outside of any authorization code.
>> >> >> If the TopicConfigCache just kept the json representation and left
>> >>it to
>> >> >> authorizer to parse it, the authorizer will have to either parse the
>> >> >>json
>> >> >> for each request(not acceptable) or it will have to keep one more
>> >>layer
>> >> >>of
>> >> >> parsed ACL instance cache. Assuming authorizer will keep an
>> >>additional
>> >> >> caching layer we will now have to implement some way to invalidate
>> >>the
>> >> >> cache which means the TopicConfigCache will have to be an observable
>> >> >>which
>> >> >> the Authorizer observes and invalidates its cache entries when
>> >> >> topicConfigCache gets updated. Seemed like unnecessary complexity
>> >>with
>> >> >>not
>> >> >> lot to gain so I went with TopicConfigCache interpreting the json and
>> >> >> caching a higher level modeled object.
>> >> >> In summary, the interpretation is done for both optimization and
>> >> >> simplicity. If you think it is important to allow custom ACL format
>> >> >> support we can add one more pluggable config(acl.parser) and
>> >> >> interface(AclParser) or it could just be another method in
>> >>Authorizer.
>> >> >> One thing to note the current ACL json is versioned so it is easy to
>> >> >>make
>> >> >> changes to it however it won’t be possible to support custom ACL
>> >>formats
>> >> >> with the current design.
>> >> >> Thanks
>> >> >> Parth
>> >> >> On 4/15/15, 4:29 PM, "Michael Herstine"
>> >><mherstine@linkedin.com.INVALID
>> >> >> <ma...@linkedin.com.INVALID>>
>> >> >> wrote:
>> >> >> >Hi Parth,
>> >> >> >
>> >> >> >I’m a little confused: why would Kafka need to interpret the JSON?
>> >> >>IIRC
>> >> >> >KIP-11 even says that the TopicConfigData will just store the JSON.
>> >>I’m
>> >> >> >not really making a design recommendation here, just trying to
>> >> >>understand
>> >> >> >what you’re proposing.
>> >> >> >
>> >> >> >On 4/15/15, 11:20 AM, "Parth Brahmbhatt"
>> >><pbrahmbhatt@hortonworks.com
>> >> >> <ma...@hortonworks.com>>
>> >> >> >wrote:
>> >> >> >
>> >> >> >>Hi Michael,
>> >> >> >>
>> >> >> >>There is code in kafka codebase that reads and interprets the topic
>> >> >> >>config JSON which has acls, owner and logconfig properties. There
>> >>are
>> >> >>3
>> >> >> >>use cases that we are supporting with current proposal:
>> >> >> >>
>> >> >> >>  *   You use out of box simpleAcl authorizer which is tied to the
>> >>acl
>> >> >> >>stored in topic config and the format is locked down.
>> >> >> >>  *   You have a custom authorizer and a custom ACL store.
>> >> >>Ranger/Argus
>> >> >> >>falls under this as they have their own acl store and ui that users
>> >> >>use
>> >> >> >>to configure acls on the cluster and cluster resources  like topic.
>> >> >>It is
>> >> >> >>upto the custom authorizer to leverage the kafka acl configs or
>> >> >> >>completely ignore them as they have set a user expectation that
>> >>only
>> >> >>acls
>> >> >> >>configured via their ui/system will be effective.
>> >> >> >>  *   You have a custom authorizer but no custom Acl store. You are
>> >> >> >>completely tied to Acl structure that we have provided in out of
>> >>box
>> >> >> >>implementation.
>> >> >> >>
>> >> >> >>Thanks
>> >> >> >>Parth
>> >> >> >>
>> >> >> >>On 4/15/15, 10:31 AM, "Michael Herstine"
>> >> >>
>> >>>><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
>> >> >> ><ma...@linkedin.com.INVALID>>
>> >> >> >>wrote:
>> >> >> >>
>> >> >> >>Hi Parth,
>> >> >> >>
>> >> >> >>One question that occurred to me at the end of today’s hangout: how
>> >> >>tied
>> >> >> >>are we to a particular ACL representation under your proposal? I
>> >>know
>> >> >> >>that
>> >> >> >>TopicConfigCache will just contain JSON— if a particular site
>> >>decides
>> >> >> >>they
>> >> >> >>want to represent their ACLs differently, and swap out the
>> >>authorizer
>> >> >> >>implementation, will that work?  I guess what I’m asking is whether
>> >> >> >>there’s any code in the Kafka codebase that will interpret that
>> >>JSON,
>> >> >>or
>> >> >> >>does that logic live exclusively in the authorizer?
>> >> >> >>
>> >> >> >>On 4/14/15, 10:56 PM, "Don Bosco Durai"
>> >> >>
>> >>>><bo...@apache.org>>
>> >> >> wrote:
>> >> >> >>
>> >> >> >>I also feel, having just IP would be more appropriate. Host lookup
>> >> >>will
>> >> >> >>unnecessary slow things down and would be insecure as you pointed
>> >>out.
>> >> >> >>
>> >> >> >>With IP, it will be also able to setup policies (in future if
>> >>needed)
>> >> >> >>with
>> >> >> >>ranges or netmasks and it would be more scalable.
>> >> >> >>
>> >> >> >>Bosco
>> >> >> >>
>> >> >> >>
>> >> >> >>On 4/14/15, 1:40 PM, "Michael Herstine"
>> >> >>
>> >>>><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
>> >> >> ><ma...@linkedin.com.INVALID>>
>> >> >> >>wrote:
>> >> >> >>
>> >> >> >>Hi Parth,
>> >> >> >>
>> >> >> >>Sorry to chime in so late, but I’ve got a minor question on the
>> >>KIP.
>> >> >> >>
>> >> >> >>Several methods take a parameter named “host” of type String. Is
>> >>that
>> >> >> >>intended to be a hostname, or an IP address? If the former, I’m
>> >> >>curious
>> >> >> >>as
>> >> >> >>to how that’s found (in my experience, when accepting an incoming
>> >> >>socket
>> >> >> >>connection, you only know the IP address, and there isn’t a way to
>> >>map
>> >> >> >>that to a hostname without a round trip to a DNS server, which is
>> >> >> >>insecure
>> >> >> >>anyway).
>> >> >> >>
>> >> >> >>
>> >> >> >>On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
>> >> >>
>> >> >>>><pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
>> >> ><mailto
>> >> >>>>:
>> >> >> pbrahmbhatt@hortonworks.com>>
>> >> >> >>wrote:
>> >> >> >>
>> >> >> >>Hi all,
>> >> >> >>
>> >> >> >>I have modified the KIP to reflect the recent change request from
>> >>the
>> >> >> >>reviewers. I have been working on the code and I have the server
>> >>side
>> >> >> >>code
>> >> >> >>for authorization ready. I am now modifying the command line
>> >> >>utilities.
>> >> >> >>I
>> >> >> >>would really appreciate if some of the committers can spend
>> >>sometime
>> >> >>to
>> >> >> >>review the KIP so we can make progress on this.
>> >> >> >>
>> >> >> >>Thanks
>> >> >> >>Parth
>> >> >> >>
>> >> >> >>On 3/18/15, 2:20 PM, "Michael Herstine"
>> >> >>
>> >>>><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
>> >> >> ><ma...@linkedin.com.INVALID>>
>> >> >> >>wrote:
>> >> >> >>
>> >> >> >>Hi Parth,
>> >> >> >>
>> >> >> >>Thanks! A few questions:
>> >> >> >>
>> >> >> >>1. Do you want to permit rules in your ACLs that DENY access as
>> >>well
>> >> >>as
>> >> >> >>ALLOW? This can be handy setting up rules that have exceptions.
>> >>E.g.
>> >> >> >>“Allow principal P to READ resource R from all hosts” with “Deny
>> >> >> >>principal
>> >> >> >>P READ access to resource R from host H1” in combination would
>> >>allow P
>> >> >> >>to
>> >> >> >>READ R from all hosts *except* H1.
>> >> >> >>
>> >> >> >>2. When a topic is newly created, will there be an ACL created for
>> >>it?
>> >> >> >>If
>> >> >> >>not, would that not deny subsequent access to it?
>> >> >> >>
>> >> >> >>(nit) Maybe use Principal instead of String to represent
>> >>principals?
>> >> >> >>
>> >> >> >>
>> >> >> >>On 3/9/15, 11:48 AM, "Don Bosco Durai"
>> >> >>
>> >>>><bo...@apache.org>>
>> >> >> wrote:
>> >> >> >>
>> >> >> >>Parth
>> >> >> >>
>> >> >> >>Overall it is looking good. Couple of questionsŠ
>> >> >> >>
>> >> >> >>- Can you give an example how the policies will look like in the
>> >> >> >>default
>> >> >> >>implementation?
>> >> >> >>- In the operations, can we support ³CONNECT² also? This can be
>> >>used
>> >> >> >>during Session connection
>> >> >> >>- Regarding access control for ³Topic Creation², since we can¹t do
>> >>it
>> >> >> >>on
>> >> >> >>the server side, can we de-scope it for? And plan it as a future
>> >> >> >>feature
>> >> >> >>request?
>> >> >> >>
>> >> >> >>Thanks
>> >> >> >>
>> >> >> >>Bosco
>> >> >> >>
>> >> >> >>
>> >> >> >>On 3/6/15, 8:10 AM, "Harsha"
>> >><kafka@harsha.io<mailto:kafka@harsha.io
>> >> >> ><ma...@harsha.io>>
>> >> >> >>wrote:
>> >> >> >>
>> >> >> >>Hi Parth,
>> >> >> >>            Thanks for putting this together. Overall it looks good
>> >> >> >>to
>> >> >> >>            me. Although AdminUtils is a concern KIP-4  can
>> >>probably
>> >> >> >>fix
>> >> >> >>            that part.
>> >> >> >>Thanks,
>> >> >> >>Harsha
>> >> >> >>
>> >> >> >>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>> >> >> >>Forgot to add links to wiki and jira.
>> >> >> >>Link to wiki:
>> >> >>
>> >>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
>> >> >> >>t
>> >> >> >>i
>> >> >> >>o
>> >> >> >>n
>> >> >> >>+
>> >> >> >>Interface
>> >> >> >>Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>> >> >> >>Thanks
>> >> >> >>Parth
>> >> >> >>From: Parth Brahmbhatt
>> >> >>
>> >> >>>><pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
>> >> ><mailto
>> >> >>>>:
>> >> >> pbrahmbhatt@hortonworks.com><mailto:p
>> >> >> >>b
>> >> >> >>rahmbhatt@hortonworks.com<ma...@hortonworks.com>>>
>> >> >> >>Date: Thursday, March 5, 2015 at 10:33 AM
>> >> >> >>To:
>> >> >> >>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:
>> >> >> dev@kafka.apache.org><mailto:dev@kafka.apach
>> >> >> >>e
>> >> >> >>.org>"
>> >> >> >><de...@kafka.apache.org><mailto:
>> >> >> dev@kafka.apache.org><mailto:dev@kafka.apach
>> >> >> >>e
>> >> >> >>.org>>
>> >> >> >>Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>> >> >> >>Hi,
>> >> >> >>KIP-11 is open for discussion , I have updated the wiki with the
>> >> >> >>design
>> >> >> >>and open questions.
>> >> >> >>Thanks
>> >> >> >>Parth
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >>
>> >>
>>
>>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Jun Rao <ju...@confluent.io>]

According to the pseudo code, if you have a rule "deny user1", then it
essentially denies all users?

Thanks,

Jun

On Mon, Apr 20, 2015 at 5:16 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:

>
> Here is a pseudo code that explains my current approach:
>
> acls = authorizer.getAcl(resource)
> if(acls == null || acls.isEmpty) {
>         allow all requests for backward compatibility. (any topics that
> were
> created prior to security support will not have acls) This is debatable ,
> generally we should block everyone which is what I would prefer but that
> means anyone moving to authorizer must go to all of his existing topics
> and add acl to allow all. If we are fine with imposing this requirement I
> can start returning deny when no acls are found.
> } else {
>         //So the user has set some acls explicitly, this means they have
> knowingly enabled authorizer. Let’t first check if they have set an Acl to
> deny this user/host/operation combination.
>         if some acl denies this request for this principal/host/operation
> combination , return deny
>
>         //this principal/host/operation does not have any explicit deny
> acl,
> check if there is some explicit acl that allows the operation
>         if at least one acl allows this request for this
> principal/host/operation
> combination , return allow
>
>         // no acl was found for this principal/host/operation combination
> to
> allow this operation, so we will deny the request
>         return deny
> }
>
>
> Thanks
> Parth
>
>
> On 4/20/15, 2:21 PM, "Jun Rao" <ju...@confluent.io> wrote:
>
> >Hmm, I thought the semantics is that if you only have rule "deny user2",
> >it
> >means that everyone except user2 has access?
> >
> >Thanks,
> >
> >Jun
> >
> >On Mon, Apr 20, 2015 at 3:25 PM, Parth Brahmbhatt <
> >pbrahmbhatt@hortonworks.com> wrote:
> >
> >> user3 does not have access and removing the deny rule does not grant him
> >> or user2 access. user2 even without the deny rule will not have access.
> >>
> >> Thanks
> >> Parth
> >>
> >> On 4/20/15, 12:03 PM, "Jun Rao" <ju...@confluent.io> wrote:
> >>
> >> >Just a followup question. Suppose there are two rules. Rule1 allows
> >>user1
> >> >and rule2 denies user2. Does user3 have access? If not, does removing
> >> >rule1
> >> >enable user3 access?
> >> >
> >> >Thanks,
> >> >
> >> >Jun
> >> >
> >> >On Mon, Apr 20, 2015 at 1:34 PM, Parth Brahmbhatt <
> >> >pbrahmbhatt@hortonworks.com> wrote:
> >> >
> >> >>
> >> >> Hi Joel,
> >> >>
> >> >> Thanks for the review and I plan to update the KIP today with all the
> >> >> updated info. My comments in line below.
> >> >>
> >> >> Thanks
> >> >> Parth
> >> >>
> >> >>
> >> >> On 4/20/15, 10:07 AM, "Joel Koshy" <jjkoshy.w@gmail.com<mailto:
> >> >> jjkoshy.w@gmail.com>> wrote:
> >> >>
> >> >> Hi Parth,
> >> >>
> >> >> Nice work on this KIP.  I did another read through and had a few more
> >> >> comments (with edits after I went through the thread). Many of these
> >> >> comments were brought up by others as well, so it appears that the
> >>KIP
> >> >> would benefit from an update at this point to incorporate comments
> >> >> from the thread and last hangout.
> >> >>
> >> >> - The operation enum is mostly self-explanatory, but it would help
> >> >>   (for the sake of clarity and completeness if nothing else) to
> >> >>   document exactly what each of the enums are. E.g., I think this
> >>came
> >> >>   up in our hangout - SEND_CONTROL_MESSAGE is unclear and I don't
> >> >>   remember what was said about it. <Edit>: After going through the
> >> >>   thread it seems the conclusion was to categorize operations. E.g.,
> >> >>   WRITE could apply to multiple requests. Again, this is unclear, so
> >> >>   if it would be great if you could update the KIP to clarify what
> >>you
> >> >>   intend.
> >> >>
> >> >> Will add to document. SEND_CONTROL_MESSAGE Probably a very bad name
> >>but
> >> >> these are intra borker API calls like controller notifying other
> >> >>brokers to
> >> >> update metadata or heartbeats. Any better naming suggestions?
> >> >>
> >> >> - When you update the KIP to categorize the requests it would also
> >> >>   help to have a column for what the resource is for each.
> >> >>
> >> >> Will add to the KIP.
> >> >>
> >> >> - FWIW I prefer a 1-1 mapping between requests and operations. I
> >>think
> >> >>   categorizing requests into these can be confusing because:
> >> >>   - The resource being protected for different requests will be
> >> >>     different. We are mostly thinking about topics (read/write) but
> >> >>     there are requests for which topic is not the right resource.
> >> >>     E.g., for topic creation, the resource as you suggested would be
> >> >>     something global/common such as “cluster”. For
> >> >>     OffsetCommit/FetchRequest, the resource may be the consumer
> >>group,
> >> >>     or maybe a tuple of <consumer group, topic>. So this can be
> >> >>     confusing - i.e., different resources and request types in the
> >> >>     same category. It may be simpler and clearer to just have a 1-1
> >> >>     mapping between the operation enum and requests.
> >> >>
> >> >> I only see 2 resource categories right now cluster and topic.  I
> >>don’t
> >> >> really care one way or another so we can probably make a quick
> >>decision
> >> >>in
> >> >> tomorrow’s meeting to either to 1-1 mapping or have categorization?
> >> >>
> >> >>   - Some requests that are intuitively READ have WRITE side-effects.
> >> >>     E.g., (currently) TopicMetadataRequest with auto-create, although
> >> >>     that will eventually go away. ConsumerMetadataRequest still
> >> >>     auto-creates the offsets topic. Likewise, ADMIN-type requests may
> >> >>     be interpreted as having side-effects (depending on who you ask).
> >> >>
> >> >> Yes and what I am doing right now is checking authorization for all
> >> >> possible actions i.e. for auto-create it checks if the config has it
> >> >> enabled and if yes, check for read + create authorization. Its not
> >>very
> >> >> meaningful right now as there is no CREATE authorization but I think
> >> >>this
> >> >> is implementation detail, we need to ensure we call authorize with
> >>all
> >> >> possible operations from KafkaAPI.
> >> >> - <quote>When an ACL is missing - fail open</quote>. What does
> >>missing
> >> >>   mean? i.e., no explicit ACL for a principal? I'm confused by this
> >> >>   especially in relation to the precedence of DENY over ALLOW. So per
> >> >>   the description:
> >> >>   - If no ACLs exist for topic A then ALLOW all operations on it by
> >> >>     anyone.
> >> >>   - If I now add an ACL for a certain principal P to ALLOW (say)
> >>WRITE
> >> >>     to the topic then either:
> >> >>   - This has the effect of DENYing WRITE to all other principals
> >> >>   - Or, this ACL serves no purpose
> >> >>   - If the effect is to DENY WRITE to all other principals, what
> >>about
> >> >>     READ. Do all principals (including P) have READ permissions to
> >> >>     topic A?
> >> >>   - In other words, it seems for a specific ACL to be meaningful then
> >> >>     fail close is necessary for an absent ACL.
> >> >>   - <edit>After through the thread: it appears that the DENY override
> >> >>     only applies to the given principal. i.e., in the above case it
> >> >>     appears that the other principals will in fact be granted access.
> >> >>     Then this makes the ACL that was added pointless right?
> >> >>
> >> >> The rule I was going with is
> >> >> - If there is no ACL I.e. This might be a topic that was created in
> >>non
> >> >> secure mode or was created before we supported ACLs. We assume you do
> >> >>not
> >> >> want authorization and let all requests go through.
> >> >> - once you add any ACL, we assume you want authorization on the topic
> >> >>and
> >> >> all the general authorization rules now start to apply, I.e we fail
> >> >>close
> >> >> if we don’t find an ACL that allows access or if we find an ACL that
> >> >>denies
> >> >> access. It does not matter if you added a READACL or WRITEACL or
> >> >>ALLOWACL
> >> >> or DENY ACL. If you add any ACL, now every user gets checked against
> >> >>that
> >> >> and if it does not satisfy the ACL, request fails. I.e. If you add an
> >> >>ACL
> >> >> “Allow write to topic-1 form user1 from all hosts” , user-1 has write
> >> >> access from all hosts and no other user has any access(except for
> >> >> superusers who have all the access).
> >> >> - Deny ACLS are suppose to be used to restrict access authorized by
> >>some
> >> >> allow ACL, they are not suppose to be required. Implicitly anyone who
> >> >>does
> >> >> not have an allow acl, gets denied. The Deny ACLs are only added to
> >>give
> >> >> more control to administrators who wants more granular control with
> >> >>lesser
> >> >> config. The scenario described in mailing list was “Allow user X
> >>access
> >> >> from all hosts but Host1,Host2”. in absence of DENY operator you will
> >> >>have
> >> >> to exhaustively list all possible hosts in your ACL which is what we
> >>are
> >> >> trying to avoid.
> >> >>
> >> >> - On ZK ACLs: I think ZK will be closed to everyone except Kafka
> >> >>   brokers. This is a dependency on KIP-4 though. i.e., eventually all
> >> >>   clients should talk to brokers only via RPC.
> >> >>
> >> >> Yes.
> >> >>
> >> >> - Topic owner: list vs single entry - both have issues off the bat
> >> >>   (although list is more intuitive at least to me), but perhaps you
> >> >>   could write up some example workflows to clarify the current
> >> >>   proposal. I was thinking that anyone in the owner list should be
> >> >>   considered a super-user of the topic and can grant/revoke
> >> >>   permissions. They should also be allowed to add other principals as
> >> >>   owners. Even with this it is unclear who should be allowed to
> >>remove
> >> >>   owners.
> >> >>
> >> >> As you pointed out in the last KIP meeting owners/creators have use
> >>out
> >> >> side of security context (plain simple auditing). I don’t think the
> >> >> authorizer work depends on this, it was my bad to even mention it in
> >> >>first
> >> >> place. I think we can have this discussion outside of
> >> >>authorizer/security
> >> >> context and once we have a way to get topic owners the default
> >> >>Authorizer
> >> >> can start using it. It makes sense to treat all owners as super users
> >> >>and I
> >> >> think it is safe to assume superusers can also modify ownership but I
> >> >>think
> >> >> this should not be treated as blocking work for authorization.
> >> >>
> >> >> - What is the effect of deleting a topic - should all associated ACLs
> >> >>   be deleted as well?
> >> >> They should be and with acls being stored as part of TopicConfig this
> >> >>was
> >> >> taken care of automatically. With the new ACL management API the
> >>users
> >> >>will
> >> >> have to call remove ACLs explicitly to perform the cleanup. If
> >>everyone
> >> >> thinks this should be automated , with the new APIs we will need a
> >> >>hook(or
> >> >> poll) to be notified when a topic is deleted to perform cleanup.
> >> >> - TopicConfigCache to store topic-ACLs. As mentioned above, not all
> >> >>   requests will be tied to topics. We may want to have an entirely
> >> >>   separate ZK directory for ACLs. We have a similar issue with
> >>quotas.
> >> >>   This ties in with dynamic config management. We can certainly
> >> >>   leverage the dynamic config management part of topic configs but I
> >> >>   think we need to have a story for non-topic resources.
> >> >>
> >> >> In the first proposal I was going with a topic-Acl and cluster-Acl
> >>where
> >> >> cluster-Acls were json acl local files on all brokers. With the new
> >>ACL
> >> >> management APIs we are planning to have /kafka-acl node under which
> >>all
> >> >> acls will be stored in /kakfa-acls/resource-name -> {acl json data}.
> >> >> Cluster acls will just have resource name kafka-cluster.
> >> >>
> >> >> Thanks,
> >> >>
> >> >> Joel
> >> >>
> >> >> On Thu, Apr 16, 2015 at 12:15:37AM +0000, Parth Brahmbhatt wrote:
> >> >> Kafka currently stores logConfig overrides specified during topic
> >> >>creation
> >> >> in zookeeper, its just an instance of java.util.Properties converted
> >>to
> >> >> json. I am proposing in addition to that we store acls and owner as
> >>well
> >> >> as part of same Properties map.
> >> >> There is some infrastructure around reading this config, converting
> >>it
> >> >> back to Properties map and most importantly propagating any changes
> >> >> efficiently which we will be able to leverage. As this
> >>infrastructure is
> >> >> common to the cluster the reading (not interpreting) of config
> >>happens
> >> >> outside of any authorization code.
> >> >> If the TopicConfigCache just kept the json representation and left
> >>it to
> >> >> authorizer to parse it, the authorizer will have to either parse the
> >> >>json
> >> >> for each request(not acceptable) or it will have to keep one more
> >>layer
> >> >>of
> >> >> parsed ACL instance cache. Assuming authorizer will keep an
> >>additional
> >> >> caching layer we will now have to implement some way to invalidate
> >>the
> >> >> cache which means the TopicConfigCache will have to be an observable
> >> >>which
> >> >> the Authorizer observes and invalidates its cache entries when
> >> >> topicConfigCache gets updated. Seemed like unnecessary complexity
> >>with
> >> >>not
> >> >> lot to gain so I went with TopicConfigCache interpreting the json and
> >> >> caching a higher level modeled object.
> >> >> In summary, the interpretation is done for both optimization and
> >> >> simplicity. If you think it is important to allow custom ACL format
> >> >> support we can add one more pluggable config(acl.parser) and
> >> >> interface(AclParser) or it could just be another method in
> >>Authorizer.
> >> >> One thing to note the current ACL json is versioned so it is easy to
> >> >>make
> >> >> changes to it however it won’t be possible to support custom ACL
> >>formats
> >> >> with the current design.
> >> >> Thanks
> >> >> Parth
> >> >> On 4/15/15, 4:29 PM, "Michael Herstine"
> >><mherstine@linkedin.com.INVALID
> >> >> <ma...@linkedin.com.INVALID>>
> >> >> wrote:
> >> >> >Hi Parth,
> >> >> >
> >> >> >I’m a little confused: why would Kafka need to interpret the JSON?
> >> >>IIRC
> >> >> >KIP-11 even says that the TopicConfigData will just store the JSON.
> >>I’m
> >> >> >not really making a design recommendation here, just trying to
> >> >>understand
> >> >> >what you’re proposing.
> >> >> >
> >> >> >On 4/15/15, 11:20 AM, "Parth Brahmbhatt"
> >><pbrahmbhatt@hortonworks.com
> >> >> <ma...@hortonworks.com>>
> >> >> >wrote:
> >> >> >
> >> >> >>Hi Michael,
> >> >> >>
> >> >> >>There is code in kafka codebase that reads and interprets the topic
> >> >> >>config JSON which has acls, owner and logconfig properties. There
> >>are
> >> >>3
> >> >> >>use cases that we are supporting with current proposal:
> >> >> >>
> >> >> >>  *   You use out of box simpleAcl authorizer which is tied to the
> >>acl
> >> >> >>stored in topic config and the format is locked down.
> >> >> >>  *   You have a custom authorizer and a custom ACL store.
> >> >>Ranger/Argus
> >> >> >>falls under this as they have their own acl store and ui that users
> >> >>use
> >> >> >>to configure acls on the cluster and cluster resources  like topic.
> >> >>It is
> >> >> >>upto the custom authorizer to leverage the kafka acl configs or
> >> >> >>completely ignore them as they have set a user expectation that
> >>only
> >> >>acls
> >> >> >>configured via their ui/system will be effective.
> >> >> >>  *   You have a custom authorizer but no custom Acl store. You are
> >> >> >>completely tied to Acl structure that we have provided in out of
> >>box
> >> >> >>implementation.
> >> >> >>
> >> >> >>Thanks
> >> >> >>Parth
> >> >> >>
> >> >> >>On 4/15/15, 10:31 AM, "Michael Herstine"
> >> >>
> >>>><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
> >> >> ><ma...@linkedin.com.INVALID>>
> >> >> >>wrote:
> >> >> >>
> >> >> >>Hi Parth,
> >> >> >>
> >> >> >>One question that occurred to me at the end of today’s hangout: how
> >> >>tied
> >> >> >>are we to a particular ACL representation under your proposal? I
> >>know
> >> >> >>that
> >> >> >>TopicConfigCache will just contain JSON— if a particular site
> >>decides
> >> >> >>they
> >> >> >>want to represent their ACLs differently, and swap out the
> >>authorizer
> >> >> >>implementation, will that work?  I guess what I’m asking is whether
> >> >> >>there’s any code in the Kafka codebase that will interpret that
> >>JSON,
> >> >>or
> >> >> >>does that logic live exclusively in the authorizer?
> >> >> >>
> >> >> >>On 4/14/15, 10:56 PM, "Don Bosco Durai"
> >> >>
> >>>><bo...@apache.org>>
> >> >> wrote:
> >> >> >>
> >> >> >>I also feel, having just IP would be more appropriate. Host lookup
> >> >>will
> >> >> >>unnecessary slow things down and would be insecure as you pointed
> >>out.
> >> >> >>
> >> >> >>With IP, it will be also able to setup policies (in future if
> >>needed)
> >> >> >>with
> >> >> >>ranges or netmasks and it would be more scalable.
> >> >> >>
> >> >> >>Bosco
> >> >> >>
> >> >> >>
> >> >> >>On 4/14/15, 1:40 PM, "Michael Herstine"
> >> >>
> >>>><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
> >> >> ><ma...@linkedin.com.INVALID>>
> >> >> >>wrote:
> >> >> >>
> >> >> >>Hi Parth,
> >> >> >>
> >> >> >>Sorry to chime in so late, but I’ve got a minor question on the
> >>KIP.
> >> >> >>
> >> >> >>Several methods take a parameter named “host” of type String. Is
> >>that
> >> >> >>intended to be a hostname, or an IP address? If the former, I’m
> >> >>curious
> >> >> >>as
> >> >> >>to how that’s found (in my experience, when accepting an incoming
> >> >>socket
> >> >> >>connection, you only know the IP address, and there isn’t a way to
> >>map
> >> >> >>that to a hostname without a round trip to a DNS server, which is
> >> >> >>insecure
> >> >> >>anyway).
> >> >> >>
> >> >> >>
> >> >> >>On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
> >> >>
> >> >>>><pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
> >> ><mailto
> >> >>>>:
> >> >> pbrahmbhatt@hortonworks.com>>
> >> >> >>wrote:
> >> >> >>
> >> >> >>Hi all,
> >> >> >>
> >> >> >>I have modified the KIP to reflect the recent change request from
> >>the
> >> >> >>reviewers. I have been working on the code and I have the server
> >>side
> >> >> >>code
> >> >> >>for authorization ready. I am now modifying the command line
> >> >>utilities.
> >> >> >>I
> >> >> >>would really appreciate if some of the committers can spend
> >>sometime
> >> >>to
> >> >> >>review the KIP so we can make progress on this.
> >> >> >>
> >> >> >>Thanks
> >> >> >>Parth
> >> >> >>
> >> >> >>On 3/18/15, 2:20 PM, "Michael Herstine"
> >> >>
> >>>><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
> >> >> ><ma...@linkedin.com.INVALID>>
> >> >> >>wrote:
> >> >> >>
> >> >> >>Hi Parth,
> >> >> >>
> >> >> >>Thanks! A few questions:
> >> >> >>
> >> >> >>1. Do you want to permit rules in your ACLs that DENY access as
> >>well
> >> >>as
> >> >> >>ALLOW? This can be handy setting up rules that have exceptions.
> >>E.g.
> >> >> >>“Allow principal P to READ resource R from all hosts” with “Deny
> >> >> >>principal
> >> >> >>P READ access to resource R from host H1” in combination would
> >>allow P
> >> >> >>to
> >> >> >>READ R from all hosts *except* H1.
> >> >> >>
> >> >> >>2. When a topic is newly created, will there be an ACL created for
> >>it?
> >> >> >>If
> >> >> >>not, would that not deny subsequent access to it?
> >> >> >>
> >> >> >>(nit) Maybe use Principal instead of String to represent
> >>principals?
> >> >> >>
> >> >> >>
> >> >> >>On 3/9/15, 11:48 AM, "Don Bosco Durai"
> >> >>
> >>>><bo...@apache.org>>
> >> >> wrote:
> >> >> >>
> >> >> >>Parth
> >> >> >>
> >> >> >>Overall it is looking good. Couple of questionsŠ
> >> >> >>
> >> >> >>- Can you give an example how the policies will look like in the
> >> >> >>default
> >> >> >>implementation?
> >> >> >>- In the operations, can we support ³CONNECT² also? This can be
> >>used
> >> >> >>during Session connection
> >> >> >>- Regarding access control for ³Topic Creation², since we can¹t do
> >>it
> >> >> >>on
> >> >> >>the server side, can we de-scope it for? And plan it as a future
> >> >> >>feature
> >> >> >>request?
> >> >> >>
> >> >> >>Thanks
> >> >> >>
> >> >> >>Bosco
> >> >> >>
> >> >> >>
> >> >> >>On 3/6/15, 8:10 AM, "Harsha"
> >><kafka@harsha.io<mailto:kafka@harsha.io
> >> >> ><ma...@harsha.io>>
> >> >> >>wrote:
> >> >> >>
> >> >> >>Hi Parth,
> >> >> >>            Thanks for putting this together. Overall it looks good
> >> >> >>to
> >> >> >>            me. Although AdminUtils is a concern KIP-4  can
> >>probably
> >> >> >>fix
> >> >> >>            that part.
> >> >> >>Thanks,
> >> >> >>Harsha
> >> >> >>
> >> >> >>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
> >> >> >>Forgot to add links to wiki and jira.
> >> >> >>Link to wiki:
> >> >>
> >>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
> >> >> >>t
> >> >> >>i
> >> >> >>o
> >> >> >>n
> >> >> >>+
> >> >> >>Interface
> >> >> >>Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
> >> >> >>Thanks
> >> >> >>Parth
> >> >> >>From: Parth Brahmbhatt
> >> >>
> >> >>>><pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
> >> ><mailto
> >> >>>>:
> >> >> pbrahmbhatt@hortonworks.com><mailto:p
> >> >> >>b
> >> >> >>rahmbhatt@hortonworks.com<ma...@hortonworks.com>>>
> >> >> >>Date: Thursday, March 5, 2015 at 10:33 AM
> >> >> >>To:
> >> >> >>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:
> >> >> dev@kafka.apache.org><mailto:dev@kafka.apach
> >> >> >>e
> >> >> >>.org>"
> >> >> >><de...@kafka.apache.org><mailto:
> >> >> dev@kafka.apache.org><mailto:dev@kafka.apach
> >> >> >>e
> >> >> >>.org>>
> >> >> >>Subject: [DISCUSS] KIP-11- Authorization design for kafka security
> >> >> >>Hi,
> >> >> >>KIP-11 is open for discussion , I have updated the wiki with the
> >> >> >>design
> >> >> >>and open questions.
> >> >> >>Thanks
> >> >> >>Parth
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >
> >> >>
> >> >>
> >> >>
> >>
> >>
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Here is a pseudo code that explains my current approach:

acls = authorizer.getAcl(resource)
if(acls == null || acls.isEmpty) {
	allow all requests for backward compatibility. (any topics that were
created prior to security support will not have acls) This is debatable ,
generally we should block everyone which is what I would prefer but that
means anyone moving to authorizer must go to all of his existing topics
and add acl to allow all. If we are fine with imposing this requirement I
can start returning deny when no acls are found.
} else {
	//So the user has set some acls explicitly, this means they have
knowingly enabled authorizer. Let’t first check if they have set an Acl to
deny this user/host/operation combination.
	if some acl denies this request for this principal/host/operation
combination , return deny

	//this principal/host/operation does not have any explicit deny acl,
check if there is some explicit acl that allows the operation
	if at least one acl allows this request for this principal/host/operation
combination , return allow
	
	// no acl was found for this principal/host/operation combination to
allow this operation, so we will deny the request
	return deny
}


Thanks
Parth


On 4/20/15, 2:21 PM, "Jun Rao" <ju...@confluent.io> wrote:

>Hmm, I thought the semantics is that if you only have rule "deny user2",
>it
>means that everyone except user2 has access?
>
>Thanks,
>
>Jun
>
>On Mon, Apr 20, 2015 at 3:25 PM, Parth Brahmbhatt <
>pbrahmbhatt@hortonworks.com> wrote:
>
>> user3 does not have access and removing the deny rule does not grant him
>> or user2 access. user2 even without the deny rule will not have access.
>>
>> Thanks
>> Parth
>>
>> On 4/20/15, 12:03 PM, "Jun Rao" <ju...@confluent.io> wrote:
>>
>> >Just a followup question. Suppose there are two rules. Rule1 allows
>>user1
>> >and rule2 denies user2. Does user3 have access? If not, does removing
>> >rule1
>> >enable user3 access?
>> >
>> >Thanks,
>> >
>> >Jun
>> >
>> >On Mon, Apr 20, 2015 at 1:34 PM, Parth Brahmbhatt <
>> >pbrahmbhatt@hortonworks.com> wrote:
>> >
>> >>
>> >> Hi Joel,
>> >>
>> >> Thanks for the review and I plan to update the KIP today with all the
>> >> updated info. My comments in line below.
>> >>
>> >> Thanks
>> >> Parth
>> >>
>> >>
>> >> On 4/20/15, 10:07 AM, "Joel Koshy" <jjkoshy.w@gmail.com<mailto:
>> >> jjkoshy.w@gmail.com>> wrote:
>> >>
>> >> Hi Parth,
>> >>
>> >> Nice work on this KIP.  I did another read through and had a few more
>> >> comments (with edits after I went through the thread). Many of these
>> >> comments were brought up by others as well, so it appears that the
>>KIP
>> >> would benefit from an update at this point to incorporate comments
>> >> from the thread and last hangout.
>> >>
>> >> - The operation enum is mostly self-explanatory, but it would help
>> >>   (for the sake of clarity and completeness if nothing else) to
>> >>   document exactly what each of the enums are. E.g., I think this
>>came
>> >>   up in our hangout - SEND_CONTROL_MESSAGE is unclear and I don't
>> >>   remember what was said about it. <Edit>: After going through the
>> >>   thread it seems the conclusion was to categorize operations. E.g.,
>> >>   WRITE could apply to multiple requests. Again, this is unclear, so
>> >>   if it would be great if you could update the KIP to clarify what
>>you
>> >>   intend.
>> >>
>> >> Will add to document. SEND_CONTROL_MESSAGE Probably a very bad name
>>but
>> >> these are intra borker API calls like controller notifying other
>> >>brokers to
>> >> update metadata or heartbeats. Any better naming suggestions?
>> >>
>> >> - When you update the KIP to categorize the requests it would also
>> >>   help to have a column for what the resource is for each.
>> >>
>> >> Will add to the KIP.
>> >>
>> >> - FWIW I prefer a 1-1 mapping between requests and operations. I
>>think
>> >>   categorizing requests into these can be confusing because:
>> >>   - The resource being protected for different requests will be
>> >>     different. We are mostly thinking about topics (read/write) but
>> >>     there are requests for which topic is not the right resource.
>> >>     E.g., for topic creation, the resource as you suggested would be
>> >>     something global/common such as “cluster”. For
>> >>     OffsetCommit/FetchRequest, the resource may be the consumer
>>group,
>> >>     or maybe a tuple of <consumer group, topic>. So this can be
>> >>     confusing - i.e., different resources and request types in the
>> >>     same category. It may be simpler and clearer to just have a 1-1
>> >>     mapping between the operation enum and requests.
>> >>
>> >> I only see 2 resource categories right now cluster and topic.  I
>>don’t
>> >> really care one way or another so we can probably make a quick
>>decision
>> >>in
>> >> tomorrow’s meeting to either to 1-1 mapping or have categorization?
>> >>
>> >>   - Some requests that are intuitively READ have WRITE side-effects.
>> >>     E.g., (currently) TopicMetadataRequest with auto-create, although
>> >>     that will eventually go away. ConsumerMetadataRequest still
>> >>     auto-creates the offsets topic. Likewise, ADMIN-type requests may
>> >>     be interpreted as having side-effects (depending on who you ask).
>> >>
>> >> Yes and what I am doing right now is checking authorization for all
>> >> possible actions i.e. for auto-create it checks if the config has it
>> >> enabled and if yes, check for read + create authorization. Its not
>>very
>> >> meaningful right now as there is no CREATE authorization but I think
>> >>this
>> >> is implementation detail, we need to ensure we call authorize with
>>all
>> >> possible operations from KafkaAPI.
>> >> - <quote>When an ACL is missing - fail open</quote>. What does
>>missing
>> >>   mean? i.e., no explicit ACL for a principal? I'm confused by this
>> >>   especially in relation to the precedence of DENY over ALLOW. So per
>> >>   the description:
>> >>   - If no ACLs exist for topic A then ALLOW all operations on it by
>> >>     anyone.
>> >>   - If I now add an ACL for a certain principal P to ALLOW (say)
>>WRITE
>> >>     to the topic then either:
>> >>   - This has the effect of DENYing WRITE to all other principals
>> >>   - Or, this ACL serves no purpose
>> >>   - If the effect is to DENY WRITE to all other principals, what
>>about
>> >>     READ. Do all principals (including P) have READ permissions to
>> >>     topic A?
>> >>   - In other words, it seems for a specific ACL to be meaningful then
>> >>     fail close is necessary for an absent ACL.
>> >>   - <edit>After through the thread: it appears that the DENY override
>> >>     only applies to the given principal. i.e., in the above case it
>> >>     appears that the other principals will in fact be granted access.
>> >>     Then this makes the ACL that was added pointless right?
>> >>
>> >> The rule I was going with is
>> >> - If there is no ACL I.e. This might be a topic that was created in
>>non
>> >> secure mode or was created before we supported ACLs. We assume you do
>> >>not
>> >> want authorization and let all requests go through.
>> >> - once you add any ACL, we assume you want authorization on the topic
>> >>and
>> >> all the general authorization rules now start to apply, I.e we fail
>> >>close
>> >> if we don’t find an ACL that allows access or if we find an ACL that
>> >>denies
>> >> access. It does not matter if you added a READACL or WRITEACL or
>> >>ALLOWACL
>> >> or DENY ACL. If you add any ACL, now every user gets checked against
>> >>that
>> >> and if it does not satisfy the ACL, request fails. I.e. If you add an
>> >>ACL
>> >> “Allow write to topic-1 form user1 from all hosts” , user-1 has write
>> >> access from all hosts and no other user has any access(except for
>> >> superusers who have all the access).
>> >> - Deny ACLS are suppose to be used to restrict access authorized by
>>some
>> >> allow ACL, they are not suppose to be required. Implicitly anyone who
>> >>does
>> >> not have an allow acl, gets denied. The Deny ACLs are only added to
>>give
>> >> more control to administrators who wants more granular control with
>> >>lesser
>> >> config. The scenario described in mailing list was “Allow user X
>>access
>> >> from all hosts but Host1,Host2”. in absence of DENY operator you will
>> >>have
>> >> to exhaustively list all possible hosts in your ACL which is what we
>>are
>> >> trying to avoid.
>> >>
>> >> - On ZK ACLs: I think ZK will be closed to everyone except Kafka
>> >>   brokers. This is a dependency on KIP-4 though. i.e., eventually all
>> >>   clients should talk to brokers only via RPC.
>> >>
>> >> Yes.
>> >>
>> >> - Topic owner: list vs single entry - both have issues off the bat
>> >>   (although list is more intuitive at least to me), but perhaps you
>> >>   could write up some example workflows to clarify the current
>> >>   proposal. I was thinking that anyone in the owner list should be
>> >>   considered a super-user of the topic and can grant/revoke
>> >>   permissions. They should also be allowed to add other principals as
>> >>   owners. Even with this it is unclear who should be allowed to
>>remove
>> >>   owners.
>> >>
>> >> As you pointed out in the last KIP meeting owners/creators have use
>>out
>> >> side of security context (plain simple auditing). I don’t think the
>> >> authorizer work depends on this, it was my bad to even mention it in
>> >>first
>> >> place. I think we can have this discussion outside of
>> >>authorizer/security
>> >> context and once we have a way to get topic owners the default
>> >>Authorizer
>> >> can start using it. It makes sense to treat all owners as super users
>> >>and I
>> >> think it is safe to assume superusers can also modify ownership but I
>> >>think
>> >> this should not be treated as blocking work for authorization.
>> >>
>> >> - What is the effect of deleting a topic - should all associated ACLs
>> >>   be deleted as well?
>> >> They should be and with acls being stored as part of TopicConfig this
>> >>was
>> >> taken care of automatically. With the new ACL management API the
>>users
>> >>will
>> >> have to call remove ACLs explicitly to perform the cleanup. If
>>everyone
>> >> thinks this should be automated , with the new APIs we will need a
>> >>hook(or
>> >> poll) to be notified when a topic is deleted to perform cleanup.
>> >> - TopicConfigCache to store topic-ACLs. As mentioned above, not all
>> >>   requests will be tied to topics. We may want to have an entirely
>> >>   separate ZK directory for ACLs. We have a similar issue with
>>quotas.
>> >>   This ties in with dynamic config management. We can certainly
>> >>   leverage the dynamic config management part of topic configs but I
>> >>   think we need to have a story for non-topic resources.
>> >>
>> >> In the first proposal I was going with a topic-Acl and cluster-Acl
>>where
>> >> cluster-Acls were json acl local files on all brokers. With the new
>>ACL
>> >> management APIs we are planning to have /kafka-acl node under which
>>all
>> >> acls will be stored in /kakfa-acls/resource-name -> {acl json data}.
>> >> Cluster acls will just have resource name kafka-cluster.
>> >>
>> >> Thanks,
>> >>
>> >> Joel
>> >>
>> >> On Thu, Apr 16, 2015 at 12:15:37AM +0000, Parth Brahmbhatt wrote:
>> >> Kafka currently stores logConfig overrides specified during topic
>> >>creation
>> >> in zookeeper, its just an instance of java.util.Properties converted
>>to
>> >> json. I am proposing in addition to that we store acls and owner as
>>well
>> >> as part of same Properties map.
>> >> There is some infrastructure around reading this config, converting
>>it
>> >> back to Properties map and most importantly propagating any changes
>> >> efficiently which we will be able to leverage. As this
>>infrastructure is
>> >> common to the cluster the reading (not interpreting) of config
>>happens
>> >> outside of any authorization code.
>> >> If the TopicConfigCache just kept the json representation and left
>>it to
>> >> authorizer to parse it, the authorizer will have to either parse the
>> >>json
>> >> for each request(not acceptable) or it will have to keep one more
>>layer
>> >>of
>> >> parsed ACL instance cache. Assuming authorizer will keep an
>>additional
>> >> caching layer we will now have to implement some way to invalidate
>>the
>> >> cache which means the TopicConfigCache will have to be an observable
>> >>which
>> >> the Authorizer observes and invalidates its cache entries when
>> >> topicConfigCache gets updated. Seemed like unnecessary complexity
>>with
>> >>not
>> >> lot to gain so I went with TopicConfigCache interpreting the json and
>> >> caching a higher level modeled object.
>> >> In summary, the interpretation is done for both optimization and
>> >> simplicity. If you think it is important to allow custom ACL format
>> >> support we can add one more pluggable config(acl.parser) and
>> >> interface(AclParser) or it could just be another method in
>>Authorizer.
>> >> One thing to note the current ACL json is versioned so it is easy to
>> >>make
>> >> changes to it however it won’t be possible to support custom ACL
>>formats
>> >> with the current design.
>> >> Thanks
>> >> Parth
>> >> On 4/15/15, 4:29 PM, "Michael Herstine"
>><mherstine@linkedin.com.INVALID
>> >> <ma...@linkedin.com.INVALID>>
>> >> wrote:
>> >> >Hi Parth,
>> >> >
>> >> >I’m a little confused: why would Kafka need to interpret the JSON?
>> >>IIRC
>> >> >KIP-11 even says that the TopicConfigData will just store the JSON.
>>I’m
>> >> >not really making a design recommendation here, just trying to
>> >>understand
>> >> >what you’re proposing.
>> >> >
>> >> >On 4/15/15, 11:20 AM, "Parth Brahmbhatt"
>><pbrahmbhatt@hortonworks.com
>> >> <ma...@hortonworks.com>>
>> >> >wrote:
>> >> >
>> >> >>Hi Michael,
>> >> >>
>> >> >>There is code in kafka codebase that reads and interprets the topic
>> >> >>config JSON which has acls, owner and logconfig properties. There
>>are
>> >>3
>> >> >>use cases that we are supporting with current proposal:
>> >> >>
>> >> >>  *   You use out of box simpleAcl authorizer which is tied to the
>>acl
>> >> >>stored in topic config and the format is locked down.
>> >> >>  *   You have a custom authorizer and a custom ACL store.
>> >>Ranger/Argus
>> >> >>falls under this as they have their own acl store and ui that users
>> >>use
>> >> >>to configure acls on the cluster and cluster resources  like topic.
>> >>It is
>> >> >>upto the custom authorizer to leverage the kafka acl configs or
>> >> >>completely ignore them as they have set a user expectation that
>>only
>> >>acls
>> >> >>configured via their ui/system will be effective.
>> >> >>  *   You have a custom authorizer but no custom Acl store. You are
>> >> >>completely tied to Acl structure that we have provided in out of
>>box
>> >> >>implementation.
>> >> >>
>> >> >>Thanks
>> >> >>Parth
>> >> >>
>> >> >>On 4/15/15, 10:31 AM, "Michael Herstine"
>> >> 
>>>><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
>> >> ><ma...@linkedin.com.INVALID>>
>> >> >>wrote:
>> >> >>
>> >> >>Hi Parth,
>> >> >>
>> >> >>One question that occurred to me at the end of today’s hangout: how
>> >>tied
>> >> >>are we to a particular ACL representation under your proposal? I
>>know
>> >> >>that
>> >> >>TopicConfigCache will just contain JSON— if a particular site
>>decides
>> >> >>they
>> >> >>want to represent their ACLs differently, and swap out the
>>authorizer
>> >> >>implementation, will that work?  I guess what I’m asking is whether
>> >> >>there’s any code in the Kafka codebase that will interpret that
>>JSON,
>> >>or
>> >> >>does that logic live exclusively in the authorizer?
>> >> >>
>> >> >>On 4/14/15, 10:56 PM, "Don Bosco Durai"
>> >> 
>>>><bo...@apache.org>>
>> >> wrote:
>> >> >>
>> >> >>I also feel, having just IP would be more appropriate. Host lookup
>> >>will
>> >> >>unnecessary slow things down and would be insecure as you pointed
>>out.
>> >> >>
>> >> >>With IP, it will be also able to setup policies (in future if
>>needed)
>> >> >>with
>> >> >>ranges or netmasks and it would be more scalable.
>> >> >>
>> >> >>Bosco
>> >> >>
>> >> >>
>> >> >>On 4/14/15, 1:40 PM, "Michael Herstine"
>> >> 
>>>><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
>> >> ><ma...@linkedin.com.INVALID>>
>> >> >>wrote:
>> >> >>
>> >> >>Hi Parth,
>> >> >>
>> >> >>Sorry to chime in so late, but I’ve got a minor question on the
>>KIP.
>> >> >>
>> >> >>Several methods take a parameter named “host” of type String. Is
>>that
>> >> >>intended to be a hostname, or an IP address? If the former, I’m
>> >>curious
>> >> >>as
>> >> >>to how that’s found (in my experience, when accepting an incoming
>> >>socket
>> >> >>connection, you only know the IP address, and there isn’t a way to
>>map
>> >> >>that to a hostname without a round trip to a DNS server, which is
>> >> >>insecure
>> >> >>anyway).
>> >> >>
>> >> >>
>> >> >>On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
>> >>
>> >>>><pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
>> ><mailto
>> >>>>:
>> >> pbrahmbhatt@hortonworks.com>>
>> >> >>wrote:
>> >> >>
>> >> >>Hi all,
>> >> >>
>> >> >>I have modified the KIP to reflect the recent change request from
>>the
>> >> >>reviewers. I have been working on the code and I have the server
>>side
>> >> >>code
>> >> >>for authorization ready. I am now modifying the command line
>> >>utilities.
>> >> >>I
>> >> >>would really appreciate if some of the committers can spend
>>sometime
>> >>to
>> >> >>review the KIP so we can make progress on this.
>> >> >>
>> >> >>Thanks
>> >> >>Parth
>> >> >>
>> >> >>On 3/18/15, 2:20 PM, "Michael Herstine"
>> >> 
>>>><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
>> >> ><ma...@linkedin.com.INVALID>>
>> >> >>wrote:
>> >> >>
>> >> >>Hi Parth,
>> >> >>
>> >> >>Thanks! A few questions:
>> >> >>
>> >> >>1. Do you want to permit rules in your ACLs that DENY access as
>>well
>> >>as
>> >> >>ALLOW? This can be handy setting up rules that have exceptions.
>>E.g.
>> >> >>“Allow principal P to READ resource R from all hosts” with “Deny
>> >> >>principal
>> >> >>P READ access to resource R from host H1” in combination would
>>allow P
>> >> >>to
>> >> >>READ R from all hosts *except* H1.
>> >> >>
>> >> >>2. When a topic is newly created, will there be an ACL created for
>>it?
>> >> >>If
>> >> >>not, would that not deny subsequent access to it?
>> >> >>
>> >> >>(nit) Maybe use Principal instead of String to represent
>>principals?
>> >> >>
>> >> >>
>> >> >>On 3/9/15, 11:48 AM, "Don Bosco Durai"
>> >> 
>>>><bo...@apache.org>>
>> >> wrote:
>> >> >>
>> >> >>Parth
>> >> >>
>> >> >>Overall it is looking good. Couple of questionsŠ
>> >> >>
>> >> >>- Can you give an example how the policies will look like in the
>> >> >>default
>> >> >>implementation?
>> >> >>- In the operations, can we support ³CONNECT² also? This can be
>>used
>> >> >>during Session connection
>> >> >>- Regarding access control for ³Topic Creation², since we can¹t do
>>it
>> >> >>on
>> >> >>the server side, can we de-scope it for? And plan it as a future
>> >> >>feature
>> >> >>request?
>> >> >>
>> >> >>Thanks
>> >> >>
>> >> >>Bosco
>> >> >>
>> >> >>
>> >> >>On 3/6/15, 8:10 AM, "Harsha"
>><kafka@harsha.io<mailto:kafka@harsha.io
>> >> ><ma...@harsha.io>>
>> >> >>wrote:
>> >> >>
>> >> >>Hi Parth,
>> >> >>            Thanks for putting this together. Overall it looks good
>> >> >>to
>> >> >>            me. Although AdminUtils is a concern KIP-4  can
>>probably
>> >> >>fix
>> >> >>            that part.
>> >> >>Thanks,
>> >> >>Harsha
>> >> >>
>> >> >>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>> >> >>Forgot to add links to wiki and jira.
>> >> >>Link to wiki:
>> >> 
>>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
>> >> >>t
>> >> >>i
>> >> >>o
>> >> >>n
>> >> >>+
>> >> >>Interface
>> >> >>Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>> >> >>Thanks
>> >> >>Parth
>> >> >>From: Parth Brahmbhatt
>> >>
>> >>>><pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
>> ><mailto
>> >>>>:
>> >> pbrahmbhatt@hortonworks.com><mailto:p
>> >> >>b
>> >> >>rahmbhatt@hortonworks.com<ma...@hortonworks.com>>>
>> >> >>Date: Thursday, March 5, 2015 at 10:33 AM
>> >> >>To:
>> >> >>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:
>> >> dev@kafka.apache.org><mailto:dev@kafka.apach
>> >> >>e
>> >> >>.org>"
>> >> >><de...@kafka.apache.org><mailto:
>> >> dev@kafka.apache.org><mailto:dev@kafka.apach
>> >> >>e
>> >> >>.org>>
>> >> >>Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>> >> >>Hi,
>> >> >>KIP-11 is open for discussion , I have updated the wiki with the
>> >> >>design
>> >> >>and open questions.
>> >> >>Thanks
>> >> >>Parth
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >
>> >>
>> >>
>> >>
>>
>>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Jun Rao <ju...@confluent.io>]

Hmm, I thought the semantics is that if you only have rule "deny user2", it
means that everyone except user2 has access?

Thanks,

Jun

On Mon, Apr 20, 2015 at 3:25 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:

> user3 does not have access and removing the deny rule does not grant him
> or user2 access. user2 even without the deny rule will not have access.
>
> Thanks
> Parth
>
> On 4/20/15, 12:03 PM, "Jun Rao" <ju...@confluent.io> wrote:
>
> >Just a followup question. Suppose there are two rules. Rule1 allows user1
> >and rule2 denies user2. Does user3 have access? If not, does removing
> >rule1
> >enable user3 access?
> >
> >Thanks,
> >
> >Jun
> >
> >On Mon, Apr 20, 2015 at 1:34 PM, Parth Brahmbhatt <
> >pbrahmbhatt@hortonworks.com> wrote:
> >
> >>
> >> Hi Joel,
> >>
> >> Thanks for the review and I plan to update the KIP today with all the
> >> updated info. My comments in line below.
> >>
> >> Thanks
> >> Parth
> >>
> >>
> >> On 4/20/15, 10:07 AM, "Joel Koshy" <jjkoshy.w@gmail.com<mailto:
> >> jjkoshy.w@gmail.com>> wrote:
> >>
> >> Hi Parth,
> >>
> >> Nice work on this KIP.  I did another read through and had a few more
> >> comments (with edits after I went through the thread). Many of these
> >> comments were brought up by others as well, so it appears that the KIP
> >> would benefit from an update at this point to incorporate comments
> >> from the thread and last hangout.
> >>
> >> - The operation enum is mostly self-explanatory, but it would help
> >>   (for the sake of clarity and completeness if nothing else) to
> >>   document exactly what each of the enums are. E.g., I think this came
> >>   up in our hangout - SEND_CONTROL_MESSAGE is unclear and I don't
> >>   remember what was said about it. <Edit>: After going through the
> >>   thread it seems the conclusion was to categorize operations. E.g.,
> >>   WRITE could apply to multiple requests. Again, this is unclear, so
> >>   if it would be great if you could update the KIP to clarify what you
> >>   intend.
> >>
> >> Will add to document. SEND_CONTROL_MESSAGE Probably a very bad name but
> >> these are intra borker API calls like controller notifying other
> >>brokers to
> >> update metadata or heartbeats. Any better naming suggestions?
> >>
> >> - When you update the KIP to categorize the requests it would also
> >>   help to have a column for what the resource is for each.
> >>
> >> Will add to the KIP.
> >>
> >> - FWIW I prefer a 1-1 mapping between requests and operations. I think
> >>   categorizing requests into these can be confusing because:
> >>   - The resource being protected for different requests will be
> >>     different. We are mostly thinking about topics (read/write) but
> >>     there are requests for which topic is not the right resource.
> >>     E.g., for topic creation, the resource as you suggested would be
> >>     something global/common such as “cluster”. For
> >>     OffsetCommit/FetchRequest, the resource may be the consumer group,
> >>     or maybe a tuple of <consumer group, topic>. So this can be
> >>     confusing - i.e., different resources and request types in the
> >>     same category. It may be simpler and clearer to just have a 1-1
> >>     mapping between the operation enum and requests.
> >>
> >> I only see 2 resource categories right now cluster and topic.  I don’t
> >> really care one way or another so we can probably make a quick decision
> >>in
> >> tomorrow’s meeting to either to 1-1 mapping or have categorization?
> >>
> >>   - Some requests that are intuitively READ have WRITE side-effects.
> >>     E.g., (currently) TopicMetadataRequest with auto-create, although
> >>     that will eventually go away. ConsumerMetadataRequest still
> >>     auto-creates the offsets topic. Likewise, ADMIN-type requests may
> >>     be interpreted as having side-effects (depending on who you ask).
> >>
> >> Yes and what I am doing right now is checking authorization for all
> >> possible actions i.e. for auto-create it checks if the config has it
> >> enabled and if yes, check for read + create authorization. Its not very
> >> meaningful right now as there is no CREATE authorization but I think
> >>this
> >> is implementation detail, we need to ensure we call authorize with all
> >> possible operations from KafkaAPI.
> >> - <quote>When an ACL is missing - fail open</quote>. What does missing
> >>   mean? i.e., no explicit ACL for a principal? I'm confused by this
> >>   especially in relation to the precedence of DENY over ALLOW. So per
> >>   the description:
> >>   - If no ACLs exist for topic A then ALLOW all operations on it by
> >>     anyone.
> >>   - If I now add an ACL for a certain principal P to ALLOW (say) WRITE
> >>     to the topic then either:
> >>   - This has the effect of DENYing WRITE to all other principals
> >>   - Or, this ACL serves no purpose
> >>   - If the effect is to DENY WRITE to all other principals, what about
> >>     READ. Do all principals (including P) have READ permissions to
> >>     topic A?
> >>   - In other words, it seems for a specific ACL to be meaningful then
> >>     fail close is necessary for an absent ACL.
> >>   - <edit>After through the thread: it appears that the DENY override
> >>     only applies to the given principal. i.e., in the above case it
> >>     appears that the other principals will in fact be granted access.
> >>     Then this makes the ACL that was added pointless right?
> >>
> >> The rule I was going with is
> >> - If there is no ACL I.e. This might be a topic that was created in non
> >> secure mode or was created before we supported ACLs. We assume you do
> >>not
> >> want authorization and let all requests go through.
> >> - once you add any ACL, we assume you want authorization on the topic
> >>and
> >> all the general authorization rules now start to apply, I.e we fail
> >>close
> >> if we don’t find an ACL that allows access or if we find an ACL that
> >>denies
> >> access. It does not matter if you added a READACL or WRITEACL or
> >>ALLOWACL
> >> or DENY ACL. If you add any ACL, now every user gets checked against
> >>that
> >> and if it does not satisfy the ACL, request fails. I.e. If you add an
> >>ACL
> >> “Allow write to topic-1 form user1 from all hosts” , user-1 has write
> >> access from all hosts and no other user has any access(except for
> >> superusers who have all the access).
> >> - Deny ACLS are suppose to be used to restrict access authorized by some
> >> allow ACL, they are not suppose to be required. Implicitly anyone who
> >>does
> >> not have an allow acl, gets denied. The Deny ACLs are only added to give
> >> more control to administrators who wants more granular control with
> >>lesser
> >> config. The scenario described in mailing list was “Allow user X access
> >> from all hosts but Host1,Host2”. in absence of DENY operator you will
> >>have
> >> to exhaustively list all possible hosts in your ACL which is what we are
> >> trying to avoid.
> >>
> >> - On ZK ACLs: I think ZK will be closed to everyone except Kafka
> >>   brokers. This is a dependency on KIP-4 though. i.e., eventually all
> >>   clients should talk to brokers only via RPC.
> >>
> >> Yes.
> >>
> >> - Topic owner: list vs single entry - both have issues off the bat
> >>   (although list is more intuitive at least to me), but perhaps you
> >>   could write up some example workflows to clarify the current
> >>   proposal. I was thinking that anyone in the owner list should be
> >>   considered a super-user of the topic and can grant/revoke
> >>   permissions. They should also be allowed to add other principals as
> >>   owners. Even with this it is unclear who should be allowed to remove
> >>   owners.
> >>
> >> As you pointed out in the last KIP meeting owners/creators have use out
> >> side of security context (plain simple auditing). I don’t think the
> >> authorizer work depends on this, it was my bad to even mention it in
> >>first
> >> place. I think we can have this discussion outside of
> >>authorizer/security
> >> context and once we have a way to get topic owners the default
> >>Authorizer
> >> can start using it. It makes sense to treat all owners as super users
> >>and I
> >> think it is safe to assume superusers can also modify ownership but I
> >>think
> >> this should not be treated as blocking work for authorization.
> >>
> >> - What is the effect of deleting a topic - should all associated ACLs
> >>   be deleted as well?
> >> They should be and with acls being stored as part of TopicConfig this
> >>was
> >> taken care of automatically. With the new ACL management API the users
> >>will
> >> have to call remove ACLs explicitly to perform the cleanup. If everyone
> >> thinks this should be automated , with the new APIs we will need a
> >>hook(or
> >> poll) to be notified when a topic is deleted to perform cleanup.
> >> - TopicConfigCache to store topic-ACLs. As mentioned above, not all
> >>   requests will be tied to topics. We may want to have an entirely
> >>   separate ZK directory for ACLs. We have a similar issue with quotas.
> >>   This ties in with dynamic config management. We can certainly
> >>   leverage the dynamic config management part of topic configs but I
> >>   think we need to have a story for non-topic resources.
> >>
> >> In the first proposal I was going with a topic-Acl and cluster-Acl where
> >> cluster-Acls were json acl local files on all brokers. With the new ACL
> >> management APIs we are planning to have /kafka-acl node under which all
> >> acls will be stored in /kakfa-acls/resource-name -> {acl json data}.
> >> Cluster acls will just have resource name kafka-cluster.
> >>
> >> Thanks,
> >>
> >> Joel
> >>
> >> On Thu, Apr 16, 2015 at 12:15:37AM +0000, Parth Brahmbhatt wrote:
> >> Kafka currently stores logConfig overrides specified during topic
> >>creation
> >> in zookeeper, its just an instance of java.util.Properties converted to
> >> json. I am proposing in addition to that we store acls and owner as well
> >> as part of same Properties map.
> >> There is some infrastructure around reading this config, converting it
> >> back to Properties map and most importantly propagating any changes
> >> efficiently which we will be able to leverage. As this infrastructure is
> >> common to the cluster the reading (not interpreting) of config happens
> >> outside of any authorization code.
> >> If the TopicConfigCache just kept the json representation and left it to
> >> authorizer to parse it, the authorizer will have to either parse the
> >>json
> >> for each request(not acceptable) or it will have to keep one more layer
> >>of
> >> parsed ACL instance cache. Assuming authorizer will keep an additional
> >> caching layer we will now have to implement some way to invalidate the
> >> cache which means the TopicConfigCache will have to be an observable
> >>which
> >> the Authorizer observes and invalidates its cache entries when
> >> topicConfigCache gets updated. Seemed like unnecessary complexity with
> >>not
> >> lot to gain so I went with TopicConfigCache interpreting the json and
> >> caching a higher level modeled object.
> >> In summary, the interpretation is done for both optimization and
> >> simplicity. If you think it is important to allow custom ACL format
> >> support we can add one more pluggable config(acl.parser) and
> >> interface(AclParser) or it could just be another method in Authorizer.
> >> One thing to note the current ACL json is versioned so it is easy to
> >>make
> >> changes to it however it won’t be possible to support custom ACL formats
> >> with the current design.
> >> Thanks
> >> Parth
> >> On 4/15/15, 4:29 PM, "Michael Herstine" <mherstine@linkedin.com.INVALID
> >> <ma...@linkedin.com.INVALID>>
> >> wrote:
> >> >Hi Parth,
> >> >
> >> >I’m a little confused: why would Kafka need to interpret the JSON?
> >>IIRC
> >> >KIP-11 even says that the TopicConfigData will just store the JSON. I’m
> >> >not really making a design recommendation here, just trying to
> >>understand
> >> >what you’re proposing.
> >> >
> >> >On 4/15/15, 11:20 AM, "Parth Brahmbhatt" <pbrahmbhatt@hortonworks.com
> >> <ma...@hortonworks.com>>
> >> >wrote:
> >> >
> >> >>Hi Michael,
> >> >>
> >> >>There is code in kafka codebase that reads and interprets the topic
> >> >>config JSON which has acls, owner and logconfig properties. There are
> >>3
> >> >>use cases that we are supporting with current proposal:
> >> >>
> >> >>  *   You use out of box simpleAcl authorizer which is tied to the acl
> >> >>stored in topic config and the format is locked down.
> >> >>  *   You have a custom authorizer and a custom ACL store.
> >>Ranger/Argus
> >> >>falls under this as they have their own acl store and ui that users
> >>use
> >> >>to configure acls on the cluster and cluster resources  like topic.
> >>It is
> >> >>upto the custom authorizer to leverage the kafka acl configs or
> >> >>completely ignore them as they have set a user expectation that only
> >>acls
> >> >>configured via their ui/system will be effective.
> >> >>  *   You have a custom authorizer but no custom Acl store. You are
> >> >>completely tied to Acl structure that we have provided in out of box
> >> >>implementation.
> >> >>
> >> >>Thanks
> >> >>Parth
> >> >>
> >> >>On 4/15/15, 10:31 AM, "Michael Herstine"
> >> >><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
> >> ><ma...@linkedin.com.INVALID>>
> >> >>wrote:
> >> >>
> >> >>Hi Parth,
> >> >>
> >> >>One question that occurred to me at the end of today’s hangout: how
> >>tied
> >> >>are we to a particular ACL representation under your proposal? I know
> >> >>that
> >> >>TopicConfigCache will just contain JSON— if a particular site decides
> >> >>they
> >> >>want to represent their ACLs differently, and swap out the authorizer
> >> >>implementation, will that work?  I guess what I’m asking is whether
> >> >>there’s any code in the Kafka codebase that will interpret that JSON,
> >>or
> >> >>does that logic live exclusively in the authorizer?
> >> >>
> >> >>On 4/14/15, 10:56 PM, "Don Bosco Durai"
> >> >><bo...@apache.org>>
> >> wrote:
> >> >>
> >> >>I also feel, having just IP would be more appropriate. Host lookup
> >>will
> >> >>unnecessary slow things down and would be insecure as you pointed out.
> >> >>
> >> >>With IP, it will be also able to setup policies (in future if needed)
> >> >>with
> >> >>ranges or netmasks and it would be more scalable.
> >> >>
> >> >>Bosco
> >> >>
> >> >>
> >> >>On 4/14/15, 1:40 PM, "Michael Herstine"
> >> >><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
> >> ><ma...@linkedin.com.INVALID>>
> >> >>wrote:
> >> >>
> >> >>Hi Parth,
> >> >>
> >> >>Sorry to chime in so late, but I’ve got a minor question on the KIP.
> >> >>
> >> >>Several methods take a parameter named “host” of type String. Is that
> >> >>intended to be a hostname, or an IP address? If the former, I’m
> >>curious
> >> >>as
> >> >>to how that’s found (in my experience, when accepting an incoming
> >>socket
> >> >>connection, you only know the IP address, and there isn’t a way to map
> >> >>that to a hostname without a round trip to a DNS server, which is
> >> >>insecure
> >> >>anyway).
> >> >>
> >> >>
> >> >>On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
> >>
> >>>><pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
> ><mailto
> >>>>:
> >> pbrahmbhatt@hortonworks.com>>
> >> >>wrote:
> >> >>
> >> >>Hi all,
> >> >>
> >> >>I have modified the KIP to reflect the recent change request from the
> >> >>reviewers. I have been working on the code and I have the server side
> >> >>code
> >> >>for authorization ready. I am now modifying the command line
> >>utilities.
> >> >>I
> >> >>would really appreciate if some of the committers can spend sometime
> >>to
> >> >>review the KIP so we can make progress on this.
> >> >>
> >> >>Thanks
> >> >>Parth
> >> >>
> >> >>On 3/18/15, 2:20 PM, "Michael Herstine"
> >> >><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
> >> ><ma...@linkedin.com.INVALID>>
> >> >>wrote:
> >> >>
> >> >>Hi Parth,
> >> >>
> >> >>Thanks! A few questions:
> >> >>
> >> >>1. Do you want to permit rules in your ACLs that DENY access as well
> >>as
> >> >>ALLOW? This can be handy setting up rules that have exceptions. E.g.
> >> >>“Allow principal P to READ resource R from all hosts” with “Deny
> >> >>principal
> >> >>P READ access to resource R from host H1” in combination would allow P
> >> >>to
> >> >>READ R from all hosts *except* H1.
> >> >>
> >> >>2. When a topic is newly created, will there be an ACL created for it?
> >> >>If
> >> >>not, would that not deny subsequent access to it?
> >> >>
> >> >>(nit) Maybe use Principal instead of String to represent principals?
> >> >>
> >> >>
> >> >>On 3/9/15, 11:48 AM, "Don Bosco Durai"
> >> >><bo...@apache.org>>
> >> wrote:
> >> >>
> >> >>Parth
> >> >>
> >> >>Overall it is looking good. Couple of questionsŠ
> >> >>
> >> >>- Can you give an example how the policies will look like in the
> >> >>default
> >> >>implementation?
> >> >>- In the operations, can we support ³CONNECT² also? This can be used
> >> >>during Session connection
> >> >>- Regarding access control for ³Topic Creation², since we can¹t do it
> >> >>on
> >> >>the server side, can we de-scope it for? And plan it as a future
> >> >>feature
> >> >>request?
> >> >>
> >> >>Thanks
> >> >>
> >> >>Bosco
> >> >>
> >> >>
> >> >>On 3/6/15, 8:10 AM, "Harsha" <kafka@harsha.io<mailto:kafka@harsha.io
> >> ><ma...@harsha.io>>
> >> >>wrote:
> >> >>
> >> >>Hi Parth,
> >> >>            Thanks for putting this together. Overall it looks good
> >> >>to
> >> >>            me. Although AdminUtils is a concern KIP-4  can probably
> >> >>fix
> >> >>            that part.
> >> >>Thanks,
> >> >>Harsha
> >> >>
> >> >>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
> >> >>Forgot to add links to wiki and jira.
> >> >>Link to wiki:
> >> >>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
> >> >>t
> >> >>i
> >> >>o
> >> >>n
> >> >>+
> >> >>Interface
> >> >>Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
> >> >>Thanks
> >> >>Parth
> >> >>From: Parth Brahmbhatt
> >>
> >>>><pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
> ><mailto
> >>>>:
> >> pbrahmbhatt@hortonworks.com><mailto:p
> >> >>b
> >> >>rahmbhatt@hortonworks.com<ma...@hortonworks.com>>>
> >> >>Date: Thursday, March 5, 2015 at 10:33 AM
> >> >>To:
> >> >>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:
> >> dev@kafka.apache.org><mailto:dev@kafka.apach
> >> >>e
> >> >>.org>"
> >> >><de...@kafka.apache.org><mailto:
> >> dev@kafka.apache.org><mailto:dev@kafka.apach
> >> >>e
> >> >>.org>>
> >> >>Subject: [DISCUSS] KIP-11- Authorization design for kafka security
> >> >>Hi,
> >> >>KIP-11 is open for discussion , I have updated the wiki with the
> >> >>design
> >> >>and open questions.
> >> >>Thanks
> >> >>Parth
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >
> >>
> >>
> >>
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

user3 does not have access and removing the deny rule does not grant him
or user2 access. user2 even without the deny rule will not have access.

Thanks
Parth

On 4/20/15, 12:03 PM, "Jun Rao" <ju...@confluent.io> wrote:

>Just a followup question. Suppose there are two rules. Rule1 allows user1
>and rule2 denies user2. Does user3 have access? If not, does removing
>rule1
>enable user3 access?
>
>Thanks,
>
>Jun
>
>On Mon, Apr 20, 2015 at 1:34 PM, Parth Brahmbhatt <
>pbrahmbhatt@hortonworks.com> wrote:
>
>>
>> Hi Joel,
>>
>> Thanks for the review and I plan to update the KIP today with all the
>> updated info. My comments in line below.
>>
>> Thanks
>> Parth
>>
>>
>> On 4/20/15, 10:07 AM, "Joel Koshy" <jjkoshy.w@gmail.com<mailto:
>> jjkoshy.w@gmail.com>> wrote:
>>
>> Hi Parth,
>>
>> Nice work on this KIP.  I did another read through and had a few more
>> comments (with edits after I went through the thread). Many of these
>> comments were brought up by others as well, so it appears that the KIP
>> would benefit from an update at this point to incorporate comments
>> from the thread and last hangout.
>>
>> - The operation enum is mostly self-explanatory, but it would help
>>   (for the sake of clarity and completeness if nothing else) to
>>   document exactly what each of the enums are. E.g., I think this came
>>   up in our hangout - SEND_CONTROL_MESSAGE is unclear and I don't
>>   remember what was said about it. <Edit>: After going through the
>>   thread it seems the conclusion was to categorize operations. E.g.,
>>   WRITE could apply to multiple requests. Again, this is unclear, so
>>   if it would be great if you could update the KIP to clarify what you
>>   intend.
>>
>> Will add to document. SEND_CONTROL_MESSAGE Probably a very bad name but
>> these are intra borker API calls like controller notifying other
>>brokers to
>> update metadata or heartbeats. Any better naming suggestions?
>>
>> - When you update the KIP to categorize the requests it would also
>>   help to have a column for what the resource is for each.
>>
>> Will add to the KIP.
>>
>> - FWIW I prefer a 1-1 mapping between requests and operations. I think
>>   categorizing requests into these can be confusing because:
>>   - The resource being protected for different requests will be
>>     different. We are mostly thinking about topics (read/write) but
>>     there are requests for which topic is not the right resource.
>>     E.g., for topic creation, the resource as you suggested would be
>>     something global/common such as “cluster”. For
>>     OffsetCommit/FetchRequest, the resource may be the consumer group,
>>     or maybe a tuple of <consumer group, topic>. So this can be
>>     confusing - i.e., different resources and request types in the
>>     same category. It may be simpler and clearer to just have a 1-1
>>     mapping between the operation enum and requests.
>>
>> I only see 2 resource categories right now cluster and topic.  I don’t
>> really care one way or another so we can probably make a quick decision
>>in
>> tomorrow’s meeting to either to 1-1 mapping or have categorization?
>>
>>   - Some requests that are intuitively READ have WRITE side-effects.
>>     E.g., (currently) TopicMetadataRequest with auto-create, although
>>     that will eventually go away. ConsumerMetadataRequest still
>>     auto-creates the offsets topic. Likewise, ADMIN-type requests may
>>     be interpreted as having side-effects (depending on who you ask).
>>
>> Yes and what I am doing right now is checking authorization for all
>> possible actions i.e. for auto-create it checks if the config has it
>> enabled and if yes, check for read + create authorization. Its not very
>> meaningful right now as there is no CREATE authorization but I think
>>this
>> is implementation detail, we need to ensure we call authorize with all
>> possible operations from KafkaAPI.
>> - <quote>When an ACL is missing - fail open</quote>. What does missing
>>   mean? i.e., no explicit ACL for a principal? I'm confused by this
>>   especially in relation to the precedence of DENY over ALLOW. So per
>>   the description:
>>   - If no ACLs exist for topic A then ALLOW all operations on it by
>>     anyone.
>>   - If I now add an ACL for a certain principal P to ALLOW (say) WRITE
>>     to the topic then either:
>>   - This has the effect of DENYing WRITE to all other principals
>>   - Or, this ACL serves no purpose
>>   - If the effect is to DENY WRITE to all other principals, what about
>>     READ. Do all principals (including P) have READ permissions to
>>     topic A?
>>   - In other words, it seems for a specific ACL to be meaningful then
>>     fail close is necessary for an absent ACL.
>>   - <edit>After through the thread: it appears that the DENY override
>>     only applies to the given principal. i.e., in the above case it
>>     appears that the other principals will in fact be granted access.
>>     Then this makes the ACL that was added pointless right?
>>
>> The rule I was going with is
>> - If there is no ACL I.e. This might be a topic that was created in non
>> secure mode or was created before we supported ACLs. We assume you do
>>not
>> want authorization and let all requests go through.
>> - once you add any ACL, we assume you want authorization on the topic
>>and
>> all the general authorization rules now start to apply, I.e we fail
>>close
>> if we don’t find an ACL that allows access or if we find an ACL that
>>denies
>> access. It does not matter if you added a READACL or WRITEACL or
>>ALLOWACL
>> or DENY ACL. If you add any ACL, now every user gets checked against
>>that
>> and if it does not satisfy the ACL, request fails. I.e. If you add an
>>ACL
>> “Allow write to topic-1 form user1 from all hosts” , user-1 has write
>> access from all hosts and no other user has any access(except for
>> superusers who have all the access).
>> - Deny ACLS are suppose to be used to restrict access authorized by some
>> allow ACL, they are not suppose to be required. Implicitly anyone who
>>does
>> not have an allow acl, gets denied. The Deny ACLs are only added to give
>> more control to administrators who wants more granular control with
>>lesser
>> config. The scenario described in mailing list was “Allow user X access
>> from all hosts but Host1,Host2”. in absence of DENY operator you will
>>have
>> to exhaustively list all possible hosts in your ACL which is what we are
>> trying to avoid.
>>
>> - On ZK ACLs: I think ZK will be closed to everyone except Kafka
>>   brokers. This is a dependency on KIP-4 though. i.e., eventually all
>>   clients should talk to brokers only via RPC.
>>
>> Yes.
>>
>> - Topic owner: list vs single entry - both have issues off the bat
>>   (although list is more intuitive at least to me), but perhaps you
>>   could write up some example workflows to clarify the current
>>   proposal. I was thinking that anyone in the owner list should be
>>   considered a super-user of the topic and can grant/revoke
>>   permissions. They should also be allowed to add other principals as
>>   owners. Even with this it is unclear who should be allowed to remove
>>   owners.
>>
>> As you pointed out in the last KIP meeting owners/creators have use out
>> side of security context (plain simple auditing). I don’t think the
>> authorizer work depends on this, it was my bad to even mention it in
>>first
>> place. I think we can have this discussion outside of
>>authorizer/security
>> context and once we have a way to get topic owners the default
>>Authorizer
>> can start using it. It makes sense to treat all owners as super users
>>and I
>> think it is safe to assume superusers can also modify ownership but I
>>think
>> this should not be treated as blocking work for authorization.
>>
>> - What is the effect of deleting a topic - should all associated ACLs
>>   be deleted as well?
>> They should be and with acls being stored as part of TopicConfig this
>>was
>> taken care of automatically. With the new ACL management API the users
>>will
>> have to call remove ACLs explicitly to perform the cleanup. If everyone
>> thinks this should be automated , with the new APIs we will need a
>>hook(or
>> poll) to be notified when a topic is deleted to perform cleanup.
>> - TopicConfigCache to store topic-ACLs. As mentioned above, not all
>>   requests will be tied to topics. We may want to have an entirely
>>   separate ZK directory for ACLs. We have a similar issue with quotas.
>>   This ties in with dynamic config management. We can certainly
>>   leverage the dynamic config management part of topic configs but I
>>   think we need to have a story for non-topic resources.
>>
>> In the first proposal I was going with a topic-Acl and cluster-Acl where
>> cluster-Acls were json acl local files on all brokers. With the new ACL
>> management APIs we are planning to have /kafka-acl node under which all
>> acls will be stored in /kakfa-acls/resource-name -> {acl json data}.
>> Cluster acls will just have resource name kafka-cluster.
>>
>> Thanks,
>>
>> Joel
>>
>> On Thu, Apr 16, 2015 at 12:15:37AM +0000, Parth Brahmbhatt wrote:
>> Kafka currently stores logConfig overrides specified during topic
>>creation
>> in zookeeper, its just an instance of java.util.Properties converted to
>> json. I am proposing in addition to that we store acls and owner as well
>> as part of same Properties map.
>> There is some infrastructure around reading this config, converting it
>> back to Properties map and most importantly propagating any changes
>> efficiently which we will be able to leverage. As this infrastructure is
>> common to the cluster the reading (not interpreting) of config happens
>> outside of any authorization code.
>> If the TopicConfigCache just kept the json representation and left it to
>> authorizer to parse it, the authorizer will have to either parse the
>>json
>> for each request(not acceptable) or it will have to keep one more layer
>>of
>> parsed ACL instance cache. Assuming authorizer will keep an additional
>> caching layer we will now have to implement some way to invalidate the
>> cache which means the TopicConfigCache will have to be an observable
>>which
>> the Authorizer observes and invalidates its cache entries when
>> topicConfigCache gets updated. Seemed like unnecessary complexity with
>>not
>> lot to gain so I went with TopicConfigCache interpreting the json and
>> caching a higher level modeled object.
>> In summary, the interpretation is done for both optimization and
>> simplicity. If you think it is important to allow custom ACL format
>> support we can add one more pluggable config(acl.parser) and
>> interface(AclParser) or it could just be another method in Authorizer.
>> One thing to note the current ACL json is versioned so it is easy to
>>make
>> changes to it however it won’t be possible to support custom ACL formats
>> with the current design.
>> Thanks
>> Parth
>> On 4/15/15, 4:29 PM, "Michael Herstine" <mherstine@linkedin.com.INVALID
>> <ma...@linkedin.com.INVALID>>
>> wrote:
>> >Hi Parth,
>> >
>> >I’m a little confused: why would Kafka need to interpret the JSON?
>>IIRC
>> >KIP-11 even says that the TopicConfigData will just store the JSON. I’m
>> >not really making a design recommendation here, just trying to
>>understand
>> >what you’re proposing.
>> >
>> >On 4/15/15, 11:20 AM, "Parth Brahmbhatt" <pbrahmbhatt@hortonworks.com
>> <ma...@hortonworks.com>>
>> >wrote:
>> >
>> >>Hi Michael,
>> >>
>> >>There is code in kafka codebase that reads and interprets the topic
>> >>config JSON which has acls, owner and logconfig properties. There are
>>3
>> >>use cases that we are supporting with current proposal:
>> >>
>> >>  *   You use out of box simpleAcl authorizer which is tied to the acl
>> >>stored in topic config and the format is locked down.
>> >>  *   You have a custom authorizer and a custom ACL store.
>>Ranger/Argus
>> >>falls under this as they have their own acl store and ui that users
>>use
>> >>to configure acls on the cluster and cluster resources  like topic.
>>It is
>> >>upto the custom authorizer to leverage the kafka acl configs or
>> >>completely ignore them as they have set a user expectation that only
>>acls
>> >>configured via their ui/system will be effective.
>> >>  *   You have a custom authorizer but no custom Acl store. You are
>> >>completely tied to Acl structure that we have provided in out of box
>> >>implementation.
>> >>
>> >>Thanks
>> >>Parth
>> >>
>> >>On 4/15/15, 10:31 AM, "Michael Herstine"
>> >><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
>> ><ma...@linkedin.com.INVALID>>
>> >>wrote:
>> >>
>> >>Hi Parth,
>> >>
>> >>One question that occurred to me at the end of today’s hangout: how
>>tied
>> >>are we to a particular ACL representation under your proposal? I know
>> >>that
>> >>TopicConfigCache will just contain JSON— if a particular site decides
>> >>they
>> >>want to represent their ACLs differently, and swap out the authorizer
>> >>implementation, will that work?  I guess what I’m asking is whether
>> >>there’s any code in the Kafka codebase that will interpret that JSON,
>>or
>> >>does that logic live exclusively in the authorizer?
>> >>
>> >>On 4/14/15, 10:56 PM, "Don Bosco Durai"
>> >><bo...@apache.org>>
>> wrote:
>> >>
>> >>I also feel, having just IP would be more appropriate. Host lookup
>>will
>> >>unnecessary slow things down and would be insecure as you pointed out.
>> >>
>> >>With IP, it will be also able to setup policies (in future if needed)
>> >>with
>> >>ranges or netmasks and it would be more scalable.
>> >>
>> >>Bosco
>> >>
>> >>
>> >>On 4/14/15, 1:40 PM, "Michael Herstine"
>> >><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
>> ><ma...@linkedin.com.INVALID>>
>> >>wrote:
>> >>
>> >>Hi Parth,
>> >>
>> >>Sorry to chime in so late, but I’ve got a minor question on the KIP.
>> >>
>> >>Several methods take a parameter named “host” of type String. Is that
>> >>intended to be a hostname, or an IP address? If the former, I’m
>>curious
>> >>as
>> >>to how that’s found (in my experience, when accepting an incoming
>>socket
>> >>connection, you only know the IP address, and there isn’t a way to map
>> >>that to a hostname without a round trip to a DNS server, which is
>> >>insecure
>> >>anyway).
>> >>
>> >>
>> >>On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
>> 
>>>><pb...@hortonworks.com><mailto
>>>>:
>> pbrahmbhatt@hortonworks.com>>
>> >>wrote:
>> >>
>> >>Hi all,
>> >>
>> >>I have modified the KIP to reflect the recent change request from the
>> >>reviewers. I have been working on the code and I have the server side
>> >>code
>> >>for authorization ready. I am now modifying the command line
>>utilities.
>> >>I
>> >>would really appreciate if some of the committers can spend sometime
>>to
>> >>review the KIP so we can make progress on this.
>> >>
>> >>Thanks
>> >>Parth
>> >>
>> >>On 3/18/15, 2:20 PM, "Michael Herstine"
>> >><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
>> ><ma...@linkedin.com.INVALID>>
>> >>wrote:
>> >>
>> >>Hi Parth,
>> >>
>> >>Thanks! A few questions:
>> >>
>> >>1. Do you want to permit rules in your ACLs that DENY access as well
>>as
>> >>ALLOW? This can be handy setting up rules that have exceptions. E.g.
>> >>“Allow principal P to READ resource R from all hosts” with “Deny
>> >>principal
>> >>P READ access to resource R from host H1” in combination would allow P
>> >>to
>> >>READ R from all hosts *except* H1.
>> >>
>> >>2. When a topic is newly created, will there be an ACL created for it?
>> >>If
>> >>not, would that not deny subsequent access to it?
>> >>
>> >>(nit) Maybe use Principal instead of String to represent principals?
>> >>
>> >>
>> >>On 3/9/15, 11:48 AM, "Don Bosco Durai"
>> >><bo...@apache.org>>
>> wrote:
>> >>
>> >>Parth
>> >>
>> >>Overall it is looking good. Couple of questionsŠ
>> >>
>> >>- Can you give an example how the policies will look like in the
>> >>default
>> >>implementation?
>> >>- In the operations, can we support ³CONNECT² also? This can be used
>> >>during Session connection
>> >>- Regarding access control for ³Topic Creation², since we can¹t do it
>> >>on
>> >>the server side, can we de-scope it for? And plan it as a future
>> >>feature
>> >>request?
>> >>
>> >>Thanks
>> >>
>> >>Bosco
>> >>
>> >>
>> >>On 3/6/15, 8:10 AM, "Harsha" <kafka@harsha.io<mailto:kafka@harsha.io
>> ><ma...@harsha.io>>
>> >>wrote:
>> >>
>> >>Hi Parth,
>> >>            Thanks for putting this together. Overall it looks good
>> >>to
>> >>            me. Although AdminUtils is a concern KIP-4  can probably
>> >>fix
>> >>            that part.
>> >>Thanks,
>> >>Harsha
>> >>
>> >>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>> >>Forgot to add links to wiki and jira.
>> >>Link to wiki:
>> >>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
>> >>t
>> >>i
>> >>o
>> >>n
>> >>+
>> >>Interface
>> >>Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>> >>Thanks
>> >>Parth
>> >>From: Parth Brahmbhatt
>> 
>>>><pb...@hortonworks.com><mailto
>>>>:
>> pbrahmbhatt@hortonworks.com><mailto:p
>> >>b
>> >>rahmbhatt@hortonworks.com<ma...@hortonworks.com>>>
>> >>Date: Thursday, March 5, 2015 at 10:33 AM
>> >>To:
>> >>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:
>> dev@kafka.apache.org><mailto:dev@kafka.apach
>> >>e
>> >>.org>"
>> >><de...@kafka.apache.org><mailto:
>> dev@kafka.apache.org><mailto:dev@kafka.apach
>> >>e
>> >>.org>>
>> >>Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>> >>Hi,
>> >>KIP-11 is open for discussion , I have updated the wiki with the
>> >>design
>> >>and open questions.
>> >>Thanks
>> >>Parth
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >
>>
>>
>>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Jun Rao <ju...@confluent.io>]

Just a followup question. Suppose there are two rules. Rule1 allows user1
and rule2 denies user2. Does user3 have access? If not, does removing rule1
enable user3 access?

Thanks,

Jun

On Mon, Apr 20, 2015 at 1:34 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:

>
> Hi Joel,
>
> Thanks for the review and I plan to update the KIP today with all the
> updated info. My comments in line below.
>
> Thanks
> Parth
>
>
> On 4/20/15, 10:07 AM, "Joel Koshy" <jjkoshy.w@gmail.com<mailto:
> jjkoshy.w@gmail.com>> wrote:
>
> Hi Parth,
>
> Nice work on this KIP.  I did another read through and had a few more
> comments (with edits after I went through the thread). Many of these
> comments were brought up by others as well, so it appears that the KIP
> would benefit from an update at this point to incorporate comments
> from the thread and last hangout.
>
> - The operation enum is mostly self-explanatory, but it would help
>   (for the sake of clarity and completeness if nothing else) to
>   document exactly what each of the enums are. E.g., I think this came
>   up in our hangout - SEND_CONTROL_MESSAGE is unclear and I don't
>   remember what was said about it. <Edit>: After going through the
>   thread it seems the conclusion was to categorize operations. E.g.,
>   WRITE could apply to multiple requests. Again, this is unclear, so
>   if it would be great if you could update the KIP to clarify what you
>   intend.
>
> Will add to document. SEND_CONTROL_MESSAGE Probably a very bad name but
> these are intra borker API calls like controller notifying other brokers to
> update metadata or heartbeats. Any better naming suggestions?
>
> - When you update the KIP to categorize the requests it would also
>   help to have a column for what the resource is for each.
>
> Will add to the KIP.
>
> - FWIW I prefer a 1-1 mapping between requests and operations. I think
>   categorizing requests into these can be confusing because:
>   - The resource being protected for different requests will be
>     different. We are mostly thinking about topics (read/write) but
>     there are requests for which topic is not the right resource.
>     E.g., for topic creation, the resource as you suggested would be
>     something global/common such as “cluster”. For
>     OffsetCommit/FetchRequest, the resource may be the consumer group,
>     or maybe a tuple of <consumer group, topic>. So this can be
>     confusing - i.e., different resources and request types in the
>     same category. It may be simpler and clearer to just have a 1-1
>     mapping between the operation enum and requests.
>
> I only see 2 resource categories right now cluster and topic.  I don’t
> really care one way or another so we can probably make a quick decision in
> tomorrow’s meeting to either to 1-1 mapping or have categorization?
>
>   - Some requests that are intuitively READ have WRITE side-effects.
>     E.g., (currently) TopicMetadataRequest with auto-create, although
>     that will eventually go away. ConsumerMetadataRequest still
>     auto-creates the offsets topic. Likewise, ADMIN-type requests may
>     be interpreted as having side-effects (depending on who you ask).
>
> Yes and what I am doing right now is checking authorization for all
> possible actions i.e. for auto-create it checks if the config has it
> enabled and if yes, check for read + create authorization. Its not very
> meaningful right now as there is no CREATE authorization but I think this
> is implementation detail, we need to ensure we call authorize with all
> possible operations from KafkaAPI.
> - <quote>When an ACL is missing - fail open</quote>. What does missing
>   mean? i.e., no explicit ACL for a principal? I'm confused by this
>   especially in relation to the precedence of DENY over ALLOW. So per
>   the description:
>   - If no ACLs exist for topic A then ALLOW all operations on it by
>     anyone.
>   - If I now add an ACL for a certain principal P to ALLOW (say) WRITE
>     to the topic then either:
>   - This has the effect of DENYing WRITE to all other principals
>   - Or, this ACL serves no purpose
>   - If the effect is to DENY WRITE to all other principals, what about
>     READ. Do all principals (including P) have READ permissions to
>     topic A?
>   - In other words, it seems for a specific ACL to be meaningful then
>     fail close is necessary for an absent ACL.
>   - <edit>After through the thread: it appears that the DENY override
>     only applies to the given principal. i.e., in the above case it
>     appears that the other principals will in fact be granted access.
>     Then this makes the ACL that was added pointless right?
>
> The rule I was going with is
> - If there is no ACL I.e. This might be a topic that was created in non
> secure mode or was created before we supported ACLs. We assume you do not
> want authorization and let all requests go through.
> - once you add any ACL, we assume you want authorization on the topic and
> all the general authorization rules now start to apply, I.e we fail close
> if we don’t find an ACL that allows access or if we find an ACL that denies
> access. It does not matter if you added a READACL or WRITEACL or ALLOWACL
> or DENY ACL. If you add any ACL, now every user gets checked against that
> and if it does not satisfy the ACL, request fails. I.e. If you add an ACL
> “Allow write to topic-1 form user1 from all hosts” , user-1 has write
> access from all hosts and no other user has any access(except for
> superusers who have all the access).
> - Deny ACLS are suppose to be used to restrict access authorized by some
> allow ACL, they are not suppose to be required. Implicitly anyone who does
> not have an allow acl, gets denied. The Deny ACLs are only added to give
> more control to administrators who wants more granular control with lesser
> config. The scenario described in mailing list was “Allow user X access
> from all hosts but Host1,Host2”. in absence of DENY operator you will have
> to exhaustively list all possible hosts in your ACL which is what we are
> trying to avoid.
>
> - On ZK ACLs: I think ZK will be closed to everyone except Kafka
>   brokers. This is a dependency on KIP-4 though. i.e., eventually all
>   clients should talk to brokers only via RPC.
>
> Yes.
>
> - Topic owner: list vs single entry - both have issues off the bat
>   (although list is more intuitive at least to me), but perhaps you
>   could write up some example workflows to clarify the current
>   proposal. I was thinking that anyone in the owner list should be
>   considered a super-user of the topic and can grant/revoke
>   permissions. They should also be allowed to add other principals as
>   owners. Even with this it is unclear who should be allowed to remove
>   owners.
>
> As you pointed out in the last KIP meeting owners/creators have use out
> side of security context (plain simple auditing). I don’t think the
> authorizer work depends on this, it was my bad to even mention it in first
> place. I think we can have this discussion outside of authorizer/security
> context and once we have a way to get topic owners the default Authorizer
> can start using it. It makes sense to treat all owners as super users and I
> think it is safe to assume superusers can also modify ownership but I think
> this should not be treated as blocking work for authorization.
>
> - What is the effect of deleting a topic - should all associated ACLs
>   be deleted as well?
> They should be and with acls being stored as part of TopicConfig this was
> taken care of automatically. With the new ACL management API the users will
> have to call remove ACLs explicitly to perform the cleanup. If everyone
> thinks this should be automated , with the new APIs we will need a hook(or
> poll) to be notified when a topic is deleted to perform cleanup.
> - TopicConfigCache to store topic-ACLs. As mentioned above, not all
>   requests will be tied to topics. We may want to have an entirely
>   separate ZK directory for ACLs. We have a similar issue with quotas.
>   This ties in with dynamic config management. We can certainly
>   leverage the dynamic config management part of topic configs but I
>   think we need to have a story for non-topic resources.
>
> In the first proposal I was going with a topic-Acl and cluster-Acl where
> cluster-Acls were json acl local files on all brokers. With the new ACL
> management APIs we are planning to have /kafka-acl node under which all
> acls will be stored in /kakfa-acls/resource-name -> {acl json data}.
> Cluster acls will just have resource name kafka-cluster.
>
> Thanks,
>
> Joel
>
> On Thu, Apr 16, 2015 at 12:15:37AM +0000, Parth Brahmbhatt wrote:
> Kafka currently stores logConfig overrides specified during topic creation
> in zookeeper, its just an instance of java.util.Properties converted to
> json. I am proposing in addition to that we store acls and owner as well
> as part of same Properties map.
> There is some infrastructure around reading this config, converting it
> back to Properties map and most importantly propagating any changes
> efficiently which we will be able to leverage. As this infrastructure is
> common to the cluster the reading (not interpreting) of config happens
> outside of any authorization code.
> If the TopicConfigCache just kept the json representation and left it to
> authorizer to parse it, the authorizer will have to either parse the json
> for each request(not acceptable) or it will have to keep one more layer of
> parsed ACL instance cache. Assuming authorizer will keep an additional
> caching layer we will now have to implement some way to invalidate the
> cache which means the TopicConfigCache will have to be an observable which
> the Authorizer observes and invalidates its cache entries when
> topicConfigCache gets updated. Seemed like unnecessary complexity with not
> lot to gain so I went with TopicConfigCache interpreting the json and
> caching a higher level modeled object.
> In summary, the interpretation is done for both optimization and
> simplicity. If you think it is important to allow custom ACL format
> support we can add one more pluggable config(acl.parser) and
> interface(AclParser) or it could just be another method in Authorizer.
> One thing to note the current ACL json is versioned so it is easy to make
> changes to it however it won’t be possible to support custom ACL formats
> with the current design.
> Thanks
> Parth
> On 4/15/15, 4:29 PM, "Michael Herstine" <mherstine@linkedin.com.INVALID
> <ma...@linkedin.com.INVALID>>
> wrote:
> >Hi Parth,
> >
> >I’m a little confused: why would Kafka need to interpret the JSON?  IIRC
> >KIP-11 even says that the TopicConfigData will just store the JSON. I’m
> >not really making a design recommendation here, just trying to understand
> >what you’re proposing.
> >
> >On 4/15/15, 11:20 AM, "Parth Brahmbhatt" <pbrahmbhatt@hortonworks.com
> <ma...@hortonworks.com>>
> >wrote:
> >
> >>Hi Michael,
> >>
> >>There is code in kafka codebase that reads and interprets the topic
> >>config JSON which has acls, owner and logconfig properties. There are 3
> >>use cases that we are supporting with current proposal:
> >>
> >>  *   You use out of box simpleAcl authorizer which is tied to the acl
> >>stored in topic config and the format is locked down.
> >>  *   You have a custom authorizer and a custom ACL store.  Ranger/Argus
> >>falls under this as they have their own acl store and ui that users use
> >>to configure acls on the cluster and cluster resources  like topic. It is
> >>upto the custom authorizer to leverage the kafka acl configs or
> >>completely ignore them as they have set a user expectation that only acls
> >>configured via their ui/system will be effective.
> >>  *   You have a custom authorizer but no custom Acl store. You are
> >>completely tied to Acl structure that we have provided in out of box
> >>implementation.
> >>
> >>Thanks
> >>Parth
> >>
> >>On 4/15/15, 10:31 AM, "Michael Herstine"
> >><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
> ><ma...@linkedin.com.INVALID>>
> >>wrote:
> >>
> >>Hi Parth,
> >>
> >>One question that occurred to me at the end of today’s hangout: how tied
> >>are we to a particular ACL representation under your proposal? I know
> >>that
> >>TopicConfigCache will just contain JSON— if a particular site decides
> >>they
> >>want to represent their ACLs differently, and swap out the authorizer
> >>implementation, will that work?  I guess what I’m asking is whether
> >>there’s any code in the Kafka codebase that will interpret that JSON, or
> >>does that logic live exclusively in the authorizer?
> >>
> >>On 4/14/15, 10:56 PM, "Don Bosco Durai"
> >><bo...@apache.org>>
> wrote:
> >>
> >>I also feel, having just IP would be more appropriate. Host lookup will
> >>unnecessary slow things down and would be insecure as you pointed out.
> >>
> >>With IP, it will be also able to setup policies (in future if needed)
> >>with
> >>ranges or netmasks and it would be more scalable.
> >>
> >>Bosco
> >>
> >>
> >>On 4/14/15, 1:40 PM, "Michael Herstine"
> >><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
> ><ma...@linkedin.com.INVALID>>
> >>wrote:
> >>
> >>Hi Parth,
> >>
> >>Sorry to chime in so late, but I’ve got a minor question on the KIP.
> >>
> >>Several methods take a parameter named “host” of type String. Is that
> >>intended to be a hostname, or an IP address? If the former, I’m curious
> >>as
> >>to how that’s found (in my experience, when accepting an incoming socket
> >>connection, you only know the IP address, and there isn’t a way to map
> >>that to a hostname without a round trip to a DNS server, which is
> >>insecure
> >>anyway).
> >>
> >>
> >>On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
> >><pb...@hortonworks.com><mailto:
> pbrahmbhatt@hortonworks.com>>
> >>wrote:
> >>
> >>Hi all,
> >>
> >>I have modified the KIP to reflect the recent change request from the
> >>reviewers. I have been working on the code and I have the server side
> >>code
> >>for authorization ready. I am now modifying the command line utilities.
> >>I
> >>would really appreciate if some of the committers can spend sometime to
> >>review the KIP so we can make progress on this.
> >>
> >>Thanks
> >>Parth
> >>
> >>On 3/18/15, 2:20 PM, "Michael Herstine"
> >><mherstine@linkedin.com.INVALID<mailto:mherstine@linkedin.com.INVALID
> ><ma...@linkedin.com.INVALID>>
> >>wrote:
> >>
> >>Hi Parth,
> >>
> >>Thanks! A few questions:
> >>
> >>1. Do you want to permit rules in your ACLs that DENY access as well as
> >>ALLOW? This can be handy setting up rules that have exceptions. E.g.
> >>“Allow principal P to READ resource R from all hosts” with “Deny
> >>principal
> >>P READ access to resource R from host H1” in combination would allow P
> >>to
> >>READ R from all hosts *except* H1.
> >>
> >>2. When a topic is newly created, will there be an ACL created for it?
> >>If
> >>not, would that not deny subsequent access to it?
> >>
> >>(nit) Maybe use Principal instead of String to represent principals?
> >>
> >>
> >>On 3/9/15, 11:48 AM, "Don Bosco Durai"
> >><bo...@apache.org>>
> wrote:
> >>
> >>Parth
> >>
> >>Overall it is looking good. Couple of questionsŠ
> >>
> >>- Can you give an example how the policies will look like in the
> >>default
> >>implementation?
> >>- In the operations, can we support ³CONNECT² also? This can be used
> >>during Session connection
> >>- Regarding access control for ³Topic Creation², since we can¹t do it
> >>on
> >>the server side, can we de-scope it for? And plan it as a future
> >>feature
> >>request?
> >>
> >>Thanks
> >>
> >>Bosco
> >>
> >>
> >>On 3/6/15, 8:10 AM, "Harsha" <kafka@harsha.io<mailto:kafka@harsha.io
> ><ma...@harsha.io>>
> >>wrote:
> >>
> >>Hi Parth,
> >>            Thanks for putting this together. Overall it looks good
> >>to
> >>            me. Although AdminUtils is a concern KIP-4  can probably
> >>fix
> >>            that part.
> >>Thanks,
> >>Harsha
> >>
> >>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
> >>Forgot to add links to wiki and jira.
> >>Link to wiki:
> >>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
> >>t
> >>i
> >>o
> >>n
> >>+
> >>Interface
> >>Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
> >>Thanks
> >>Parth
> >>From: Parth Brahmbhatt
> >><pb...@hortonworks.com><mailto:
> pbrahmbhatt@hortonworks.com><mailto:p
> >>b
> >>rahmbhatt@hortonworks.com<ma...@hortonworks.com>>>
> >>Date: Thursday, March 5, 2015 at 10:33 AM
> >>To:
> >>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:
> dev@kafka.apache.org><mailto:dev@kafka.apach
> >>e
> >>.org>"
> >><de...@kafka.apache.org><mailto:
> dev@kafka.apache.org><mailto:dev@kafka.apach
> >>e
> >>.org>>
> >>Subject: [DISCUSS] KIP-11- Authorization design for kafka security
> >>Hi,
> >>KIP-11 is open for discussion , I have updated the wiki with the
> >>design
> >>and open questions.
> >>Thanks
> >>Parth
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
>
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Hi Joel,

Thanks for the review and I plan to update the KIP today with all the updated info. My comments in line below.

Thanks
Parth


On 4/20/15, 10:07 AM, "Joel Koshy" <jj...@gmail.com>> wrote:

Hi Parth,

Nice work on this KIP.  I did another read through and had a few more
comments (with edits after I went through the thread). Many of these
comments were brought up by others as well, so it appears that the KIP
would benefit from an update at this point to incorporate comments
from the thread and last hangout.

- The operation enum is mostly self-explanatory, but it would help
  (for the sake of clarity and completeness if nothing else) to
  document exactly what each of the enums are. E.g., I think this came
  up in our hangout - SEND_CONTROL_MESSAGE is unclear and I don't
  remember what was said about it. <Edit>: After going through the
  thread it seems the conclusion was to categorize operations. E.g.,
  WRITE could apply to multiple requests. Again, this is unclear, so
  if it would be great if you could update the KIP to clarify what you
  intend.

Will add to document. SEND_CONTROL_MESSAGE Probably a very bad name but these are intra borker API calls like controller notifying other brokers to update metadata or heartbeats. Any better naming suggestions?

- When you update the KIP to categorize the requests it would also
  help to have a column for what the resource is for each.

Will add to the KIP.

- FWIW I prefer a 1-1 mapping between requests and operations. I think
  categorizing requests into these can be confusing because:
  - The resource being protected for different requests will be
    different. We are mostly thinking about topics (read/write) but
    there are requests for which topic is not the right resource.
    E.g., for topic creation, the resource as you suggested would be
    something global/common such as “cluster”. For
    OffsetCommit/FetchRequest, the resource may be the consumer group,
    or maybe a tuple of <consumer group, topic>. So this can be
    confusing - i.e., different resources and request types in the
    same category. It may be simpler and clearer to just have a 1-1
    mapping between the operation enum and requests.

I only see 2 resource categories right now cluster and topic.  I don’t really care one way or another so we can probably make a quick decision in tomorrow’s meeting to either to 1-1 mapping or have categorization?

  - Some requests that are intuitively READ have WRITE side-effects.
    E.g., (currently) TopicMetadataRequest with auto-create, although
    that will eventually go away. ConsumerMetadataRequest still
    auto-creates the offsets topic. Likewise, ADMIN-type requests may
    be interpreted as having side-effects (depending on who you ask).

Yes and what I am doing right now is checking authorization for all possible actions i.e. for auto-create it checks if the config has it enabled and if yes, check for read + create authorization. Its not very meaningful right now as there is no CREATE authorization but I think this is implementation detail, we need to ensure we call authorize with all possible operations from KafkaAPI.
- <quote>When an ACL is missing - fail open</quote>. What does missing
  mean? i.e., no explicit ACL for a principal? I'm confused by this
  especially in relation to the precedence of DENY over ALLOW. So per
  the description:
  - If no ACLs exist for topic A then ALLOW all operations on it by
    anyone.
  - If I now add an ACL for a certain principal P to ALLOW (say) WRITE
    to the topic then either:
  - This has the effect of DENYing WRITE to all other principals
  - Or, this ACL serves no purpose
  - If the effect is to DENY WRITE to all other principals, what about
    READ. Do all principals (including P) have READ permissions to
    topic A?
  - In other words, it seems for a specific ACL to be meaningful then
    fail close is necessary for an absent ACL.
  - <edit>After through the thread: it appears that the DENY override
    only applies to the given principal. i.e., in the above case it
    appears that the other principals will in fact be granted access.
    Then this makes the ACL that was added pointless right?

The rule I was going with is
- If there is no ACL I.e. This might be a topic that was created in non secure mode or was created before we supported ACLs. We assume you do not want authorization and let all requests go through.
- once you add any ACL, we assume you want authorization on the topic and all the general authorization rules now start to apply, I.e we fail close if we don’t find an ACL that allows access or if we find an ACL that denies access. It does not matter if you added a READACL or WRITEACL or ALLOWACL or DENY ACL. If you add any ACL, now every user gets checked against that and if it does not satisfy the ACL, request fails. I.e. If you add an ACL “Allow write to topic-1 form user1 from all hosts” , user-1 has write access from all hosts and no other user has any access(except for superusers who have all the access).
- Deny ACLS are suppose to be used to restrict access authorized by some allow ACL, they are not suppose to be required. Implicitly anyone who does not have an allow acl, gets denied. The Deny ACLs are only added to give more control to administrators who wants more granular control with lesser config. The scenario described in mailing list was “Allow user X access from all hosts but Host1,Host2”. in absence of DENY operator you will have to exhaustively list all possible hosts in your ACL which is what we are trying to avoid.

- On ZK ACLs: I think ZK will be closed to everyone except Kafka
  brokers. This is a dependency on KIP-4 though. i.e., eventually all
  clients should talk to brokers only via RPC.

Yes.

- Topic owner: list vs single entry - both have issues off the bat
  (although list is more intuitive at least to me), but perhaps you
  could write up some example workflows to clarify the current
  proposal. I was thinking that anyone in the owner list should be
  considered a super-user of the topic and can grant/revoke
  permissions. They should also be allowed to add other principals as
  owners. Even with this it is unclear who should be allowed to remove
  owners.

As you pointed out in the last KIP meeting owners/creators have use out side of security context (plain simple auditing). I don’t think the authorizer work depends on this, it was my bad to even mention it in first place. I think we can have this discussion outside of authorizer/security context and once we have a way to get topic owners the default Authorizer can start using it. It makes sense to treat all owners as super users and I think it is safe to assume superusers can also modify ownership but I think this should not be treated as blocking work for authorization.

- What is the effect of deleting a topic - should all associated ACLs
  be deleted as well?
They should be and with acls being stored as part of TopicConfig this was taken care of automatically. With the new ACL management API the users will have to call remove ACLs explicitly to perform the cleanup. If everyone thinks this should be automated , with the new APIs we will need a hook(or poll) to be notified when a topic is deleted to perform cleanup.
- TopicConfigCache to store topic-ACLs. As mentioned above, not all
  requests will be tied to topics. We may want to have an entirely
  separate ZK directory for ACLs. We have a similar issue with quotas.
  This ties in with dynamic config management. We can certainly
  leverage the dynamic config management part of topic configs but I
  think we need to have a story for non-topic resources.

In the first proposal I was going with a topic-Acl and cluster-Acl where cluster-Acls were json acl local files on all brokers. With the new ACL management APIs we are planning to have /kafka-acl node under which all acls will be stored in /kakfa-acls/resource-name -> {acl json data}. Cluster acls will just have resource name kafka-cluster.

Thanks,

Joel

On Thu, Apr 16, 2015 at 12:15:37AM +0000, Parth Brahmbhatt wrote:
Kafka currently stores logConfig overrides specified during topic creation
in zookeeper, its just an instance of java.util.Properties converted to
json. I am proposing in addition to that we store acls and owner as well
as part of same Properties map.
There is some infrastructure around reading this config, converting it
back to Properties map and most importantly propagating any changes
efficiently which we will be able to leverage. As this infrastructure is
common to the cluster the reading (not interpreting) of config happens
outside of any authorization code.
If the TopicConfigCache just kept the json representation and left it to
authorizer to parse it, the authorizer will have to either parse the json
for each request(not acceptable) or it will have to keep one more layer of
parsed ACL instance cache. Assuming authorizer will keep an additional
caching layer we will now have to implement some way to invalidate the
cache which means the TopicConfigCache will have to be an observable which
the Authorizer observes and invalidates its cache entries when
topicConfigCache gets updated. Seemed like unnecessary complexity with not
lot to gain so I went with TopicConfigCache interpreting the json and
caching a higher level modeled object.
In summary, the interpretation is done for both optimization and
simplicity. If you think it is important to allow custom ACL format
support we can add one more pluggable config(acl.parser) and
interface(AclParser) or it could just be another method in Authorizer.
One thing to note the current ACL json is versioned so it is easy to make
changes to it however it won’t be possible to support custom ACL formats
with the current design.
Thanks
Parth
On 4/15/15, 4:29 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>>
wrote:
>Hi Parth,
>
>I’m a little confused: why would Kafka need to interpret the JSON?  IIRC
>KIP-11 even says that the TopicConfigData will just store the JSON. I’m
>not really making a design recommendation here, just trying to understand
>what you’re proposing.
>
>On 4/15/15, 11:20 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>>
>wrote:
>
>>Hi Michael,
>>
>>There is code in kafka codebase that reads and interprets the topic
>>config JSON which has acls, owner and logconfig properties. There are 3
>>use cases that we are supporting with current proposal:
>>
>>  *   You use out of box simpleAcl authorizer which is tied to the acl
>>stored in topic config and the format is locked down.
>>  *   You have a custom authorizer and a custom ACL store.  Ranger/Argus
>>falls under this as they have their own acl store and ui that users use
>>to configure acls on the cluster and cluster resources  like topic. It is
>>upto the custom authorizer to leverage the kafka acl configs or
>>completely ignore them as they have set a user expectation that only acls
>>configured via their ui/system will be effective.
>>  *   You have a custom authorizer but no custom Acl store. You are
>>completely tied to Acl structure that we have provided in out of box
>>implementation.
>>
>>Thanks
>>Parth
>>
>>On 4/15/15, 10:31 AM, "Michael Herstine"
>><mh...@linkedin.com.INVALID>>
>>wrote:
>>
>>Hi Parth,
>>
>>One question that occurred to me at the end of today’s hangout: how tied
>>are we to a particular ACL representation under your proposal? I know
>>that
>>TopicConfigCache will just contain JSON— if a particular site decides
>>they
>>want to represent their ACLs differently, and swap out the authorizer
>>implementation, will that work?  I guess what I’m asking is whether
>>there’s any code in the Kafka codebase that will interpret that JSON, or
>>does that logic live exclusively in the authorizer?
>>
>>On 4/14/15, 10:56 PM, "Don Bosco Durai"
>><bo...@apache.org>> wrote:
>>
>>I also feel, having just IP would be more appropriate. Host lookup will
>>unnecessary slow things down and would be insecure as you pointed out.
>>
>>With IP, it will be also able to setup policies (in future if needed)
>>with
>>ranges or netmasks and it would be more scalable.
>>
>>Bosco
>>
>>
>>On 4/14/15, 1:40 PM, "Michael Herstine"
>><mh...@linkedin.com.INVALID>>
>>wrote:
>>
>>Hi Parth,
>>
>>Sorry to chime in so late, but I’ve got a minor question on the KIP.
>>
>>Several methods take a parameter named “host” of type String. Is that
>>intended to be a hostname, or an IP address? If the former, I’m curious
>>as
>>to how that’s found (in my experience, when accepting an incoming socket
>>connection, you only know the IP address, and there isn’t a way to map
>>that to a hostname without a round trip to a DNS server, which is
>>insecure
>>anyway).
>>
>>
>>On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
>><pb...@hortonworks.com>>
>>wrote:
>>
>>Hi all,
>>
>>I have modified the KIP to reflect the recent change request from the
>>reviewers. I have been working on the code and I have the server side
>>code
>>for authorization ready. I am now modifying the command line utilities.
>>I
>>would really appreciate if some of the committers can spend sometime to
>>review the KIP so we can make progress on this.
>>
>>Thanks
>>Parth
>>
>>On 3/18/15, 2:20 PM, "Michael Herstine"
>><mh...@linkedin.com.INVALID>>
>>wrote:
>>
>>Hi Parth,
>>
>>Thanks! A few questions:
>>
>>1. Do you want to permit rules in your ACLs that DENY access as well as
>>ALLOW? This can be handy setting up rules that have exceptions. E.g.
>>“Allow principal P to READ resource R from all hosts” with “Deny
>>principal
>>P READ access to resource R from host H1” in combination would allow P
>>to
>>READ R from all hosts *except* H1.
>>
>>2. When a topic is newly created, will there be an ACL created for it?
>>If
>>not, would that not deny subsequent access to it?
>>
>>(nit) Maybe use Principal instead of String to represent principals?
>>
>>
>>On 3/9/15, 11:48 AM, "Don Bosco Durai"
>><bo...@apache.org>> wrote:
>>
>>Parth
>>
>>Overall it is looking good. Couple of questionsŠ
>>
>>- Can you give an example how the policies will look like in the
>>default
>>implementation?
>>- In the operations, can we support ³CONNECT² also? This can be used
>>during Session connection
>>- Regarding access control for ³Topic Creation², since we can¹t do it
>>on
>>the server side, can we de-scope it for? And plan it as a future
>>feature
>>request?
>>
>>Thanks
>>
>>Bosco
>>
>>
>>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io>>
>>wrote:
>>
>>Hi Parth,
>>            Thanks for putting this together. Overall it looks good
>>to
>>            me. Although AdminUtils is a concern KIP-4  can probably
>>fix
>>            that part.
>>Thanks,
>>Harsha
>>
>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>Forgot to add links to wiki and jira.
>>Link to wiki:
>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
>>t
>>i
>>o
>>n
>>+
>>Interface
>>Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>Thanks
>>Parth
>>From: Parth Brahmbhatt
>><pb...@hortonworks.com><mailto:p
>>b
>>rahmbhatt@hortonworks.com<ma...@hortonworks.com>>>
>>Date: Thursday, March 5, 2015 at 10:33 AM
>>To:
>>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:dev@kafka.apach
>>e
>>.org>"
>><de...@kafka.apache.org><mailto:dev@kafka.apach
>>e
>>.org>>
>>Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>Hi,
>>KIP-11 is open for discussion , I have updated the wiki with the
>>design
>>and open questions.
>>Thanks
>>Parth
>>
>>
>>
>>
>>
>>
>>
>>
>>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Joel Koshy <jj...@gmail.com>]

Hi Parth,

Nice work on this KIP.  I did another read through and had a few more
comments (with edits after I went through the thread). Many of these
comments were brought up by others as well, so it appears that the KIP
would benefit from an update at this point to incorporate comments
from the thread and last hangout.

- The operation enum is mostly self-explanatory, but it would help
  (for the sake of clarity and completeness if nothing else) to
  document exactly what each of the enums are. E.g., I think this came
  up in our hangout - SEND_CONTROL_MESSAGE is unclear and I don't
  remember what was said about it. <Edit>: After going through the
  thread it seems the conclusion was to categorize operations. E.g.,
  WRITE could apply to multiple requests. Again, this is unclear, so
  if it would be great if you could update the KIP to clarify what you
  intend.
- When you update the KIP to categorize the requests it would also
  help to have a column for what the resource is for each.
- FWIW I prefer a 1-1 mapping between requests and operations. I think
  categorizing requests into these can be confusing because:
  - The resource being protected for different requests will be
    different. We are mostly thinking about topics (read/write) but
    there are requests for which topic is not the right resource.
    E.g., for topic creation, the resource as you suggested would be
    something global/common such as “cluster”. For
    OffsetCommit/FetchRequest, the resource may be the consumer group,
    or maybe a tuple of <consumer group, topic>. So this can be
    confusing - i.e., different resources and request types in the
    same category. It may be simpler and clearer to just have a 1-1
    mapping between the operation enum and requests.
  - Some requests that are intuitively READ have WRITE side-effects.
    E.g., (currently) TopicMetadataRequest with auto-create, although
    that will eventually go away. ConsumerMetadataRequest still
    auto-creates the offsets topic. Likewise, ADMIN-type requests may
    be interpreted as having side-effects (depending on who you ask).
- <quote>When an ACL is missing - fail open</quote>. What does missing
  mean? i.e., no explicit ACL for a principal? I'm confused by this
  especially in relation to the precedence of DENY over ALLOW. So per
  the description:
  - If no ACLs exist for topic A then ALLOW all operations on it by
    anyone.
  - If I now add an ACL for a certain principal P to ALLOW (say) WRITE
    to the topic then either:
  - This has the effect of DENYing WRITE to all other principals
  - Or, this ACL serves no purpose
  - If the effect is to DENY WRITE to all other principals, what about
    READ. Do all principals (including P) have READ permissions to
    topic A?
  - In other words, it seems for a specific ACL to be meaningful then
    fail close is necessary for an absent ACL.
  - <edit>After through the thread: it appears that the DENY override
    only applies to the given principal. i.e., in the above case it
    appears that the other principals will in fact be granted access.
    Then this makes the ACL that was added pointless right?
- On ZK ACLs: I think ZK will be closed to everyone except Kafka
  brokers. This is a dependency on KIP-4 though. i.e., eventually all
  clients should talk to brokers only via RPC.
- Topic owner: list vs single entry - both have issues off the bat
  (although list is more intuitive at least to me), but perhaps you
  could write up some example workflows to clarify the current
  proposal. I was thinking that anyone in the owner list should be
  considered a super-user of the topic and can grant/revoke
  permissions. They should also be allowed to add other principals as
  owners. Even with this it is unclear who should be allowed to remove
  owners.
- What is the effect of deleting a topic - should all associated ACLs
  be deleted as well?
- TopicConfigCache to store topic-ACLs. As mentioned above, not all
  requests will be tied to topics. We may want to have an entirely
  separate ZK directory for ACLs. We have a similar issue with quotas.
  This ties in with dynamic config management. We can certainly
  leverage the dynamic config management part of topic configs but I
  think we need to have a story for non-topic resources.

Thanks,

Joel

On Thu, Apr 16, 2015 at 12:15:37AM +0000, Parth Brahmbhatt wrote:
> Kafka currently stores logConfig overrides specified during topic creation
> in zookeeper, its just an instance of java.util.Properties converted to
> json. I am proposing in addition to that we store acls and owner as well
> as part of same Properties map.
> There is some infrastructure around reading this config, converting it
> back to Properties map and most importantly propagating any changes
> efficiently which we will be able to leverage. As this infrastructure is
> common to the cluster the reading (not interpreting) of config happens
> outside of any authorization code.
> 
> If the TopicConfigCache just kept the json representation and left it to
> authorizer to parse it, the authorizer will have to either parse the json
> for each request(not acceptable) or it will have to keep one more layer of
> parsed ACL instance cache. Assuming authorizer will keep an additional
> caching layer we will now have to implement some way to invalidate the
> cache which means the TopicConfigCache will have to be an observable which
> the Authorizer observes and invalidates its cache entries when
> topicConfigCache gets updated. Seemed like unnecessary complexity with not
> lot to gain so I went with TopicConfigCache interpreting the json and
> caching a higher level modeled object.
> 
> In summary, the interpretation is done for both optimization and
> simplicity. If you think it is important to allow custom ACL format
> support we can add one more pluggable config(acl.parser) and
> interface(AclParser) or it could just be another method in Authorizer.
> One thing to note the current ACL json is versioned so it is easy to make
> changes to it however it won’t be possible to support custom ACL formats
> with the current design.
> 
> Thanks
> Parth
> 
> On 4/15/15, 4:29 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
> wrote:
> 
> >Hi Parth,
> >
> >I’m a little confused: why would Kafka need to interpret the JSON?  IIRC
> >KIP-11 even says that the TopicConfigData will just store the JSON. I’m
> >not really making a design recommendation here, just trying to understand
> >what you’re proposing.
> >
> >On 4/15/15, 11:20 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>
> >wrote:
> >
> >>Hi Michael,
> >>
> >>There is code in kafka codebase that reads and interprets the topic
> >>config JSON which has acls, owner and logconfig properties. There are 3
> >>use cases that we are supporting with current proposal:
> >>
> >>  *   You use out of box simpleAcl authorizer which is tied to the acl
> >>stored in topic config and the format is locked down.
> >>  *   You have a custom authorizer and a custom ACL store.  Ranger/Argus
> >>falls under this as they have their own acl store and ui that users use
> >>to configure acls on the cluster and cluster resources  like topic. It is
> >>upto the custom authorizer to leverage the kafka acl configs or
> >>completely ignore them as they have set a user expectation that only acls
> >>configured via their ui/system will be effective.
> >>  *   You have a custom authorizer but no custom Acl store. You are
> >>completely tied to Acl structure that we have provided in out of box
> >>implementation.
> >>
> >>Thanks
> >>Parth
> >>
> >>On 4/15/15, 10:31 AM, "Michael Herstine"
> >><mh...@linkedin.com.INVALID>>
> >>wrote:
> >>
> >>Hi Parth,
> >>
> >>One question that occurred to me at the end of today’s hangout: how tied
> >>are we to a particular ACL representation under your proposal? I know
> >>that
> >>TopicConfigCache will just contain JSON— if a particular site decides
> >>they
> >>want to represent their ACLs differently, and swap out the authorizer
> >>implementation, will that work?  I guess what I’m asking is whether
> >>there’s any code in the Kafka codebase that will interpret that JSON, or
> >>does that logic live exclusively in the authorizer?
> >>
> >>On 4/14/15, 10:56 PM, "Don Bosco Durai"
> >><bo...@apache.org>> wrote:
> >>
> >>I also feel, having just IP would be more appropriate. Host lookup will
> >>unnecessary slow things down and would be insecure as you pointed out.
> >>
> >>With IP, it will be also able to setup policies (in future if needed)
> >>with
> >>ranges or netmasks and it would be more scalable.
> >>
> >>Bosco
> >>
> >>
> >>On 4/14/15, 1:40 PM, "Michael Herstine"
> >><mh...@linkedin.com.INVALID>>
> >>wrote:
> >>
> >>Hi Parth,
> >>
> >>Sorry to chime in so late, but I’ve got a minor question on the KIP.
> >>
> >>Several methods take a parameter named “host” of type String. Is that
> >>intended to be a hostname, or an IP address? If the former, I’m curious
> >>as
> >>to how that’s found (in my experience, when accepting an incoming socket
> >>connection, you only know the IP address, and there isn’t a way to map
> >>that to a hostname without a round trip to a DNS server, which is
> >>insecure
> >>anyway).
> >>
> >>
> >>On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
> >><pb...@hortonworks.com>>
> >>wrote:
> >>
> >>Hi all,
> >>
> >>I have modified the KIP to reflect the recent change request from the
> >>reviewers. I have been working on the code and I have the server side
> >>code
> >>for authorization ready. I am now modifying the command line utilities.
> >>I
> >>would really appreciate if some of the committers can spend sometime to
> >>review the KIP so we can make progress on this.
> >>
> >>Thanks
> >>Parth
> >>
> >>On 3/18/15, 2:20 PM, "Michael Herstine"
> >><mh...@linkedin.com.INVALID>>
> >>wrote:
> >>
> >>Hi Parth,
> >>
> >>Thanks! A few questions:
> >>
> >>1. Do you want to permit rules in your ACLs that DENY access as well as
> >>ALLOW? This can be handy setting up rules that have exceptions. E.g.
> >>“Allow principal P to READ resource R from all hosts” with “Deny
> >>principal
> >>P READ access to resource R from host H1” in combination would allow P
> >>to
> >>READ R from all hosts *except* H1.
> >>
> >>2. When a topic is newly created, will there be an ACL created for it?
> >>If
> >>not, would that not deny subsequent access to it?
> >>
> >>(nit) Maybe use Principal instead of String to represent principals?
> >>
> >>
> >>On 3/9/15, 11:48 AM, "Don Bosco Durai"
> >><bo...@apache.org>> wrote:
> >>
> >>Parth
> >>
> >>Overall it is looking good. Couple of questionsŠ
> >>
> >>- Can you give an example how the policies will look like in the
> >>default
> >>implementation?
> >>- In the operations, can we support ³CONNECT² also? This can be used
> >>during Session connection
> >>- Regarding access control for ³Topic Creation², since we can¹t do it
> >>on
> >>the server side, can we de-scope it for? And plan it as a future
> >>feature
> >>request?
> >>
> >>Thanks
> >>
> >>Bosco
> >>
> >>
> >>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io>>
> >>wrote:
> >>
> >>Hi Parth,
> >>            Thanks for putting this together. Overall it looks good
> >>to
> >>            me. Although AdminUtils is a concern KIP-4  can probably
> >>fix
> >>            that part.
> >>Thanks,
> >>Harsha
> >>
> >>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
> >>Forgot to add links to wiki and jira.
> >>Link to wiki:
> >>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
> >>t
> >>i
> >>o
> >>n
> >>+
> >>Interface
> >>Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
> >>Thanks
> >>Parth
> >>From: Parth Brahmbhatt
> >><pb...@hortonworks.com><mailto:p
> >>b
> >>rahmbhatt@hortonworks.com>>
> >>Date: Thursday, March 5, 2015 at 10:33 AM
> >>To: 
> >>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:dev@kafka.apach
> >>e
> >>.org>"
> >><de...@kafka.apache.org><mailto:dev@kafka.apach
> >>e
> >>.org>>
> >>Subject: [DISCUSS] KIP-11- Authorization design for kafka security
> >>Hi,
> >>KIP-11 is open for discussion , I have updated the wiki with the
> >>design
> >>and open questions.
> >>Thanks
> >>Parth
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> 

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Michael Herstine <mh...@linkedin.com.INVALID>]

Ignore my last; I was catching up on this list & hadn’t seen Gwen’s
subsequent proposal.

On 4/15/15, 5:15 PM, "Parth Brahmbhatt" <pb...@hortonworks.com>
wrote:

>Kafka currently stores logConfig overrides specified during topic creation
>in zookeeper, its just an instance of java.util.Properties converted to
>json. I am proposing in addition to that we store acls and owner as well
>as part of same Properties map.
>There is some infrastructure around reading this config, converting it
>back to Properties map and most importantly propagating any changes
>efficiently which we will be able to leverage. As this infrastructure is
>common to the cluster the reading (not interpreting) of config happens
>outside of any authorization code.
>
>If the TopicConfigCache just kept the json representation and left it to
>authorizer to parse it, the authorizer will have to either parse the json
>for each request(not acceptable) or it will have to keep one more layer of
>parsed ACL instance cache. Assuming authorizer will keep an additional
>caching layer we will now have to implement some way to invalidate the
>cache which means the TopicConfigCache will have to be an observable which
>the Authorizer observes and invalidates its cache entries when
>topicConfigCache gets updated. Seemed like unnecessary complexity with not
>lot to gain so I went with TopicConfigCache interpreting the json and
>caching a higher level modeled object.
>
>In summary, the interpretation is done for both optimization and
>simplicity. If you think it is important to allow custom ACL format
>support we can add one more pluggable config(acl.parser) and
>interface(AclParser) or it could just be another method in Authorizer.
>One thing to note the current ACL json is versioned so it is easy to make
>changes to it however it won’t be possible to support custom ACL formats
>with the current design.
>
>Thanks
>Parth
>
>On 4/15/15, 4:29 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
>wrote:
>
>>Hi Parth,
>>
>>I’m a little confused: why would Kafka need to interpret the JSON?  IIRC
>>KIP-11 even says that the TopicConfigData will just store the JSON. I’m
>>not really making a design recommendation here, just trying to understand
>>what you’re proposing.
>>
>>On 4/15/15, 11:20 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>
>>wrote:
>>
>>>Hi Michael,
>>>
>>>There is code in kafka codebase that reads and interprets the topic
>>>config JSON which has acls, owner and logconfig properties. There are 3
>>>use cases that we are supporting with current proposal:
>>>
>>>  *   You use out of box simpleAcl authorizer which is tied to the acl
>>>stored in topic config and the format is locked down.
>>>  *   You have a custom authorizer and a custom ACL store.  Ranger/Argus
>>>falls under this as they have their own acl store and ui that users use
>>>to configure acls on the cluster and cluster resources  like topic. It
>>>is
>>>upto the custom authorizer to leverage the kafka acl configs or
>>>completely ignore them as they have set a user expectation that only
>>>acls
>>>configured via their ui/system will be effective.
>>>  *   You have a custom authorizer but no custom Acl store. You are
>>>completely tied to Acl structure that we have provided in out of box
>>>implementation.
>>>
>>>Thanks
>>>Parth
>>>
>>>On 4/15/15, 10:31 AM, "Michael Herstine"
>>><mh...@linkedin.com.INVALID>>
>>>wrote:
>>>
>>>Hi Parth,
>>>
>>>One question that occurred to me at the end of today’s hangout: how tied
>>>are we to a particular ACL representation under your proposal? I know
>>>that
>>>TopicConfigCache will just contain JSON— if a particular site decides
>>>they
>>>want to represent their ACLs differently, and swap out the authorizer
>>>implementation, will that work?  I guess what I’m asking is whether
>>>there’s any code in the Kafka codebase that will interpret that JSON, or
>>>does that logic live exclusively in the authorizer?
>>>
>>>On 4/14/15, 10:56 PM, "Don Bosco Durai"
>>><bo...@apache.org>> wrote:
>>>
>>>I also feel, having just IP would be more appropriate. Host lookup will
>>>unnecessary slow things down and would be insecure as you pointed out.
>>>
>>>With IP, it will be also able to setup policies (in future if needed)
>>>with
>>>ranges or netmasks and it would be more scalable.
>>>
>>>Bosco
>>>
>>>
>>>On 4/14/15, 1:40 PM, "Michael Herstine"
>>><mh...@linkedin.com.INVALID>>
>>>wrote:
>>>
>>>Hi Parth,
>>>
>>>Sorry to chime in so late, but I’ve got a minor question on the KIP.
>>>
>>>Several methods take a parameter named “host” of type String. Is that
>>>intended to be a hostname, or an IP address? If the former, I’m curious
>>>as
>>>to how that’s found (in my experience, when accepting an incoming socket
>>>connection, you only know the IP address, and there isn’t a way to map
>>>that to a hostname without a round trip to a DNS server, which is
>>>insecure
>>>anyway).
>>>
>>>
>>>On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
>>><pb...@hortonworks.com>>
>>>wrote:
>>>
>>>Hi all,
>>>
>>>I have modified the KIP to reflect the recent change request from the
>>>reviewers. I have been working on the code and I have the server side
>>>code
>>>for authorization ready. I am now modifying the command line utilities.
>>>I
>>>would really appreciate if some of the committers can spend sometime to
>>>review the KIP so we can make progress on this.
>>>
>>>Thanks
>>>Parth
>>>
>>>On 3/18/15, 2:20 PM, "Michael Herstine"
>>><mh...@linkedin.com.INVALID>>
>>>wrote:
>>>
>>>Hi Parth,
>>>
>>>Thanks! A few questions:
>>>
>>>1. Do you want to permit rules in your ACLs that DENY access as well as
>>>ALLOW? This can be handy setting up rules that have exceptions. E.g.
>>>“Allow principal P to READ resource R from all hosts” with “Deny
>>>principal
>>>P READ access to resource R from host H1” in combination would allow P
>>>to
>>>READ R from all hosts *except* H1.
>>>
>>>2. When a topic is newly created, will there be an ACL created for it?
>>>If
>>>not, would that not deny subsequent access to it?
>>>
>>>(nit) Maybe use Principal instead of String to represent principals?
>>>
>>>
>>>On 3/9/15, 11:48 AM, "Don Bosco Durai"
>>><bo...@apache.org>> wrote:
>>>
>>>Parth
>>>
>>>Overall it is looking good. Couple of questionsŠ
>>>
>>>- Can you give an example how the policies will look like in the
>>>default
>>>implementation?
>>>- In the operations, can we support ³CONNECT² also? This can be used
>>>during Session connection
>>>- Regarding access control for ³Topic Creation², since we can¹t do it
>>>on
>>>the server side, can we de-scope it for? And plan it as a future
>>>feature
>>>request?
>>>
>>>Thanks
>>>
>>>Bosco
>>>
>>>
>>>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io>>
>>>wrote:
>>>
>>>Hi Parth,
>>>            Thanks for putting this together. Overall it looks good
>>>to
>>>            me. Although AdminUtils is a concern KIP-4  can probably
>>>fix
>>>            that part.
>>>Thanks,
>>>Harsha
>>>
>>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>>Forgot to add links to wiki and jira.
>>>Link to wiki:
>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
>>>t
>>>i
>>>o
>>>n
>>>+
>>>Interface
>>>Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>>Thanks
>>>Parth
>>>From: Parth Brahmbhatt
>>><pb...@hortonworks.com><mailto:
>>>p
>>>b
>>>rahmbhatt@hortonworks.com>>
>>>Date: Thursday, March 5, 2015 at 10:33 AM
>>>To: 
>>>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:dev@kafka.apac
>>>h
>>>e
>>>.org>"
>>><de...@kafka.apache.org><mailto:dev@kafka.apac
>>>h
>>>e
>>>.org>>
>>>Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>>Hi,
>>>KIP-11 is open for discussion , I have updated the wiki with the
>>>design
>>>and open questions.
>>>Thanks
>>>Parth
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Michael Herstine <mh...@linkedin.com.INVALID>]

Hi Parth,

OK, I understand your approach much better, now. I don’t really have a
strong opinion; I guess I just shared Gwen’s concern (in the hangout)
that, with this approach, if a site implements their own authorizer using
their own ACL implementation, it will work, but they’ll have this
vestigial infrastructure for creating & maintaining ACLs that could cause
confusion.

One final thought: whatever the abstraction you’re planning for to
represent an ACL in the TopicConfig, could you make that pluggable?

On 4/15/15, 5:15 PM, "Parth Brahmbhatt" <pb...@hortonworks.com>
wrote:

>Kafka currently stores logConfig overrides specified during topic creation
>in zookeeper, its just an instance of java.util.Properties converted to
>json. I am proposing in addition to that we store acls and owner as well
>as part of same Properties map.
>There is some infrastructure around reading this config, converting it
>back to Properties map and most importantly propagating any changes
>efficiently which we will be able to leverage. As this infrastructure is
>common to the cluster the reading (not interpreting) of config happens
>outside of any authorization code.
>
>If the TopicConfigCache just kept the json representation and left it to
>authorizer to parse it, the authorizer will have to either parse the json
>for each request(not acceptable) or it will have to keep one more layer of
>parsed ACL instance cache. Assuming authorizer will keep an additional
>caching layer we will now have to implement some way to invalidate the
>cache which means the TopicConfigCache will have to be an observable which
>the Authorizer observes and invalidates its cache entries when
>topicConfigCache gets updated. Seemed like unnecessary complexity with not
>lot to gain so I went with TopicConfigCache interpreting the json and
>caching a higher level modeled object.
>
>In summary, the interpretation is done for both optimization and
>simplicity. If you think it is important to allow custom ACL format
>support we can add one more pluggable config(acl.parser) and
>interface(AclParser) or it could just be another method in Authorizer.
>One thing to note the current ACL json is versioned so it is easy to make
>changes to it however it won’t be possible to support custom ACL formats
>with the current design.
>
>Thanks
>Parth
>
>On 4/15/15, 4:29 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
>wrote:
>
>>Hi Parth,
>>
>>I’m a little confused: why would Kafka need to interpret the JSON?  IIRC
>>KIP-11 even says that the TopicConfigData will just store the JSON. I’m
>>not really making a design recommendation here, just trying to understand
>>what you’re proposing.
>>
>>On 4/15/15, 11:20 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>
>>wrote:
>>
>>>Hi Michael,
>>>
>>>There is code in kafka codebase that reads and interprets the topic
>>>config JSON which has acls, owner and logconfig properties. There are 3
>>>use cases that we are supporting with current proposal:
>>>
>>>  *   You use out of box simpleAcl authorizer which is tied to the acl
>>>stored in topic config and the format is locked down.
>>>  *   You have a custom authorizer and a custom ACL store.  Ranger/Argus
>>>falls under this as they have their own acl store and ui that users use
>>>to configure acls on the cluster and cluster resources  like topic. It
>>>is
>>>upto the custom authorizer to leverage the kafka acl configs or
>>>completely ignore them as they have set a user expectation that only
>>>acls
>>>configured via their ui/system will be effective.
>>>  *   You have a custom authorizer but no custom Acl store. You are
>>>completely tied to Acl structure that we have provided in out of box
>>>implementation.
>>>
>>>Thanks
>>>Parth
>>>
>>>On 4/15/15, 10:31 AM, "Michael Herstine"
>>><mh...@linkedin.com.INVALID>>
>>>wrote:
>>>
>>>Hi Parth,
>>>
>>>One question that occurred to me at the end of today’s hangout: how tied
>>>are we to a particular ACL representation under your proposal? I know
>>>that
>>>TopicConfigCache will just contain JSON— if a particular site decides
>>>they
>>>want to represent their ACLs differently, and swap out the authorizer
>>>implementation, will that work?  I guess what I’m asking is whether
>>>there’s any code in the Kafka codebase that will interpret that JSON, or
>>>does that logic live exclusively in the authorizer?
>>>
>>>On 4/14/15, 10:56 PM, "Don Bosco Durai"
>>><bo...@apache.org>> wrote:
>>>
>>>I also feel, having just IP would be more appropriate. Host lookup will
>>>unnecessary slow things down and would be insecure as you pointed out.
>>>
>>>With IP, it will be also able to setup policies (in future if needed)
>>>with
>>>ranges or netmasks and it would be more scalable.
>>>
>>>Bosco
>>>
>>>
>>>On 4/14/15, 1:40 PM, "Michael Herstine"
>>><mh...@linkedin.com.INVALID>>
>>>wrote:
>>>
>>>Hi Parth,
>>>
>>>Sorry to chime in so late, but I’ve got a minor question on the KIP.
>>>
>>>Several methods take a parameter named “host” of type String. Is that
>>>intended to be a hostname, or an IP address? If the former, I’m curious
>>>as
>>>to how that’s found (in my experience, when accepting an incoming socket
>>>connection, you only know the IP address, and there isn’t a way to map
>>>that to a hostname without a round trip to a DNS server, which is
>>>insecure
>>>anyway).
>>>
>>>
>>>On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
>>><pb...@hortonworks.com>>
>>>wrote:
>>>
>>>Hi all,
>>>
>>>I have modified the KIP to reflect the recent change request from the
>>>reviewers. I have been working on the code and I have the server side
>>>code
>>>for authorization ready. I am now modifying the command line utilities.
>>>I
>>>would really appreciate if some of the committers can spend sometime to
>>>review the KIP so we can make progress on this.
>>>
>>>Thanks
>>>Parth
>>>
>>>On 3/18/15, 2:20 PM, "Michael Herstine"
>>><mh...@linkedin.com.INVALID>>
>>>wrote:
>>>
>>>Hi Parth,
>>>
>>>Thanks! A few questions:
>>>
>>>1. Do you want to permit rules in your ACLs that DENY access as well as
>>>ALLOW? This can be handy setting up rules that have exceptions. E.g.
>>>“Allow principal P to READ resource R from all hosts” with “Deny
>>>principal
>>>P READ access to resource R from host H1” in combination would allow P
>>>to
>>>READ R from all hosts *except* H1.
>>>
>>>2. When a topic is newly created, will there be an ACL created for it?
>>>If
>>>not, would that not deny subsequent access to it?
>>>
>>>(nit) Maybe use Principal instead of String to represent principals?
>>>
>>>
>>>On 3/9/15, 11:48 AM, "Don Bosco Durai"
>>><bo...@apache.org>> wrote:
>>>
>>>Parth
>>>
>>>Overall it is looking good. Couple of questionsŠ
>>>
>>>- Can you give an example how the policies will look like in the
>>>default
>>>implementation?
>>>- In the operations, can we support ³CONNECT² also? This can be used
>>>during Session connection
>>>- Regarding access control for ³Topic Creation², since we can¹t do it
>>>on
>>>the server side, can we de-scope it for? And plan it as a future
>>>feature
>>>request?
>>>
>>>Thanks
>>>
>>>Bosco
>>>
>>>
>>>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io>>
>>>wrote:
>>>
>>>Hi Parth,
>>>            Thanks for putting this together. Overall it looks good
>>>to
>>>            me. Although AdminUtils is a concern KIP-4  can probably
>>>fix
>>>            that part.
>>>Thanks,
>>>Harsha
>>>
>>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>>Forgot to add links to wiki and jira.
>>>Link to wiki:
>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
>>>t
>>>i
>>>o
>>>n
>>>+
>>>Interface
>>>Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>>Thanks
>>>Parth
>>>From: Parth Brahmbhatt
>>><pb...@hortonworks.com><mailto:
>>>p
>>>b
>>>rahmbhatt@hortonworks.com>>
>>>Date: Thursday, March 5, 2015 at 10:33 AM
>>>To: 
>>>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:dev@kafka.apac
>>>h
>>>e
>>>.org>"
>>><de...@kafka.apache.org><mailto:dev@kafka.apac
>>>h
>>>e
>>>.org>>
>>>Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>>Hi,
>>>KIP-11 is open for discussion , I have updated the wiki with the
>>>design
>>>and open questions.
>>>Thanks
>>>Parth
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Kafka currently stores logConfig overrides specified during topic creation
in zookeeper, its just an instance of java.util.Properties converted to
json. I am proposing in addition to that we store acls and owner as well
as part of same Properties map.
There is some infrastructure around reading this config, converting it
back to Properties map and most importantly propagating any changes
efficiently which we will be able to leverage. As this infrastructure is
common to the cluster the reading (not interpreting) of config happens
outside of any authorization code.

If the TopicConfigCache just kept the json representation and left it to
authorizer to parse it, the authorizer will have to either parse the json
for each request(not acceptable) or it will have to keep one more layer of
parsed ACL instance cache. Assuming authorizer will keep an additional
caching layer we will now have to implement some way to invalidate the
cache which means the TopicConfigCache will have to be an observable which
the Authorizer observes and invalidates its cache entries when
topicConfigCache gets updated. Seemed like unnecessary complexity with not
lot to gain so I went with TopicConfigCache interpreting the json and
caching a higher level modeled object.

In summary, the interpretation is done for both optimization and
simplicity. If you think it is important to allow custom ACL format
support we can add one more pluggable config(acl.parser) and
interface(AclParser) or it could just be another method in Authorizer.
One thing to note the current ACL json is versioned so it is easy to make
changes to it however it won’t be possible to support custom ACL formats
with the current design.

Thanks
Parth

On 4/15/15, 4:29 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
wrote:

>Hi Parth,
>
>I’m a little confused: why would Kafka need to interpret the JSON?  IIRC
>KIP-11 even says that the TopicConfigData will just store the JSON. I’m
>not really making a design recommendation here, just trying to understand
>what you’re proposing.
>
>On 4/15/15, 11:20 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>
>wrote:
>
>>Hi Michael,
>>
>>There is code in kafka codebase that reads and interprets the topic
>>config JSON which has acls, owner and logconfig properties. There are 3
>>use cases that we are supporting with current proposal:
>>
>>  *   You use out of box simpleAcl authorizer which is tied to the acl
>>stored in topic config and the format is locked down.
>>  *   You have a custom authorizer and a custom ACL store.  Ranger/Argus
>>falls under this as they have their own acl store and ui that users use
>>to configure acls on the cluster and cluster resources  like topic. It is
>>upto the custom authorizer to leverage the kafka acl configs or
>>completely ignore them as they have set a user expectation that only acls
>>configured via their ui/system will be effective.
>>  *   You have a custom authorizer but no custom Acl store. You are
>>completely tied to Acl structure that we have provided in out of box
>>implementation.
>>
>>Thanks
>>Parth
>>
>>On 4/15/15, 10:31 AM, "Michael Herstine"
>><mh...@linkedin.com.INVALID>>
>>wrote:
>>
>>Hi Parth,
>>
>>One question that occurred to me at the end of today’s hangout: how tied
>>are we to a particular ACL representation under your proposal? I know
>>that
>>TopicConfigCache will just contain JSON— if a particular site decides
>>they
>>want to represent their ACLs differently, and swap out the authorizer
>>implementation, will that work?  I guess what I’m asking is whether
>>there’s any code in the Kafka codebase that will interpret that JSON, or
>>does that logic live exclusively in the authorizer?
>>
>>On 4/14/15, 10:56 PM, "Don Bosco Durai"
>><bo...@apache.org>> wrote:
>>
>>I also feel, having just IP would be more appropriate. Host lookup will
>>unnecessary slow things down and would be insecure as you pointed out.
>>
>>With IP, it will be also able to setup policies (in future if needed)
>>with
>>ranges or netmasks and it would be more scalable.
>>
>>Bosco
>>
>>
>>On 4/14/15, 1:40 PM, "Michael Herstine"
>><mh...@linkedin.com.INVALID>>
>>wrote:
>>
>>Hi Parth,
>>
>>Sorry to chime in so late, but I’ve got a minor question on the KIP.
>>
>>Several methods take a parameter named “host” of type String. Is that
>>intended to be a hostname, or an IP address? If the former, I’m curious
>>as
>>to how that’s found (in my experience, when accepting an incoming socket
>>connection, you only know the IP address, and there isn’t a way to map
>>that to a hostname without a round trip to a DNS server, which is
>>insecure
>>anyway).
>>
>>
>>On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
>><pb...@hortonworks.com>>
>>wrote:
>>
>>Hi all,
>>
>>I have modified the KIP to reflect the recent change request from the
>>reviewers. I have been working on the code and I have the server side
>>code
>>for authorization ready. I am now modifying the command line utilities.
>>I
>>would really appreciate if some of the committers can spend sometime to
>>review the KIP so we can make progress on this.
>>
>>Thanks
>>Parth
>>
>>On 3/18/15, 2:20 PM, "Michael Herstine"
>><mh...@linkedin.com.INVALID>>
>>wrote:
>>
>>Hi Parth,
>>
>>Thanks! A few questions:
>>
>>1. Do you want to permit rules in your ACLs that DENY access as well as
>>ALLOW? This can be handy setting up rules that have exceptions. E.g.
>>“Allow principal P to READ resource R from all hosts” with “Deny
>>principal
>>P READ access to resource R from host H1” in combination would allow P
>>to
>>READ R from all hosts *except* H1.
>>
>>2. When a topic is newly created, will there be an ACL created for it?
>>If
>>not, would that not deny subsequent access to it?
>>
>>(nit) Maybe use Principal instead of String to represent principals?
>>
>>
>>On 3/9/15, 11:48 AM, "Don Bosco Durai"
>><bo...@apache.org>> wrote:
>>
>>Parth
>>
>>Overall it is looking good. Couple of questionsŠ
>>
>>- Can you give an example how the policies will look like in the
>>default
>>implementation?
>>- In the operations, can we support ³CONNECT² also? This can be used
>>during Session connection
>>- Regarding access control for ³Topic Creation², since we can¹t do it
>>on
>>the server side, can we de-scope it for? And plan it as a future
>>feature
>>request?
>>
>>Thanks
>>
>>Bosco
>>
>>
>>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io>>
>>wrote:
>>
>>Hi Parth,
>>            Thanks for putting this together. Overall it looks good
>>to
>>            me. Although AdminUtils is a concern KIP-4  can probably
>>fix
>>            that part.
>>Thanks,
>>Harsha
>>
>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>Forgot to add links to wiki and jira.
>>Link to wiki:
>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
>>t
>>i
>>o
>>n
>>+
>>Interface
>>Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>Thanks
>>Parth
>>From: Parth Brahmbhatt
>><pb...@hortonworks.com><mailto:p
>>b
>>rahmbhatt@hortonworks.com>>
>>Date: Thursday, March 5, 2015 at 10:33 AM
>>To: 
>>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:dev@kafka.apach
>>e
>>.org>"
>><de...@kafka.apache.org><mailto:dev@kafka.apach
>>e
>>.org>>
>>Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>Hi,
>>KIP-11 is open for discussion , I have updated the wiki with the
>>design
>>and open questions.
>>Thanks
>>Parth
>>
>>
>>
>>
>>
>>
>>
>>
>>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Michael Herstine <mh...@linkedin.com.INVALID>]

Hi Parth,

I’m a little confused: why would Kafka need to interpret the JSON?  IIRC
KIP-11 even says that the TopicConfigData will just store the JSON. I’m
not really making a design recommendation here, just trying to understand
what you’re proposing.

On 4/15/15, 11:20 AM, "Parth Brahmbhatt" <pb...@hortonworks.com>
wrote:

>Hi Michael,
>
>There is code in kafka codebase that reads and interprets the topic
>config JSON which has acls, owner and logconfig properties. There are 3
>use cases that we are supporting with current proposal:
>
>  *   You use out of box simpleAcl authorizer which is tied to the acl
>stored in topic config and the format is locked down.
>  *   You have a custom authorizer and a custom ACL store.  Ranger/Argus
>falls under this as they have their own acl store and ui that users use
>to configure acls on the cluster and cluster resources  like topic. It is
>upto the custom authorizer to leverage the kafka acl configs or
>completely ignore them as they have set a user expectation that only acls
>configured via their ui/system will be effective.
>  *   You have a custom authorizer but no custom Acl store. You are
>completely tied to Acl structure that we have provided in out of box
>implementation.
>
>Thanks
>Parth
>
>On 4/15/15, 10:31 AM, "Michael Herstine"
><mh...@linkedin.com.INVALID>>
>wrote:
>
>Hi Parth,
>
>One question that occurred to me at the end of today’s hangout: how tied
>are we to a particular ACL representation under your proposal? I know that
>TopicConfigCache will just contain JSON— if a particular site decides they
>want to represent their ACLs differently, and swap out the authorizer
>implementation, will that work?  I guess what I’m asking is whether
>there’s any code in the Kafka codebase that will interpret that JSON, or
>does that logic live exclusively in the authorizer?
>
>On 4/14/15, 10:56 PM, "Don Bosco Durai"
><bo...@apache.org>> wrote:
>
>I also feel, having just IP would be more appropriate. Host lookup will
>unnecessary slow things down and would be insecure as you pointed out.
>
>With IP, it will be also able to setup policies (in future if needed) with
>ranges or netmasks and it would be more scalable.
>
>Bosco
>
>
>On 4/14/15, 1:40 PM, "Michael Herstine"
><mh...@linkedin.com.INVALID>>
>wrote:
>
>Hi Parth,
>
>Sorry to chime in so late, but I’ve got a minor question on the KIP.
>
>Several methods take a parameter named “host” of type String. Is that
>intended to be a hostname, or an IP address? If the former, I’m curious
>as
>to how that’s found (in my experience, when accepting an incoming socket
>connection, you only know the IP address, and there isn’t a way to map
>that to a hostname without a round trip to a DNS server, which is
>insecure
>anyway).
>
>
>On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
><pb...@hortonworks.com>>
>wrote:
>
>Hi all,
>
>I have modified the KIP to reflect the recent change request from the
>reviewers. I have been working on the code and I have the server side
>code
>for authorization ready. I am now modifying the command line utilities.
>I
>would really appreciate if some of the committers can spend sometime to
>review the KIP so we can make progress on this.
>
>Thanks
>Parth
>
>On 3/18/15, 2:20 PM, "Michael Herstine"
><mh...@linkedin.com.INVALID>>
>wrote:
>
>Hi Parth,
>
>Thanks! A few questions:
>
>1. Do you want to permit rules in your ACLs that DENY access as well as
>ALLOW? This can be handy setting up rules that have exceptions. E.g.
>“Allow principal P to READ resource R from all hosts” with “Deny
>principal
>P READ access to resource R from host H1” in combination would allow P
>to
>READ R from all hosts *except* H1.
>
>2. When a topic is newly created, will there be an ACL created for it?
>If
>not, would that not deny subsequent access to it?
>
>(nit) Maybe use Principal instead of String to represent principals?
>
>
>On 3/9/15, 11:48 AM, "Don Bosco Durai"
><bo...@apache.org>> wrote:
>
>Parth
>
>Overall it is looking good. Couple of questionsŠ
>
>- Can you give an example how the policies will look like in the
>default
>implementation?
>- In the operations, can we support ³CONNECT² also? This can be used
>during Session connection
>- Regarding access control for ³Topic Creation², since we can¹t do it
>on
>the server side, can we de-scope it for? And plan it as a future
>feature
>request?
>
>Thanks
>
>Bosco
>
>
>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io>>
>wrote:
>
>Hi Parth,
>            Thanks for putting this together. Overall it looks good
>to
>            me. Although AdminUtils is a concern KIP-4  can probably
>fix
>            that part.
>Thanks,
>Harsha
>
>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>Forgot to add links to wiki and jira.
>Link to wiki:
>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
>t
>i
>o
>n
>+
>Interface
>Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>Thanks
>Parth
>From: Parth Brahmbhatt
><pb...@hortonworks.com><mailto:pb
>rahmbhatt@hortonworks.com>>
>Date: Thursday, March 5, 2015 at 10:33 AM
>To: 
>"dev@kafka.apache.org<ma...@kafka.apache.org><mailto:dev@kafka.apache
>.org>"
><de...@kafka.apache.org><mailto:dev@kafka.apache
>.org>>
>Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>Hi,
>KIP-11 is open for discussion , I have updated the wiki with the
>design
>and open questions.
>Thanks
>Parth
>
>
>
>
>
>
>
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Tong Li <li...@us.ibm.com>]

Parth,
     If one wants to use his or her own access control including
authentication system, with this design what will be needed to be done? Can
one completely turn this off so that the system behaves exactly same as it
is today?

Thanks.

Tong

Sent from my iPhone

> On Apr 15, 2015, at 1:51 PM, Parth Brahmbhatt
<pb...@hortonworks.com> wrote:
>
> Hi Michael,
>
> There is code in kafka codebase that reads and interprets the topic
config JSON which has acls, owner and logconfig properties. There are 3 use
cases that we are supporting with current proposal:
>
>   *   You use out of box simpleAcl authorizer which is tied to the acl
stored in topic config and the format is locked down.
>   *   You have a custom authorizer and a custom ACL store.  Ranger/Argus
falls under this as they have their own acl store and ui that users use to
configure acls on the cluster and cluster resources  like topic. It is upto
the custom authorizer to leverage the kafka acl configs or completely
ignore them as they have set a user expectation that only acls configured
via their ui/system will be effective.
>   *   You have a custom authorizer but no custom Acl store. You are
completely tied to Acl structure that we have provided in out of box
implementation.
>
> Thanks
> Parth
>
> On 4/15/15, 10:31 AM, "Michael Herstine"
<mh...@linkedin.com.INVALID>>
wrote:
>
> Hi Parth,
>
> One question that occurred to me at the end of today’s hangout: how tied
> are we to a particular ACL representation under your proposal? I know
that
> TopicConfigCache will just contain JSON— if a particular site decides
they
> want to represent their ACLs differently, and swap out the authorizer
> implementation, will that work?  I guess what I’m asking is whether
> there’s any code in the Kafka codebase that will interpret that JSON, or
> does that logic live exclusively in the authorizer?
>
> On 4/14/15, 10:56 PM, "Don Bosco Durai"
<bo...@apache.org>> wrote:
>
> I also feel, having just IP would be more appropriate. Host lookup will
> unnecessary slow things down and would be insecure as you pointed out.
>
> With IP, it will be also able to setup policies (in future if needed)
with
> ranges or netmasks and it would be more scalable.
>
> Bosco
>
>
> On 4/14/15, 1:40 PM, "Michael Herstine"
<mh...@linkedin.com.INVALID>>
> wrote:
>
> Hi Parth,
>
> Sorry to chime in so late, but I’ve got a minor question on the KIP.
>
> Several methods take a parameter named “host” of type String. Is that
> intended to be a hostname, or an IP address? If the former, I’m curious
> as
> to how that’s found (in my experience, when accepting an incoming socket
> connection, you only know the IP address, and there isn’t a way to map
> that to a hostname without a round trip to a DNS server, which is
> insecure
> anyway).
>
>
> On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
<pb...@hortonworks.com>>
> wrote:
>
> Hi all,
>
> I have modified the KIP to reflect the recent change request from the
> reviewers. I have been working on the code and I have the server side
> code
> for authorization ready. I am now modifying the command line utilities.
> I
> would really appreciate if some of the committers can spend sometime to
> review the KIP so we can make progress on this.
>
> Thanks
> Parth
>
> On 3/18/15, 2:20 PM, "Michael Herstine"
<mh...@linkedin.com.INVALID>>
> wrote:
>
> Hi Parth,
>
> Thanks! A few questions:
>
> 1. Do you want to permit rules in your ACLs that DENY access as well as
> ALLOW? This can be handy setting up rules that have exceptions. E.g.
> “Allow principal P to READ resource R from all hosts” with “Deny
> principal
> P READ access to resource R from host H1” in combination would allow P
> to
> READ R from all hosts *except* H1.
>
> 2. When a topic is newly created, will there be an ACL created for it?
> If
> not, would that not deny subsequent access to it?
>
> (nit) Maybe use Principal instead of String to represent principals?
>
>
> On 3/9/15, 11:48 AM, "Don Bosco Durai"
<bo...@apache.org>> wrote:
>
> Parth
>
> Overall it is looking good. Couple of questionsŠ
>
> - Can you give an example how the policies will look like in the
> default
> implementation?
> - In the operations, can we support ³CONNECT² also? This can be used
> during Session connection
> - Regarding access control for ³Topic Creation², since we can¹t do it
> on
> the server side, can we de-scope it for? And plan it as a future
> feature
> request?
>
> Thanks
>
> Bosco
>
>
> On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io>>
wrote:
>
> Hi Parth,
>             Thanks for putting this together. Overall it looks good
> to
>             me. Although AdminUtils is a concern KIP-4  can probably
> fix
>             that part.
> Thanks,
> Harsha
>
> On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
> Forgot to add links to wiki and jira.
> Link to wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
> t
> i
> o
> n
> +
> Interface
> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
> Thanks
> Parth
> From: Parth Brahmbhatt
>
<pb...@hortonworks.com>>

> Date: Thursday, March 5, 2015 at 10:33 AM
> To:
"dev@kafka.apache.org<ma...@kafka.apache.org>"

>
<de...@kafka.apache.org>>

> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
> Hi,
> KIP-11 is open for discussion , I have updated the wiki with the
> design
> and open questions.
> Thanks
> Parth
>
>
>
>
>
>
>
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Hi Michael,

There is code in kafka codebase that reads and interprets the topic config JSON which has acls, owner and logconfig properties. There are 3 use cases that we are supporting with current proposal:

  *   You use out of box simpleAcl authorizer which is tied to the acl stored in topic config and the format is locked down.
  *   You have a custom authorizer and a custom ACL store.  Ranger/Argus falls under this as they have their own acl store and ui that users use to configure acls on the cluster and cluster resources  like topic. It is upto the custom authorizer to leverage the kafka acl configs or completely ignore them as they have set a user expectation that only acls configured via their ui/system will be effective.
  *   You have a custom authorizer but no custom Acl store. You are completely tied to Acl structure that we have provided in out of box implementation.

Thanks
Parth

On 4/15/15, 10:31 AM, "Michael Herstine" <mh...@linkedin.com.INVALID>> wrote:

Hi Parth,

One question that occurred to me at the end of today’s hangout: how tied
are we to a particular ACL representation under your proposal? I know that
TopicConfigCache will just contain JSON— if a particular site decides they
want to represent their ACLs differently, and swap out the authorizer
implementation, will that work?  I guess what I’m asking is whether
there’s any code in the Kafka codebase that will interpret that JSON, or
does that logic live exclusively in the authorizer?

On 4/14/15, 10:56 PM, "Don Bosco Durai" <bo...@apache.org>> wrote:

I also feel, having just IP would be more appropriate. Host lookup will
unnecessary slow things down and would be insecure as you pointed out.

With IP, it will be also able to setup policies (in future if needed) with
ranges or netmasks and it would be more scalable.

Bosco


On 4/14/15, 1:40 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>>
wrote:

Hi Parth,

Sorry to chime in so late, but I’ve got a minor question on the KIP.

Several methods take a parameter named “host” of type String. Is that
intended to be a hostname, or an IP address? If the former, I’m curious
as
to how that’s found (in my experience, when accepting an incoming socket
connection, you only know the IP address, and there isn’t a way to map
that to a hostname without a round trip to a DNS server, which is
insecure
anyway).


On 3/25/15, 1:07 PM, "Parth Brahmbhatt" <pb...@hortonworks.com>>
wrote:

Hi all,

I have modified the KIP to reflect the recent change request from the
reviewers. I have been working on the code and I have the server side
code
for authorization ready. I am now modifying the command line utilities.
I
would really appreciate if some of the committers can spend sometime to
review the KIP so we can make progress on this.

Thanks
Parth

On 3/18/15, 2:20 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>>
wrote:

Hi Parth,

Thanks! A few questions:

1. Do you want to permit rules in your ACLs that DENY access as well as
ALLOW? This can be handy setting up rules that have exceptions. E.g.
“Allow principal P to READ resource R from all hosts” with “Deny
principal
P READ access to resource R from host H1” in combination would allow P
to
READ R from all hosts *except* H1.

2. When a topic is newly created, will there be an ACL created for it?
If
not, would that not deny subsequent access to it?

(nit) Maybe use Principal instead of String to represent principals?


On 3/9/15, 11:48 AM, "Don Bosco Durai" <bo...@apache.org>> wrote:

Parth

Overall it is looking good. Couple of questionsŠ

- Can you give an example how the policies will look like in the
default
implementation?
- In the operations, can we support ³CONNECT² also? This can be used
during Session connection
- Regarding access control for ³Topic Creation², since we can¹t do it
on
the server side, can we de-scope it for? And plan it as a future
feature
request?

Thanks

Bosco


On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io>> wrote:

Hi Parth,
            Thanks for putting this together. Overall it looks good
to
            me. Although AdminUtils is a concern KIP-4  can probably
fix
            that part.
Thanks,
Harsha

On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
Forgot to add links to wiki and jira.
Link to wiki:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
t
i
o
n
+
Interface
Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
Thanks
Parth
From: Parth Brahmbhatt
<pb...@hortonworks.com>>
Date: Thursday, March 5, 2015 at 10:33 AM
To: "dev@kafka.apache.org<ma...@kafka.apache.org>"
<de...@kafka.apache.org>>
Subject: [DISCUSS] KIP-11- Authorization design for kafka security
Hi,
KIP-11 is open for discussion , I have updated the wiki with the
design
and open questions.
Thanks
Parth

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Michael Herstine <mh...@linkedin.com.INVALID>]

Hi Parth,

One question that occurred to me at the end of today’s hangout: how tied
are we to a particular ACL representation under your proposal? I know that
TopicConfigCache will just contain JSON— if a particular site decides they
want to represent their ACLs differently, and swap out the authorizer
implementation, will that work?  I guess what I’m asking is whether
there’s any code in the Kafka codebase that will interpret that JSON, or
does that logic live exclusively in the authorizer?

On 4/14/15, 10:56 PM, "Don Bosco Durai" <bo...@apache.org> wrote:

>I also feel, having just IP would be more appropriate. Host lookup will
>unnecessary slow things down and would be insecure as you pointed out.
>
>With IP, it will be also able to setup policies (in future if needed) with
>ranges or netmasks and it would be more scalable.
>
>Bosco
>
>
>On 4/14/15, 1:40 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
>wrote:
>
>>Hi Parth,
>>
>>Sorry to chime in so late, but I’ve got a minor question on the KIP.
>>
>>Several methods take a parameter named “host” of type String. Is that
>>intended to be a hostname, or an IP address? If the former, I’m curious
>>as
>>to how that’s found (in my experience, when accepting an incoming socket
>>connection, you only know the IP address, and there isn’t a way to map
>>that to a hostname without a round trip to a DNS server, which is
>>insecure
>>anyway).
>>
>>
>>On 3/25/15, 1:07 PM, "Parth Brahmbhatt" <pb...@hortonworks.com>
>>wrote:
>>
>>>Hi all,
>>>
>>>I have modified the KIP to reflect the recent change request from the
>>>reviewers. I have been working on the code and I have the server side
>>>code
>>>for authorization ready. I am now modifying the command line utilities.
>>>I
>>>would really appreciate if some of the committers can spend sometime to
>>>review the KIP so we can make progress on this.
>>>
>>>Thanks
>>>Parth
>>>
>>>On 3/18/15, 2:20 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
>>>wrote:
>>>
>>>>Hi Parth,
>>>>
>>>>Thanks! A few questions:
>>>>
>>>>1. Do you want to permit rules in your ACLs that DENY access as well as
>>>>ALLOW? This can be handy setting up rules that have exceptions. E.g.
>>>>“Allow principal P to READ resource R from all hosts” with “Deny
>>>>principal
>>>>P READ access to resource R from host H1” in combination would allow P
>>>>to
>>>>READ R from all hosts *except* H1.
>>>>
>>>>2. When a topic is newly created, will there be an ACL created for it?
>>>>If
>>>>not, would that not deny subsequent access to it?
>>>>
>>>>(nit) Maybe use Principal instead of String to represent principals?
>>>>
>>>>
>>>>On 3/9/15, 11:48 AM, "Don Bosco Durai" <bo...@apache.org> wrote:
>>>>
>>>>>Parth
>>>>>
>>>>>Overall it is looking good. Couple of questionsŠ
>>>>>
>>>>>- Can you give an example how the policies will look like in the
>>>>>default
>>>>>implementation?
>>>>>- In the operations, can we support ³CONNECT² also? This can be used
>>>>>during Session connection
>>>>>- Regarding access control for ³Topic Creation², since we can¹t do it
>>>>>on
>>>>>the server side, can we de-scope it for? And plan it as a future
>>>>>feature
>>>>>request?
>>>>>
>>>>>Thanks
>>>>>
>>>>>Bosco
>>>>>
>>>>> 
>>>>>
>>>>>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io> wrote:
>>>>>
>>>>>>Hi Parth,
>>>>>>            Thanks for putting this together. Overall it looks good
>>>>>>to
>>>>>>            me. Although AdminUtils is a concern KIP-4  can probably
>>>>>>fix
>>>>>>            that part.
>>>>>>Thanks,
>>>>>>Harsha
>>>>>>
>>>>>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>>>>>> Forgot to add links to wiki and jira.
>>>>>>> 
>>>>>>> Link to wiki:
>>>>>>> 
>>>>>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
>>>>>>>t
>>>>>>>i
>>>>>>>o
>>>>>>>n
>>>>>>>+
>>>>>>>Interface
>>>>>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Parth
>>>>>>> 
>>>>>>> From: Parth Brahmbhatt
>>>>>>> <pb...@hortonworks.com>>
>>>>>>> Date: Thursday, March 5, 2015 at 10:33 AM
>>>>>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org>"
>>>>>>> <de...@kafka.apache.org>>
>>>>>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> KIP-11 is open for discussion , I have updated the wiki with the
>>>>>>>design
>>>>>>> and open questions.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Parth
>>>>>
>>>>>
>>>>
>>>
>>
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Currently the authorizer does not perform any dns lookups and uses the
hostname it receives as part of request.session as is. So in a way we are
allowing only ip addresses. The only match is equality based so no ip
ranges yet but that is easy to add.

However, I think it is ok to allow for both ip addresses and hostnames and
we should allow both. I am not sure why would I want to secure dns lookups
and the host lookups extending to dns server are only necessary when the
dns cache does not have the entry or the cache entry expires. This can be
controlled by setting networkaddress.cache.ttl setting in jvm.


Thanks
Parth

On 4/14/15, 10:56 PM, "Don Bosco Durai" <bo...@apache.org> wrote:

>I also feel, having just IP would be more appropriate. Host lookup will
>unnecessary slow things down and would be insecure as you pointed out.
>
>With IP, it will be also able to setup policies (in future if needed) with
>ranges or netmasks and it would be more scalable.
>
>Bosco
>
>
>On 4/14/15, 1:40 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
>wrote:
>
>>Hi Parth,
>>
>>Sorry to chime in so late, but I’ve got a minor question on the KIP.
>>
>>Several methods take a parameter named “host” of type String. Is that
>>intended to be a hostname, or an IP address? If the former, I’m curious
>>as
>>to how that’s found (in my experience, when accepting an incoming socket
>>connection, you only know the IP address, and there isn’t a way to map
>>that to a hostname without a round trip to a DNS server, which is
>>insecure
>>anyway).
>>
>>
>>On 3/25/15, 1:07 PM, "Parth Brahmbhatt" <pb...@hortonworks.com>
>>wrote:
>>
>>>Hi all,
>>>
>>>I have modified the KIP to reflect the recent change request from the
>>>reviewers. I have been working on the code and I have the server side
>>>code
>>>for authorization ready. I am now modifying the command line utilities.
>>>I
>>>would really appreciate if some of the committers can spend sometime to
>>>review the KIP so we can make progress on this.
>>>
>>>Thanks
>>>Parth
>>>
>>>On 3/18/15, 2:20 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
>>>wrote:
>>>
>>>>Hi Parth,
>>>>
>>>>Thanks! A few questions:
>>>>
>>>>1. Do you want to permit rules in your ACLs that DENY access as well as
>>>>ALLOW? This can be handy setting up rules that have exceptions. E.g.
>>>>“Allow principal P to READ resource R from all hosts” with “Deny
>>>>principal
>>>>P READ access to resource R from host H1” in combination would allow P
>>>>to
>>>>READ R from all hosts *except* H1.
>>>>
>>>>2. When a topic is newly created, will there be an ACL created for it?
>>>>If
>>>>not, would that not deny subsequent access to it?
>>>>
>>>>(nit) Maybe use Principal instead of String to represent principals?
>>>>
>>>>
>>>>On 3/9/15, 11:48 AM, "Don Bosco Durai" <bo...@apache.org> wrote:
>>>>
>>>>>Parth
>>>>>
>>>>>Overall it is looking good. Couple of questionsŠ
>>>>>
>>>>>- Can you give an example how the policies will look like in the
>>>>>default
>>>>>implementation?
>>>>>- In the operations, can we support ³CONNECT² also? This can be used
>>>>>during Session connection
>>>>>- Regarding access control for ³Topic Creation², since we can¹t do it
>>>>>on
>>>>>the server side, can we de-scope it for? And plan it as a future
>>>>>feature
>>>>>request?
>>>>>
>>>>>Thanks
>>>>>
>>>>>Bosco
>>>>>
>>>>> 
>>>>>
>>>>>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io> wrote:
>>>>>
>>>>>>Hi Parth,
>>>>>>            Thanks for putting this together. Overall it looks good
>>>>>>to
>>>>>>            me. Although AdminUtils is a concern KIP-4  can probably
>>>>>>fix
>>>>>>            that part.
>>>>>>Thanks,
>>>>>>Harsha
>>>>>>
>>>>>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>>>>>> Forgot to add links to wiki and jira.
>>>>>>> 
>>>>>>> Link to wiki:
>>>>>>> 
>>>>>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
>>>>>>>t
>>>>>>>i
>>>>>>>o
>>>>>>>n
>>>>>>>+
>>>>>>>Interface
>>>>>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Parth
>>>>>>> 
>>>>>>> From: Parth Brahmbhatt
>>>>>>> <pb...@hortonworks.com>>
>>>>>>> Date: Thursday, March 5, 2015 at 10:33 AM
>>>>>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org>"
>>>>>>> <de...@kafka.apache.org>>
>>>>>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>>>>>> 
>>>>>>> Hi,
>>>>>>> 
>>>>>>> KIP-11 is open for discussion , I have updated the wiki with the
>>>>>>>design
>>>>>>> and open questions.
>>>>>>> 
>>>>>>> Thanks
>>>>>>> Parth
>>>>>
>>>>>
>>>>
>>>
>>
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Don Bosco Durai <bo...@apache.org>]

I also feel, having just IP would be more appropriate. Host lookup will
unnecessary slow things down and would be insecure as you pointed out.

With IP, it will be also able to setup policies (in future if needed) with
ranges or netmasks and it would be more scalable.

Bosco


On 4/14/15, 1:40 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
wrote:

>Hi Parth,
>
>Sorry to chime in so late, but I’ve got a minor question on the KIP.
>
>Several methods take a parameter named “host” of type String. Is that
>intended to be a hostname, or an IP address? If the former, I’m curious as
>to how that’s found (in my experience, when accepting an incoming socket
>connection, you only know the IP address, and there isn’t a way to map
>that to a hostname without a round trip to a DNS server, which is insecure
>anyway).
>
>
>On 3/25/15, 1:07 PM, "Parth Brahmbhatt" <pb...@hortonworks.com>
>wrote:
>
>>Hi all,
>>
>>I have modified the KIP to reflect the recent change request from the
>>reviewers. I have been working on the code and I have the server side
>>code
>>for authorization ready. I am now modifying the command line utilities. I
>>would really appreciate if some of the committers can spend sometime to
>>review the KIP so we can make progress on this.
>>
>>Thanks
>>Parth
>>
>>On 3/18/15, 2:20 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
>>wrote:
>>
>>>Hi Parth,
>>>
>>>Thanks! A few questions:
>>>
>>>1. Do you want to permit rules in your ACLs that DENY access as well as
>>>ALLOW? This can be handy setting up rules that have exceptions. E.g.
>>>“Allow principal P to READ resource R from all hosts” with “Deny
>>>principal
>>>P READ access to resource R from host H1” in combination would allow P
>>>to
>>>READ R from all hosts *except* H1.
>>>
>>>2. When a topic is newly created, will there be an ACL created for it?
>>>If
>>>not, would that not deny subsequent access to it?
>>>
>>>(nit) Maybe use Principal instead of String to represent principals?
>>>
>>>
>>>On 3/9/15, 11:48 AM, "Don Bosco Durai" <bo...@apache.org> wrote:
>>>
>>>>Parth
>>>>
>>>>Overall it is looking good. Couple of questionsŠ
>>>>
>>>>- Can you give an example how the policies will look like in the
>>>>default
>>>>implementation?
>>>>- In the operations, can we support ³CONNECT² also? This can be used
>>>>during Session connection
>>>>- Regarding access control for ³Topic Creation², since we can¹t do it
>>>>on
>>>>the server side, can we de-scope it for? And plan it as a future
>>>>feature
>>>>request?
>>>>
>>>>Thanks
>>>>
>>>>Bosco
>>>>
>>>> 
>>>>
>>>>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io> wrote:
>>>>
>>>>>Hi Parth,
>>>>>            Thanks for putting this together. Overall it looks good to
>>>>>            me. Although AdminUtils is a concern KIP-4  can probably
>>>>>fix
>>>>>            that part.
>>>>>Thanks,
>>>>>Harsha
>>>>>
>>>>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>>>>> Forgot to add links to wiki and jira.
>>>>>> 
>>>>>> Link to wiki:
>>>>>> 
>>>>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizat
>>>>>>i
>>>>>>o
>>>>>>n
>>>>>>+
>>>>>>Interface
>>>>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>>>>> 
>>>>>> Thanks
>>>>>> Parth
>>>>>> 
>>>>>> From: Parth Brahmbhatt
>>>>>> <pb...@hortonworks.com>>
>>>>>> Date: Thursday, March 5, 2015 at 10:33 AM
>>>>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org>"
>>>>>> <de...@kafka.apache.org>>
>>>>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> KIP-11 is open for discussion , I have updated the wiki with the
>>>>>>design
>>>>>> and open questions.
>>>>>> 
>>>>>> Thanks
>>>>>> Parth
>>>>
>>>>
>>>
>>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Michael Herstine <mh...@linkedin.com.INVALID>]

Hi Parth,

Sorry to chime in so late, but I’ve got a minor question on the KIP.

Several methods take a parameter named “host” of type String. Is that
intended to be a hostname, or an IP address? If the former, I’m curious as
to how that’s found (in my experience, when accepting an incoming socket
connection, you only know the IP address, and there isn’t a way to map
that to a hostname without a round trip to a DNS server, which is insecure
anyway).


On 3/25/15, 1:07 PM, "Parth Brahmbhatt" <pb...@hortonworks.com>
wrote:

>Hi all,
>
>I have modified the KIP to reflect the recent change request from the
>reviewers. I have been working on the code and I have the server side code
>for authorization ready. I am now modifying the command line utilities. I
>would really appreciate if some of the committers can spend sometime to
>review the KIP so we can make progress on this.
>
>Thanks
>Parth
>
>On 3/18/15, 2:20 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
>wrote:
>
>>Hi Parth,
>>
>>Thanks! A few questions:
>>
>>1. Do you want to permit rules in your ACLs that DENY access as well as
>>ALLOW? This can be handy setting up rules that have exceptions. E.g.
>>“Allow principal P to READ resource R from all hosts” with “Deny
>>principal
>>P READ access to resource R from host H1” in combination would allow P to
>>READ R from all hosts *except* H1.
>>
>>2. When a topic is newly created, will there be an ACL created for it? If
>>not, would that not deny subsequent access to it?
>>
>>(nit) Maybe use Principal instead of String to represent principals?
>>
>>
>>On 3/9/15, 11:48 AM, "Don Bosco Durai" <bo...@apache.org> wrote:
>>
>>>Parth
>>>
>>>Overall it is looking good. Couple of questionsŠ
>>>
>>>- Can you give an example how the policies will look like in the default
>>>implementation?
>>>- In the operations, can we support ³CONNECT² also? This can be used
>>>during Session connection
>>>- Regarding access control for ³Topic Creation², since we can¹t do it on
>>>the server side, can we de-scope it for? And plan it as a future feature
>>>request?
>>>
>>>Thanks
>>>
>>>Bosco
>>>
>>> 
>>>
>>>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io> wrote:
>>>
>>>>Hi Parth,
>>>>            Thanks for putting this together. Overall it looks good to
>>>>            me. Although AdminUtils is a concern KIP-4  can probably
>>>>fix
>>>>            that part.
>>>>Thanks,
>>>>Harsha
>>>>
>>>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>>>> Forgot to add links to wiki and jira.
>>>>> 
>>>>> Link to wiki:
>>>>> 
>>>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizati
>>>>>o
>>>>>n
>>>>>+
>>>>>Interface
>>>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>>>> 
>>>>> Thanks
>>>>> Parth
>>>>> 
>>>>> From: Parth Brahmbhatt
>>>>> <pb...@hortonworks.com>>
>>>>> Date: Thursday, March 5, 2015 at 10:33 AM
>>>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org>"
>>>>> <de...@kafka.apache.org>>
>>>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> KIP-11 is open for discussion , I have updated the wiki with the
>>>>>design
>>>>> and open questions.
>>>>> 
>>>>> Thanks
>>>>> Parth
>>>
>>>
>>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Jun Rao <ju...@confluent.io>]

21. What you suggested makes sense. Could you include the categorization of
each request in the wiki?

24. We have a jira (KAFKA-1595) to look for a better json parser. However,
it depends on dropping the scala 2.9.x support, which is being discussed.

Thanks,

Jun

On Tue, Mar 31, 2015 at 10:56 AM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:

> Thanks for reviewing, comments inline:
>
> On 3/31/15, 9:21 AM, "Jun Rao" <ju...@confluent.io>>
> wrote:
>
> Thanks for the writeup. A few more comments.
>
> 20. I agree that it would be better to do this after KIP-4 (admin commands)
> is done. With KIP-4, all admin operations will be sent as requests to the
> brokers instead of accessing ZK directly. This will make authorization
> easier.
>
> 21. Operation: What about other types of requests not covered in the list,
> such as committing and fetching offsets, list topics, fetching consumer
> metadata, heartbeat, join group, etc?
>
>   *   I was actually considering any kind of write (like commit offset) as
> WRITE operation, and kind of read (fetching offset, get consumer metadata)
> as READ and any kind of list(list topics) as DESCRIBE. We can either create
> a one to one mapping between API and operation or classify each API as one
> of the operation. I was going with the classification but if you think one
> to one mapping will be easier to understand I am open to that.
>
> 22. TopicConfigCache: We will need such a cache in KIP-4 as well. It would
> be useful to make sure that the implementation can be reused.
>
>   *   I already opened a separate jira<
> https://issues.apache.org/jira/browse/KAFKA-2035> for this and posted a
> review<https://reviews.apache.org/r/32460/diff/#>. I plan to add Acl and
> owner as instance variables of TopicConfig class as part of authZ patch.
>
> 23. Authorizer:
> 23.1 Do cluster level operations go through authorize() too? If so, what
> will be the resource?
>
>   *   Yes and I was considering to use a constant string like
> “Kafka-Cluster” for cluster operations.
>
> 23.2 I assume that the authorize() check will be called on every request.
> So, we will have to make sure that the check is cheap.
>
>   *   Yes , that is why by design we will trade off for speed and cache
> all acls, which means if you update acls it may take a few minutes before
> the changes take effect.
>
> 24. The acl json string in the config: Should we version this so that we
> can evolve it in the future (e.g., adding group support)?
>
>   *   I am looking into this right now but this seemed like implementation
> details so I did not capture it in design. I will update the json format
> once I have settled on a solution. What are your thoughts on using some
> existing libraries that support json parsing with versioning? The current
> json encoding/decoding used by kafka is already failing for me when I try
> to parse a map that has an already json encoded string as value for some
> key.
>
> Jun
>
> On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt <
> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>> wrote:
>
> Hi Gwen,
>
> Thanks a lot for taking the time to review this. I have tried to address
> all your questions below.
>
> Thanks
> Parth
> On 3/28/15, 8:08 PM, "Gwen Shapira" <gshapira@cloudera.com<mailto:
> gshapira@cloudera.com><mailto:
> gshapira@cloudera.com<ma...@cloudera.com>>> wrote:
>
> Preparing for Tuesday meeting, I went over the KIP :)
>
> First, Parth did an amazing job, the KIP is fantastic - detailed and
> readable. Thank you!
>
> Second, I have a loooong list of questions :) No objections, just some
> things I'm unclear on and random minor comments. In general, I like
> the design, I just feel I'm missing parts of the picture.
>
> 1. "Yes, Create topic will have an optional acls, the output of
> describe will display owner and acls and alter topic will allow to
> modify the acls.”  - will be nice to see what the CLI will look like.
>
>    *   I will modify the KIP but I was going to add “—acl <acl-file.json>”
> to create-topic and alter-topic.
>
> 2. I like the addition of Topic owner. We made the mistake of
> forgetting about it when adding authorization to Sqoop2. We probably
> want to add “chown” command to the topic commands.
>
>    *   Again we can add “—owner <user-name>” to alter topic.
>
> 3. "Kafka server will read "authorizer.class” config value at startup
> time, create an instance of the specified class and call initialize
> method."
> We’ll need to validate that users specify only one of those.
>
>    *   The config type will be string so type validation should take care
> of it.
>
> 4. "One added assumption is that on non-secure connections the session
> will have principal set to an object whose name() method will return
> "Anonymous”."
> Can we keep DrWho? :)
>
>    *   Sure, its up to you actually as you are the owner of the jira that
> introduces session concept.
>
> 5. "For cluster actions that do not apply to a specific topic like
> CREATE we have 2 options. We can either add a broker config called
> broker.acls which will point to a json file. This file will be
> available on all broker hosts and authorizer will read the acls on
> initialization and keep refreshing it every X minutes. Any changes
> will require re-distribution of the acl json file. Alternatively we
> can add a zookeeper path /brokers/acls and store the acl json as data.
> Authorizer can refresh the acl from json every X minutes. In absence
> of broker acls the authorizer will fail open, in other words it will
> allow all users from all hosts to perform all cluster actions”
> I prefer a file to ZK - since thats where we store all use-defined
> configurations for now. Everyone knows how to secure a file system :)
>
>    *   I will let everyone vote, file system is fine by me.
>
> 6. "When an Acl is missing , this implementation will always fail open
> for backward compatibility. “ <- agree, but we need to document that
> this makes the default authorizer non-secure
>
>    *   Sure.
>
> 7. "If the value of authorizer.class.name is null, in secure mode the
> cluster will fail with ConfigException. In non secure mode in absence
> of config value forauthorizer.class.name the server will allow all
> requests to all topics that , even if the topic has configured acls”
> <- I don’t think Kafka has “secure mode” - it can support SSL and
> plaintext (un-authenticated) on two different ports simultaneously. I
> don’t object to adding such configuration, but we need to decide
> exactly what it means.
>
>    *   This is one area of confusion so I will add an open question.
>
> 8. "Currently all topic creation/modification/deletion actions are
> performed using KafkaAdminUtil which mostly interacts directly with
> zookeeper instead of forwarding requests to a broker host. Given all
> the code is executed on client side there is no easy way to perform
> authorization. “ - since we are adding the admin protocol requests in
> KIP-4, perhaps addressing those makes sense.
>
>    *   Yes, We will have to wait for KIP-4 to be delivered.
>
> 9. I didn’t see a specification of what is “resource”, I assume its an
> enum with things like Topic and… ?
>
>    *   Resource is not an enum, Right now its just name of the topic.
> Currently we will not be able to support cluster actions like CREATE in
> which case the resource can be some constant string like “Kafka-Cluster”.
>
> 10. I’m also unclear on where and how “PermissionType” will be used
> and what does “DENY takes precedence” mean here.
>
>    *   I originally did not have the “PermissionType” but someone suggested
> in DISCUSS thread that we should add support for ALLOW and DENY acls. The
> use case is to support acls like “Allow user U to Perform READ from all
> Hosts except from Host1 and Host2”. Deny take precedence only means if you
> have define 2 ACLs for the same user/resource/operation. DENY acl will take
> effect I.e. “Allow user X to read from *” and “Deny User X to Read from
> host1 and host2” will satisfy the use case described in the previous
> statement.
>
> 11. "What does acls on zookeeper node look like given all our admin
> APIs are currently performed directly from client?” <- “secure mode”
> Kafka will need to set ACLs on ZK (using ZK’s model of ACLs) and both
> Kafka and everyone else will need to use them (this is limited to
> Kerberos authentication AFAIK.)
>
>    *   Yes, again I think this can be a separate issue for now and can only
> be worked on after KIP-4 is delivered.
>
> 12. "Do we want to support group acls as part of this authorizer? Do
> we want to support principal to local user mapping? If yes we need to
> add plugins for UserToGroupMapper and PrincipalToUserMapper.” <-
> Sentry uses Groups for authorizing, so we need to support that. I
> figured that as long as the API specifies Principal, it typically
> contains both user and group, so nothing else is needed. Did I miss
> anything?
>
>    *   Once we support group acls we will need someway to indicate if a
> principal is of type group or user(as part of acl) or we can have group
> acls and user acls stored separately with topic config. We will also need a
> way to map an authenticated user to list of groups the user belongs to.
>
> 13. It looks like the Authorizer stores the ACLs and not Kafka. So we
> need an API for Kafka to notify Authorizer when a topic is added and
> when ACLs are modified, right? I didn’t see that.
>
>    *   ACLs will be stored under /topic/config at time of topic creation in
> json format. Authorizer will get these acls using newly introduced
> TopicConfigCache which gets updated anytime topic config changes.
>
> 14. Are we going to have any API for Kafka to give out the ACLs on a
> topic? Or we leave this to the Authorizer?
>
>    *   I think it is better to leave this out side of Kafka. Mainly because
> most 3rd party authorizers will have their own ACL stores and configuration
> Uis like (Ranger-Argus not sure what are they calling it now).
>
>
> On Wed, Mar 25, 2015 at 9:26 PM, Neha Narkhede <neha@confluent.io<mailto:
> neha@confluent.io><mailto:
> neha@confluent.io<ma...@confluent.io>>> wrote:
> Parth,
>
> We can make some 15 mins or so to discuss this at the next KIP hangout.
>
> Thanks,
> Neha
>
> On Wed, Mar 25, 2015 at 1:07 PM, Parth Brahmbhatt <
> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com><mailto:
> pbrahmbhatt@hortonworks.com>> wrote:
>
> Hi all,
>
> I have modified the KIP to reflect the recent change request from the
> reviewers. I have been working on the code and I have the server side code
> for authorization ready. I am now modifying the command line utilities. I
> would really appreciate if some of the committers can spend sometime to
> review the KIP so we can make progress on this.
>
> Thanks
> Parth
>
> On 3/18/15, 2:20 PM, "Michael Herstine" <mherstine@linkedin.com.INVALID
> <ma...@linkedin.com.INVALID>
> <ma...@linkedin.com.INVALID>>
> wrote:
>
> >Hi Parth,
> >
> >Thanks! A few questions:
> >
> >1. Do you want to permit rules in your ACLs that DENY access as well as
> >ALLOW? This can be handy setting up rules that have exceptions. E.g.
> >“Allow principal P to READ resource R from all hosts” with “Deny principal
> >P READ access to resource R from host H1” in combination would allow P to
> >READ R from all hosts *except* H1.
> >
> >2. When a topic is newly created, will there be an ACL created for it? If
> >not, would that not deny subsequent access to it?
> >
> >(nit) Maybe use Principal instead of String to represent principals?
> >
> >
> >On 3/9/15, 11:48 AM, "Don Bosco Durai" <bosco@apache.org<mailto:
> bosco@apache.org><mailto:
> bosco@apache.org<ma...@apache.org>>> wrote:
> >
> >>Parth
> >>
> >>Overall it is looking good. Couple of questionsŠ
> >>
> >>- Can you give an example how the policies will look like in the default
> >>implementation?
> >>- In the operations, can we support ³CONNECT² also? This can be used
> >>during Session connection
> >>- Regarding access control for ³Topic Creation², since we can¹t do it on
> >>the server side, can we de-scope it for? And plan it as a future feature
> >>request?
> >>
> >>Thanks
> >>
> >>Bosco
> >>
> >>
> >>
> >>On 3/6/15, 8:10 AM, "Harsha" <kafka@harsha.io<mailto:kafka@harsha.io
> ><ma...@harsha.io>>
> wrote:
> >>
> >>>Hi Parth,
> >>>            Thanks for putting this together. Overall it looks good to
> >>>            me. Although AdminUtils is a concern KIP-4  can probably fix
> >>>            that part.
> >>>Thanks,
> >>>Harsha
> >>>
> >>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
> >>>> Forgot to add links to wiki and jira.
> >>>>
> >>>> Link to wiki:
> >>>>
> >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizatio
> >>>>n
> >>>>+
> >>>>Interface
> >>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
> >>>>
> >>>> Thanks
> >>>> Parth
> >>>>
> >>>> From: Parth Brahmbhatt
> >>>> <pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
> ><mailto:pbrahmbhatt@hortonworks.com
> ><ma...@hortonworks.com>>
> >>>> Date: Thursday, March 5, 2015 at 10:33 AM
> >>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org><mailto:
> dev@kafka.apache.org><mailto:
> dev@kafka.apache.org<ma...@kafka.apache.org>>"
> >>>> <de...@kafka.apache.org><mailto:
> dev@kafka.apache.org><mailto:
> dev@kafka.apache.org<ma...@kafka.apache.org>>>
> >>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
> >>>>
> >>>> Hi,
> >>>>
> >>>> KIP-11 is open for discussion , I have updated the wiki with the
> >>>>design
> >>>> and open questions.
> >>>>
> >>>> Thanks
> >>>> Parth
> >>
> >>
> >
>
>
>
>
> --
> Thanks,
> Neha
>
>
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Thanks for reviewing, comments inline:

On 3/31/15, 9:21 AM, "Jun Rao" <ju...@confluent.io>> wrote:

Thanks for the writeup. A few more comments.

20. I agree that it would be better to do this after KIP-4 (admin commands)
is done. With KIP-4, all admin operations will be sent as requests to the
brokers instead of accessing ZK directly. This will make authorization
easier.

21. Operation: What about other types of requests not covered in the list,
such as committing and fetching offsets, list topics, fetching consumer
metadata, heartbeat, join group, etc?

  *   I was actually considering any kind of write (like commit offset) as WRITE operation, and kind of read (fetching offset, get consumer metadata) as READ and any kind of list(list topics) as DESCRIBE. We can either create a one to one mapping between API and operation or classify each API as one of the operation. I was going with the classification but if you think one to one mapping will be easier to understand I am open to that.

22. TopicConfigCache: We will need such a cache in KIP-4 as well. It would
be useful to make sure that the implementation can be reused.

  *   I already opened a separate jira<https://issues.apache.org/jira/browse/KAFKA-2035> for this and posted a review<https://reviews.apache.org/r/32460/diff/#>. I plan to add Acl and owner as instance variables of TopicConfig class as part of authZ patch.

23. Authorizer:
23.1 Do cluster level operations go through authorize() too? If so, what
will be the resource?

  *   Yes and I was considering to use a constant string like “Kafka-Cluster” for cluster operations.

23.2 I assume that the authorize() check will be called on every request.
So, we will have to make sure that the check is cheap.

  *   Yes , that is why by design we will trade off for speed and cache all acls, which means if you update acls it may take a few minutes before the changes take effect.

24. The acl json string in the config: Should we version this so that we
can evolve it in the future (e.g., adding group support)?

  *   I am looking into this right now but this seemed like implementation details so I did not capture it in design. I will update the json format once I have settled on a solution. What are your thoughts on using some existing libraries that support json parsing with versioning? The current json encoding/decoding used by kafka is already failing for me when I try to parse a map that has an already json encoded string as value for some key.

Jun

On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>> wrote:

Hi Gwen,

Thanks a lot for taking the time to review this. I have tried to address
all your questions below.

Thanks
Parth
On 3/28/15, 8:08 PM, "Gwen Shapira" <gs...@cloudera.com><mailto:
gshapira@cloudera.com<ma...@cloudera.com>>> wrote:

Preparing for Tuesday meeting, I went over the KIP :)

First, Parth did an amazing job, the KIP is fantastic - detailed and
readable. Thank you!

Second, I have a loooong list of questions :) No objections, just some
things I'm unclear on and random minor comments. In general, I like
the design, I just feel I'm missing parts of the picture.

1. "Yes, Create topic will have an optional acls, the output of
describe will display owner and acls and alter topic will allow to
modify the acls.”  - will be nice to see what the CLI will look like.

   *   I will modify the KIP but I was going to add “—acl <acl-file.json>”
to create-topic and alter-topic.

2. I like the addition of Topic owner. We made the mistake of
forgetting about it when adding authorization to Sqoop2. We probably
want to add “chown” command to the topic commands.

   *   Again we can add “—owner <user-name>” to alter topic.

3. "Kafka server will read "authorizer.class” config value at startup
time, create an instance of the specified class and call initialize
method."
We’ll need to validate that users specify only one of those.

   *   The config type will be string so type validation should take care
of it.

4. "One added assumption is that on non-secure connections the session
will have principal set to an object whose name() method will return
"Anonymous”."
Can we keep DrWho? :)

   *   Sure, its up to you actually as you are the owner of the jira that
introduces session concept.

5. "For cluster actions that do not apply to a specific topic like
CREATE we have 2 options. We can either add a broker config called
broker.acls which will point to a json file. This file will be
available on all broker hosts and authorizer will read the acls on
initialization and keep refreshing it every X minutes. Any changes
will require re-distribution of the acl json file. Alternatively we
can add a zookeeper path /brokers/acls and store the acl json as data.
Authorizer can refresh the acl from json every X minutes. In absence
of broker acls the authorizer will fail open, in other words it will
allow all users from all hosts to perform all cluster actions”
I prefer a file to ZK - since thats where we store all use-defined
configurations for now. Everyone knows how to secure a file system :)

   *   I will let everyone vote, file system is fine by me.

6. "When an Acl is missing , this implementation will always fail open
for backward compatibility. “ <- agree, but we need to document that
this makes the default authorizer non-secure

   *   Sure.

7. "If the value of authorizer.class.name is null, in secure mode the
cluster will fail with ConfigException. In non secure mode in absence
of config value forauthorizer.class.name the server will allow all
requests to all topics that , even if the topic has configured acls”
<- I don’t think Kafka has “secure mode” - it can support SSL and
plaintext (un-authenticated) on two different ports simultaneously. I
don’t object to adding such configuration, but we need to decide
exactly what it means.

   *   This is one area of confusion so I will add an open question.

8. "Currently all topic creation/modification/deletion actions are
performed using KafkaAdminUtil which mostly interacts directly with
zookeeper instead of forwarding requests to a broker host. Given all
the code is executed on client side there is no easy way to perform
authorization. “ - since we are adding the admin protocol requests in
KIP-4, perhaps addressing those makes sense.

   *   Yes, We will have to wait for KIP-4 to be delivered.

9. I didn’t see a specification of what is “resource”, I assume its an
enum with things like Topic and… ?

   *   Resource is not an enum, Right now its just name of the topic.
Currently we will not be able to support cluster actions like CREATE in
which case the resource can be some constant string like “Kafka-Cluster”.

10. I’m also unclear on where and how “PermissionType” will be used
and what does “DENY takes precedence” mean here.

   *   I originally did not have the “PermissionType” but someone suggested
in DISCUSS thread that we should add support for ALLOW and DENY acls. The
use case is to support acls like “Allow user U to Perform READ from all
Hosts except from Host1 and Host2”. Deny take precedence only means if you
have define 2 ACLs for the same user/resource/operation. DENY acl will take
effect I.e. “Allow user X to read from *” and “Deny User X to Read from
host1 and host2” will satisfy the use case described in the previous
statement.

11. "What does acls on zookeeper node look like given all our admin
APIs are currently performed directly from client?” <- “secure mode”
Kafka will need to set ACLs on ZK (using ZK’s model of ACLs) and both
Kafka and everyone else will need to use them (this is limited to
Kerberos authentication AFAIK.)

   *   Yes, again I think this can be a separate issue for now and can only
be worked on after KIP-4 is delivered.

12. "Do we want to support group acls as part of this authorizer? Do
we want to support principal to local user mapping? If yes we need to
add plugins for UserToGroupMapper and PrincipalToUserMapper.” <-
Sentry uses Groups for authorizing, so we need to support that. I
figured that as long as the API specifies Principal, it typically
contains both user and group, so nothing else is needed. Did I miss
anything?

   *   Once we support group acls we will need someway to indicate if a
principal is of type group or user(as part of acl) or we can have group
acls and user acls stored separately with topic config. We will also need a
way to map an authenticated user to list of groups the user belongs to.

13. It looks like the Authorizer stores the ACLs and not Kafka. So we
need an API for Kafka to notify Authorizer when a topic is added and
when ACLs are modified, right? I didn’t see that.

   *   ACLs will be stored under /topic/config at time of topic creation in
json format. Authorizer will get these acls using newly introduced
TopicConfigCache which gets updated anytime topic config changes.

14. Are we going to have any API for Kafka to give out the ACLs on a
topic? Or we leave this to the Authorizer?

   *   I think it is better to leave this out side of Kafka. Mainly because
most 3rd party authorizers will have their own ACL stores and configuration
Uis like (Ranger-Argus not sure what are they calling it now).


On Wed, Mar 25, 2015 at 9:26 PM, Neha Narkhede <ne...@confluent.io><mailto:
neha@confluent.io<ma...@confluent.io>>> wrote:
Parth,

We can make some 15 mins or so to discuss this at the next KIP hangout.

Thanks,
Neha

On Wed, Mar 25, 2015 at 1:07 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>> wrote:

Hi all,

I have modified the KIP to reflect the recent change request from the
reviewers. I have been working on the code and I have the server side code
for authorization ready. I am now modifying the command line utilities. I
would really appreciate if some of the committers can spend sometime to
review the KIP so we can make progress on this.

Thanks
Parth

On 3/18/15, 2:20 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
<ma...@linkedin.com.INVALID>>
wrote:

>Hi Parth,
>
>Thanks! A few questions:
>
>1. Do you want to permit rules in your ACLs that DENY access as well as
>ALLOW? This can be handy setting up rules that have exceptions. E.g.
>“Allow principal P to READ resource R from all hosts” with “Deny principal
>P READ access to resource R from host H1” in combination would allow P to
>READ R from all hosts *except* H1.
>
>2. When a topic is newly created, will there be an ACL created for it? If
>not, would that not deny subsequent access to it?
>
>(nit) Maybe use Principal instead of String to represent principals?
>
>
>On 3/9/15, 11:48 AM, "Don Bosco Durai" <bo...@apache.org><mailto:
bosco@apache.org<ma...@apache.org>>> wrote:
>
>>Parth
>>
>>Overall it is looking good. Couple of questionsŠ
>>
>>- Can you give an example how the policies will look like in the default
>>implementation?
>>- In the operations, can we support ³CONNECT² also? This can be used
>>during Session connection
>>- Regarding access control for ³Topic Creation², since we can¹t do it on
>>the server side, can we de-scope it for? And plan it as a future feature
>>request?
>>
>>Thanks
>>
>>Bosco
>>
>>
>>
>>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io>>
wrote:
>>
>>>Hi Parth,
>>>            Thanks for putting this together. Overall it looks good to
>>>            me. Although AdminUtils is a concern KIP-4  can probably fix
>>>            that part.
>>>Thanks,
>>>Harsha
>>>
>>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>>> Forgot to add links to wiki and jira.
>>>>
>>>> Link to wiki:
>>>>
>>>>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizatio
>>>>n
>>>>+
>>>>Interface
>>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>>>
>>>> Thanks
>>>> Parth
>>>>
>>>> From: Parth Brahmbhatt
>>>> <pb...@hortonworks.com><mailto:pbrahmbhatt@hortonworks.com
><ma...@hortonworks.com>>
>>>> Date: Thursday, March 5, 2015 at 10:33 AM
>>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org><mailto:
dev@kafka.apache.org<ma...@kafka.apache.org>>"
>>>> <de...@kafka.apache.org><mailto:
dev@kafka.apache.org<ma...@kafka.apache.org>>>
>>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>>>
>>>> Hi,
>>>>
>>>> KIP-11 is open for discussion , I have updated the wiki with the
>>>>design
>>>> and open questions.
>>>>
>>>> Thanks
>>>> Parth
>>
>>
>




--
Thanks,
Neha

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

From the design doc , one of the added config:

* kafka.superusers: list of users that will be given superuser access.
These users will have access to everything. Users should set this to the
user kafka broker processes are running as to avoid duplicate
configuration for every single topic like ALLOW REPLICATION to BROKER_USER
for TOPIC from ALL hosts.

Thanks

Parth

On 3/31/15, 10:20 AM, "Gwen Shapira" <gs...@cloudera.com> wrote:

>Related interesting question:
>Since a broker is a consumer (of lead replicas), how do we handle the
>broker level of permissions? Do we hardcode a broker-principal name
>and automatically authorize brokers to do anything? Or is there a
>cleaner way?
>
>
>On Tue, Mar 31, 2015 at 10:17 AM, Don Bosco Durai <bo...@apache.org>
>wrote:
>>>21. Operation: What about other types of requests not covered in the
>>>list,
>> such as committing and fetching offsets, list topics, fetching consumer
>> metadata, heartbeat, join group, etc?
>>
>> Would “CONFIGURE”, “DESCRIBE”, etc take care of this? Or should we add
>> high level grouping like “ADMIN”, “OPERATIONS/MANAGEMENT” to cover
>>related
>> permissions?
>>
>> Bosco
>>
>>
>>
>> On 3/31/15, 9:21 AM, "Jun Rao" <ju...@confluent.io> wrote:
>>
>>>Thanks for the writeup. A few more comments.
>>>
>>>20. I agree that it would be better to do this after KIP-4 (admin
>>>commands)
>>>is done. With KIP-4, all admin operations will be sent as requests to
>>>the
>>>brokers instead of accessing ZK directly. This will make authorization
>>>easier.
>>>
>>>21. Operation: What about other types of requests not covered in the
>>>list,
>>>such as committing and fetching offsets, list topics, fetching consumer
>>>metadata, heartbeat, join group, etc?
>>>
>>>22. TopicConfigCache: We will need such a cache in KIP-4 as well. It
>>>would
>>>be useful to make sure that the implementation can be reused.
>>>
>>>23. Authorizer:
>>>23.1 Do cluster level operations go through authorize() too? If so, what
>>>will be the resource?
>>>23.2 I assume that the authorize() check will be called on every
>>>request.
>>>So, we will have to make sure that the check is cheap.
>>>
>>>24. The acl json string in the config: Should we version this so that we
>>>can evolve it in the future (e.g., adding group support)?
>>>
>>>Jun
>>>
>>>On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt <
>>>pbrahmbhatt@hortonworks.com> wrote:
>>>
>>>> Hi Gwen,
>>>>
>>>> Thanks a lot for taking the time to review this. I have tried to
>>>>address
>>>> all your questions below.
>>>>
>>>> Thanks
>>>> Parth
>>>> On 3/28/15, 8:08 PM, "Gwen Shapira" <gshapira@cloudera.com<mailto:
>>>> gshapira@cloudera.com>> wrote:
>>>>
>>>> Preparing for Tuesday meeting, I went over the KIP :)
>>>>
>>>> First, Parth did an amazing job, the KIP is fantastic - detailed and
>>>> readable. Thank you!
>>>>
>>>> Second, I have a loooong list of questions :) No objections, just some
>>>> things I'm unclear on and random minor comments. In general, I like
>>>> the design, I just feel I'm missing parts of the picture.
>>>>
>>>> 1. "Yes, Create topic will have an optional acls, the output of
>>>> describe will display owner and acls and alter topic will allow to
>>>> modify the acls.”  - will be nice to see what the CLI will look like.
>>>>
>>>>   *   I will modify the KIP but I was going to add “—acl
>>>><acl-file.json>”
>>>> to create-topic and alter-topic.
>>>>
>>>> 2. I like the addition of Topic owner. We made the mistake of
>>>> forgetting about it when adding authorization to Sqoop2. We probably
>>>> want to add “chown” command to the topic commands.
>>>>
>>>>   *   Again we can add “—owner <user-name>” to alter topic.
>>>>
>>>> 3. "Kafka server will read "authorizer.class” config value at startup
>>>> time, create an instance of the specified class and call initialize
>>>> method."
>>>> We’ll need to validate that users specify only one of those.
>>>>
>>>>   *   The config type will be string so type validation should take
>>>>care
>>>> of it.
>>>>
>>>> 4. "One added assumption is that on non-secure connections the session
>>>> will have principal set to an object whose name() method will return
>>>> "Anonymous”."
>>>> Can we keep DrWho? :)
>>>>
>>>>   *   Sure, its up to you actually as you are the owner of the jira
>>>>that
>>>> introduces session concept.
>>>>
>>>> 5. "For cluster actions that do not apply to a specific topic like
>>>> CREATE we have 2 options. We can either add a broker config called
>>>> broker.acls which will point to a json file. This file will be
>>>> available on all broker hosts and authorizer will read the acls on
>>>> initialization and keep refreshing it every X minutes. Any changes
>>>> will require re-distribution of the acl json file. Alternatively we
>>>> can add a zookeeper path /brokers/acls and store the acl json as data.
>>>> Authorizer can refresh the acl from json every X minutes. In absence
>>>> of broker acls the authorizer will fail open, in other words it will
>>>> allow all users from all hosts to perform all cluster actions”
>>>> I prefer a file to ZK - since thats where we store all use-defined
>>>> configurations for now. Everyone knows how to secure a file system :)
>>>>
>>>>   *   I will let everyone vote, file system is fine by me.
>>>>
>>>> 6. "When an Acl is missing , this implementation will always fail open
>>>> for backward compatibility. “ <- agree, but we need to document that
>>>> this makes the default authorizer non-secure
>>>>
>>>>   *   Sure.
>>>>
>>>> 7. "If the value of authorizer.class.name is null, in secure mode the
>>>> cluster will fail with ConfigException. In non secure mode in absence
>>>> of config value forauthorizer.class.name the server will allow all
>>>> requests to all topics that , even if the topic has configured acls”
>>>> <- I don’t think Kafka has “secure mode” - it can support SSL and
>>>> plaintext (un-authenticated) on two different ports simultaneously. I
>>>> don’t object to adding such configuration, but we need to decide
>>>> exactly what it means.
>>>>
>>>>   *   This is one area of confusion so I will add an open question.
>>>>
>>>> 8. "Currently all topic creation/modification/deletion actions are
>>>> performed using KafkaAdminUtil which mostly interacts directly with
>>>> zookeeper instead of forwarding requests to a broker host. Given all
>>>> the code is executed on client side there is no easy way to perform
>>>> authorization. “ - since we are adding the admin protocol requests in
>>>> KIP-4, perhaps addressing those makes sense.
>>>>
>>>>   *   Yes, We will have to wait for KIP-4 to be delivered.
>>>>
>>>> 9. I didn’t see a specification of what is “resource”, I assume its an
>>>> enum with things like Topic and… ?
>>>>
>>>>   *   Resource is not an enum, Right now its just name of the topic.
>>>> Currently we will not be able to support cluster actions like CREATE
>>>>in
>>>> which case the resource can be some constant string like
>>>>“Kafka-Cluster”.
>>>>
>>>> 10. I’m also unclear on where and how “PermissionType” will be used
>>>> and what does “DENY takes precedence” mean here.
>>>>
>>>>   *   I originally did not have the “PermissionType” but someone
>>>>suggested
>>>> in DISCUSS thread that we should add support for ALLOW and DENY acls.
>>>>The
>>>> use case is to support acls like “Allow user U to Perform READ from
>>>>all
>>>> Hosts except from Host1 and Host2”. Deny take precedence only means if
>>>>you
>>>> have define 2 ACLs for the same user/resource/operation. DENY acl will
>>>>take
>>>> effect I.e. “Allow user X to read from *” and “Deny User X to Read
>>>>from
>>>> host1 and host2” will satisfy the use case described in the previous
>>>> statement.
>>>>
>>>> 11. "What does acls on zookeeper node look like given all our admin
>>>> APIs are currently performed directly from client?” <- “secure mode”
>>>> Kafka will need to set ACLs on ZK (using ZK’s model of ACLs) and both
>>>> Kafka and everyone else will need to use them (this is limited to
>>>> Kerberos authentication AFAIK.)
>>>>
>>>>   *   Yes, again I think this can be a separate issue for now and can
>>>>only
>>>> be worked on after KIP-4 is delivered.
>>>>
>>>> 12. "Do we want to support group acls as part of this authorizer? Do
>>>> we want to support principal to local user mapping? If yes we need to
>>>> add plugins for UserToGroupMapper and PrincipalToUserMapper.” <-
>>>> Sentry uses Groups for authorizing, so we need to support that. I
>>>> figured that as long as the API specifies Principal, it typically
>>>> contains both user and group, so nothing else is needed. Did I miss
>>>> anything?
>>>>
>>>>   *   Once we support group acls we will need someway to indicate if a
>>>> principal is of type group or user(as part of acl) or we can have
>>>>group
>>>> acls and user acls stored separately with topic config. We will also
>>>>need a
>>>> way to map an authenticated user to list of groups the user belongs
>>>>to.
>>>>
>>>> 13. It looks like the Authorizer stores the ACLs and not Kafka. So we
>>>> need an API for Kafka to notify Authorizer when a topic is added and
>>>> when ACLs are modified, right? I didn’t see that.
>>>>
>>>>   *   ACLs will be stored under /topic/config at time of topic
>>>>creation
>>>>in
>>>> json format. Authorizer will get these acls using newly introduced
>>>> TopicConfigCache which gets updated anytime topic config changes.
>>>>
>>>> 14. Are we going to have any API for Kafka to give out the ACLs on a
>>>> topic? Or we leave this to the Authorizer?
>>>>
>>>>   *   I think it is better to leave this out side of Kafka. Mainly
>>>>because
>>>> most 3rd party authorizers will have their own ACL stores and
>>>>configuration
>>>> Uis like (Ranger-Argus not sure what are they calling it now).
>>>>
>>>>
>>>> On Wed, Mar 25, 2015 at 9:26 PM, Neha Narkhede
>>>><neha@confluent.io<mailto:
>>>> neha@confluent.io>> wrote:
>>>> Parth,
>>>>
>>>> We can make some 15 mins or so to discuss this at the next KIP
>>>>hangout.
>>>>
>>>> Thanks,
>>>> Neha
>>>>
>>>> On Wed, Mar 25, 2015 at 1:07 PM, Parth Brahmbhatt <
>>>> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
>>>>wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I have modified the KIP to reflect the recent change request from the
>>>> reviewers. I have been working on the code and I have the server side
>>>>code
>>>> for authorization ready. I am now modifying the command line
>>>>utilities.
>>>>I
>>>> would really appreciate if some of the committers can spend sometime
>>>>to
>>>> review the KIP so we can make progress on this.
>>>>
>>>> Thanks
>>>> Parth
>>>>
>>>> On 3/18/15, 2:20 PM, "Michael Herstine"
>>>><mherstine@linkedin.com.INVALID
>>>> <ma...@linkedin.com.INVALID>>
>>>> wrote:
>>>>
>>>> >Hi Parth,
>>>> >
>>>> >Thanks! A few questions:
>>>> >
>>>> >1. Do you want to permit rules in your ACLs that DENY access as well
>>>>as
>>>> >ALLOW? This can be handy setting up rules that have exceptions. E.g.
>>>> >“Allow principal P to READ resource R from all hosts” with “Deny
>>>>principal
>>>> >P READ access to resource R from host H1” in combination would allow
>>>>P
>>>>to
>>>> >READ R from all hosts *except* H1.
>>>> >
>>>> >2. When a topic is newly created, will there be an ACL created for
>>>>it?
>>>>If
>>>> >not, would that not deny subsequent access to it?
>>>> >
>>>> >(nit) Maybe use Principal instead of String to represent principals?
>>>> >
>>>> >
>>>> >On 3/9/15, 11:48 AM, "Don Bosco Durai" <bosco@apache.org<mailto:
>>>> bosco@apache.org>> wrote:
>>>> >
>>>> >>Parth
>>>> >>
>>>> >>Overall it is looking good. Couple of questionsŠ
>>>> >>
>>>> >>- Can you give an example how the policies will look like in the
>>>>default
>>>> >>implementation?
>>>> >>- In the operations, can we support ³CONNECT² also? This can be used
>>>> >>during Session connection
>>>> >>- Regarding access control for ³Topic Creation², since we can¹t do
>>>>it
>>>>on
>>>> >>the server side, can we de-scope it for? And plan it as a future
>>>>feature
>>>> >>request?
>>>> >>
>>>> >>Thanks
>>>> >>
>>>> >>Bosco
>>>> >>
>>>> >>
>>>> >>
>>>> >>On 3/6/15, 8:10 AM, "Harsha"
>>>><ka...@harsha.io>>
>>>> wrote:
>>>> >>
>>>> >>>Hi Parth,
>>>> >>>            Thanks for putting this together. Overall it looks good
>>>>to
>>>> >>>            me. Although AdminUtils is a concern KIP-4  can
>>>>probably
>>>>fix
>>>> >>>            that part.
>>>> >>>Thanks,
>>>> >>>Harsha
>>>> >>>
>>>> >>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>>> >>>> Forgot to add links to wiki and jira.
>>>> >>>>
>>>> >>>> Link to wiki:
>>>> >>>>
>>>> >>>>
>>>> 
>>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizatio
>>>> >>>>n
>>>> >>>>+
>>>> >>>>Interface
>>>> >>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>>> >>>>
>>>> >>>> Thanks
>>>> >>>> Parth
>>>> >>>>
>>>> >>>> From: Parth Brahmbhatt
>>>> >>>> <pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
>>>> ><ma...@hortonworks.com>>
>>>> >>>> Date: Thursday, March 5, 2015 at 10:33 AM
>>>> >>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org><mailto:
>>>> dev@kafka.apache.org>"
>>>> >>>> <de...@kafka.apache.org><mailto:
>>>> dev@kafka.apache.org>>
>>>> >>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka
>>>>security
>>>> >>>>
>>>> >>>> Hi,
>>>> >>>>
>>>> >>>> KIP-11 is open for discussion , I have updated the wiki with the
>>>> >>>>design
>>>> >>>> and open questions.
>>>> >>>>
>>>> >>>> Thanks
>>>> >>>> Parth
>>>> >>
>>>> >>
>>>> >
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Neha
>>>>
>>>>
>>
>>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Harsha <ka...@harsha.io>]

Yes in case of kerberos we will use superACL and this will be equivalent to kafka broker’s principal name.
But in SSL as two-way auth is not mandatory the only option if we want enforce authorizer in case of ssl is to force two-way auth.
Again this can be an issue on client side , lets say if a producer doesn’t want to provide client auth and just needs wire encryption there won’t be any
identity , in this case and we won’t be able to enforce an authorizer as the client will be anonymous.

-- 
Harsha


On March 31, 2015 at 10:29:33 AM, Don Bosco Durai (bosco@apache.org) wrote:

>Related interesting question:  
Since a broker is a consumer (of lead replicas), how do we handle the  
broker level of permissions? Do we hardcode a broker-principal name  
and automatically authorize brokers to do anything? Or is there a  
cleaner way?  

I feel, in Kerberos environment, “kafka” keytab would be the ideal  
solution. And “kafka” principal will need to be white listed. SSL  
certificate is another option, but it would be painful to set it up. IP  
whitelisting is another low impact, but less secure option.  

Bosco  



On 3/31/15, 10:20 AM, "Gwen Shapira" <gs...@cloudera.com> wrote:  

>Related interesting question:  
>Since a broker is a consumer (of lead replicas), how do we handle the  
>broker level of permissions? Do we hardcode a broker-principal name  
>and automatically authorize brokers to do anything? Or is there a  
>cleaner way?  
>  
>  
>On Tue, Mar 31, 2015 at 10:17 AM, Don Bosco Durai <bo...@apache.org>  
>wrote:  
>>>21. Operation: What about other types of requests not covered in the  
>>>list,  
>> such as committing and fetching offsets, list topics, fetching consumer  
>> metadata, heartbeat, join group, etc?  
>>  
>> Would “CONFIGURE”, “DESCRIBE”, etc take care of this? Or should we add  
>> high level grouping like “ADMIN”, “OPERATIONS/MANAGEMENT” to cover  
>>related  
>> permissions?  
>>  
>> Bosco  
>>  
>>  
>>  
>> On 3/31/15, 9:21 AM, "Jun Rao" <ju...@confluent.io> wrote:  
>>  
>>>Thanks for the writeup. A few more comments.  
>>>  
>>>20. I agree that it would be better to do this after KIP-4 (admin  
>>>commands)  
>>>is done. With KIP-4, all admin operations will be sent as requests to  
>>>the  
>>>brokers instead of accessing ZK directly. This will make authorization  
>>>easier.  
>>>  
>>>21. Operation: What about other types of requests not covered in the  
>>>list,  
>>>such as committing and fetching offsets, list topics, fetching consumer  
>>>metadata, heartbeat, join group, etc?  
>>>  
>>>22. TopicConfigCache: We will need such a cache in KIP-4 as well. It  
>>>would  
>>>be useful to make sure that the implementation can be reused.  
>>>  
>>>23. Authorizer:  
>>>23.1 Do cluster level operations go through authorize() too? If so, what  
>>>will be the resource?  
>>>23.2 I assume that the authorize() check will be called on every  
>>>request.  
>>>So, we will have to make sure that the check is cheap.  
>>>  
>>>24. The acl json string in the config: Should we version this so that we  
>>>can evolve it in the future (e.g., adding group support)?  
>>>  
>>>Jun  
>>>  
>>>On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt <  
>>>pbrahmbhatt@hortonworks.com> wrote:  
>>>  
>>>> Hi Gwen,  
>>>>  
>>>> Thanks a lot for taking the time to review this. I have tried to  
>>>>address  
>>>> all your questions below.  
>>>>  
>>>> Thanks  
>>>> Parth  
>>>> On 3/28/15, 8:08 PM, "Gwen Shapira" <gshapira@cloudera.com<mailto:  
>>>> gshapira@cloudera.com>> wrote:  
>>>>  
>>>> Preparing for Tuesday meeting, I went over the KIP :)  
>>>>  
>>>> First, Parth did an amazing job, the KIP is fantastic - detailed and  
>>>> readable. Thank you!  
>>>>  
>>>> Second, I have a loooong list of questions :) No objections, just some  
>>>> things I'm unclear on and random minor comments. In general, I like  
>>>> the design, I just feel I'm missing parts of the picture.  
>>>>  
>>>> 1. "Yes, Create topic will have an optional acls, the output of  
>>>> describe will display owner and acls and alter topic will allow to  
>>>> modify the acls.” - will be nice to see what the CLI will look like.  
>>>>  
>>>> * I will modify the KIP but I was going to add “—acl  
>>>><acl-file.json>”  
>>>> to create-topic and alter-topic.  
>>>>  
>>>> 2. I like the addition of Topic owner. We made the mistake of  
>>>> forgetting about it when adding authorization to Sqoop2. We probably  
>>>> want to add “chown” command to the topic commands.  
>>>>  
>>>> * Again we can add “—owner <user-name>” to alter topic.  
>>>>  
>>>> 3. "Kafka server will read "authorizer.class” config value at startup  
>>>> time, create an instance of the specified class and call initialize  
>>>> method."  
>>>> We’ll need to validate that users specify only one of those.  
>>>>  
>>>> * The config type will be string so type validation should take  
>>>>care  
>>>> of it.  
>>>>  
>>>> 4. "One added assumption is that on non-secure connections the session  
>>>> will have principal set to an object whose name() method will return  
>>>> "Anonymous”."  
>>>> Can we keep DrWho? :)  
>>>>  
>>>> * Sure, its up to you actually as you are the owner of the jira  
>>>>that  
>>>> introduces session concept.  
>>>>  
>>>> 5. "For cluster actions that do not apply to a specific topic like  
>>>> CREATE we have 2 options. We can either add a broker config called  
>>>> broker.acls which will point to a json file. This file will be  
>>>> available on all broker hosts and authorizer will read the acls on  
>>>> initialization and keep refreshing it every X minutes. Any changes  
>>>> will require re-distribution of the acl json file. Alternatively we  
>>>> can add a zookeeper path /brokers/acls and store the acl json as data.  
>>>> Authorizer can refresh the acl from json every X minutes. In absence  
>>>> of broker acls the authorizer will fail open, in other words it will  
>>>> allow all users from all hosts to perform all cluster actions”  
>>>> I prefer a file to ZK - since thats where we store all use-defined  
>>>> configurations for now. Everyone knows how to secure a file system :)  
>>>>  
>>>> * I will let everyone vote, file system is fine by me.  
>>>>  
>>>> 6. "When an Acl is missing , this implementation will always fail open  
>>>> for backward compatibility. “ <- agree, but we need to document that  
>>>> this makes the default authorizer non-secure  
>>>>  
>>>> * Sure.  
>>>>  
>>>> 7. "If the value of authorizer.class.name is null, in secure mode the  
>>>> cluster will fail with ConfigException. In non secure mode in absence  
>>>> of config value forauthorizer.class.name the server will allow all  
>>>> requests to all topics that , even if the topic has configured acls”  
>>>> <- I don’t think Kafka has “secure mode” - it can support SSL and  
>>>> plaintext (un-authenticated) on two different ports simultaneously. I  
>>>> don’t object to adding such configuration, but we need to decide  
>>>> exactly what it means.  
>>>>  
>>>> * This is one area of confusion so I will add an open question.  
>>>>  
>>>> 8. "Currently all topic creation/modification/deletion actions are  
>>>> performed using KafkaAdminUtil which mostly interacts directly with  
>>>> zookeeper instead of forwarding requests to a broker host. Given all  
>>>> the code is executed on client side there is no easy way to perform  
>>>> authorization. “ - since we are adding the admin protocol requests in  
>>>> KIP-4, perhaps addressing those makes sense.  
>>>>  
>>>> * Yes, We will have to wait for KIP-4 to be delivered.  
>>>>  
>>>> 9. I didn’t see a specification of what is “resource”, I assume its an  
>>>> enum with things like Topic and… ?  
>>>>  
>>>> * Resource is not an enum, Right now its just name of the topic.  
>>>> Currently we will not be able to support cluster actions like CREATE  
>>>>in  
>>>> which case the resource can be some constant string like  
>>>>“Kafka-Cluster”.  
>>>>  
>>>> 10. I’m also unclear on where and how “PermissionType” will be used  
>>>> and what does “DENY takes precedence” mean here.  
>>>>  
>>>> * I originally did not have the “PermissionType” but someone  
>>>>suggested  
>>>> in DISCUSS thread that we should add support for ALLOW and DENY acls.  
>>>>The  
>>>> use case is to support acls like “Allow user U to Perform READ from  
>>>>all  
>>>> Hosts except from Host1 and Host2”. Deny take precedence only means if  
>>>>you  
>>>> have define 2 ACLs for the same user/resource/operation. DENY acl will  
>>>>take  
>>>> effect I.e. “Allow user X to read from *” and “Deny User X to Read  
>>>>from  
>>>> host1 and host2” will satisfy the use case described in the previous  
>>>> statement.  
>>>>  
>>>> 11. "What does acls on zookeeper node look like given all our admin  
>>>> APIs are currently performed directly from client?” <- “secure mode”  
>>>> Kafka will need to set ACLs on ZK (using ZK’s model of ACLs) and both  
>>>> Kafka and everyone else will need to use them (this is limited to  
>>>> Kerberos authentication AFAIK.)  
>>>>  
>>>> * Yes, again I think this can be a separate issue for now and can  
>>>>only  
>>>> be worked on after KIP-4 is delivered.  
>>>>  
>>>> 12. "Do we want to support group acls as part of this authorizer? Do  
>>>> we want to support principal to local user mapping? If yes we need to  
>>>> add plugins for UserToGroupMapper and PrincipalToUserMapper.” <-  
>>>> Sentry uses Groups for authorizing, so we need to support that. I  
>>>> figured that as long as the API specifies Principal, it typically  
>>>> contains both user and group, so nothing else is needed. Did I miss  
>>>> anything?  
>>>>  
>>>> * Once we support group acls we will need someway to indicate if a  
>>>> principal is of type group or user(as part of acl) or we can have  
>>>>group  
>>>> acls and user acls stored separately with topic config. We will also  
>>>>need a  
>>>> way to map an authenticated user to list of groups the user belongs  
>>>>to.  
>>>>  
>>>> 13. It looks like the Authorizer stores the ACLs and not Kafka. So we  
>>>> need an API for Kafka to notify Authorizer when a topic is added and  
>>>> when ACLs are modified, right? I didn’t see that.  
>>>>  
>>>> * ACLs will be stored under /topic/config at time of topic  
>>>>creation  
>>>>in  
>>>> json format. Authorizer will get these acls using newly introduced  
>>>> TopicConfigCache which gets updated anytime topic config changes.  
>>>>  
>>>> 14. Are we going to have any API for Kafka to give out the ACLs on a  
>>>> topic? Or we leave this to the Authorizer?  
>>>>  
>>>> * I think it is better to leave this out side of Kafka. Mainly  
>>>>because  
>>>> most 3rd party authorizers will have their own ACL stores and  
>>>>configuration  
>>>> Uis like (Ranger-Argus not sure what are they calling it now).  
>>>>  
>>>>  
>>>> On Wed, Mar 25, 2015 at 9:26 PM, Neha Narkhede  
>>>><neha@confluent.io<mailto:  
>>>> neha@confluent.io>> wrote:  
>>>> Parth,  
>>>>  
>>>> We can make some 15 mins or so to discuss this at the next KIP  
>>>>hangout.  
>>>>  
>>>> Thanks,  
>>>> Neha  
>>>>  
>>>> On Wed, Mar 25, 2015 at 1:07 PM, Parth Brahmbhatt <  
>>>> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>  
>>>>wrote:  
>>>>  
>>>> Hi all,  
>>>>  
>>>> I have modified the KIP to reflect the recent change request from the  
>>>> reviewers. I have been working on the code and I have the server side  
>>>>code  
>>>> for authorization ready. I am now modifying the command line  
>>>>utilities.  
>>>>I  
>>>> would really appreciate if some of the committers can spend sometime  
>>>>to  
>>>> review the KIP so we can make progress on this.  
>>>>  
>>>> Thanks  
>>>> Parth  
>>>>  
>>>> On 3/18/15, 2:20 PM, "Michael Herstine"  
>>>><mherstine@linkedin.com.INVALID  
>>>> <ma...@linkedin.com.INVALID>>  
>>>> wrote:  
>>>>  
>>>> >Hi Parth,  
>>>> >  
>>>> >Thanks! A few questions:  
>>>> >  
>>>> >1. Do you want to permit rules in your ACLs that DENY access as well  
>>>>as  
>>>> >ALLOW? This can be handy setting up rules that have exceptions. E.g.  
>>>> >“Allow principal P to READ resource R from all hosts” with “Deny  
>>>>principal  
>>>> >P READ access to resource R from host H1” in combination would allow  
>>>>P  
>>>>to  
>>>> >READ R from all hosts *except* H1.  
>>>> >  
>>>> >2. When a topic is newly created, will there be an ACL created for  
>>>>it?  
>>>>If  
>>>> >not, would that not deny subsequent access to it?  
>>>> >  
>>>> >(nit) Maybe use Principal instead of String to represent principals?  
>>>> >  
>>>> >  
>>>> >On 3/9/15, 11:48 AM, "Don Bosco Durai" <bosco@apache.org<mailto:  
>>>> bosco@apache.org>> wrote:  
>>>> >  
>>>> >>Parth  
>>>> >>  
>>>> >>Overall it is looking good. Couple of questionsŠ  
>>>> >>  
>>>> >>- Can you give an example how the policies will look like in the  
>>>>default  
>>>> >>implementation?  
>>>> >>- In the operations, can we support ³CONNECT² also? This can be used  
>>>> >>during Session connection  
>>>> >>- Regarding access control for ³Topic Creation², since we can¹t do  
>>>>it  
>>>>on  
>>>> >>the server side, can we de-scope it for? And plan it as a future  
>>>>feature  
>>>> >>request?  
>>>> >>  
>>>> >>Thanks  
>>>> >>  
>>>> >>Bosco  
>>>> >>  
>>>> >>  
>>>> >>  
>>>> >>On 3/6/15, 8:10 AM, "Harsha"  
>>>><ka...@harsha.io>>  
>>>> wrote:  
>>>> >>  
>>>> >>>Hi Parth,  
>>>> >>> Thanks for putting this together. Overall it looks good  
>>>>to  
>>>> >>> me. Although AdminUtils is a concern KIP-4 can  
>>>>probably  
>>>>fix  
>>>> >>> that part.  
>>>> >>>Thanks,  
>>>> >>>Harsha  
>>>> >>>  
>>>> >>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:  
>>>> >>>> Forgot to add links to wiki and jira.  
>>>> >>>>  
>>>> >>>> Link to wiki:  
>>>> >>>>  
>>>> >>>>  
>>>>  
>>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizatio  
>>>> >>>>n  
>>>> >>>>+  
>>>> >>>>Interface  
>>>> >>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688  
>>>> >>>>  
>>>> >>>> Thanks  
>>>> >>>> Parth  
>>>> >>>>  
>>>> >>>> From: Parth Brahmbhatt  
>>>> >>>> <pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com  
>>>> ><ma...@hortonworks.com>>  
>>>> >>>> Date: Thursday, March 5, 2015 at 10:33 AM  
>>>> >>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org><mailto:  
>>>> dev@kafka.apache.org>"  
>>>> >>>> <de...@kafka.apache.org><mailto:  
>>>> dev@kafka.apache.org>>  
>>>> >>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka  
>>>>security  
>>>> >>>>  
>>>> >>>> Hi,  
>>>> >>>>  
>>>> >>>> KIP-11 is open for discussion , I have updated the wiki with the  
>>>> >>>>design  
>>>> >>>> and open questions.  
>>>> >>>>  
>>>> >>>> Thanks  
>>>> >>>> Parth  
>>>> >>  
>>>> >>  
>>>> >  
>>>>  
>>>>  
>>>>  
>>>>  
>>>> --  
>>>> Thanks,  
>>>> Neha  
>>>>  
>>>>  
>>  
>>  

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Don Bosco Durai <bo...@apache.org>]

>Related interesting question:
Since a broker is a consumer (of lead replicas), how do we handle the
broker level of permissions? Do we hardcode a broker-principal name
and automatically authorize brokers to do anything? Or is there a
cleaner way?

I feel, in Kerberos environment, “kafka” keytab would be the ideal
solution. And “kafka” principal will need to be white listed. SSL
certificate is another option, but it would be painful to set it up. IP
whitelisting is another low impact, but less secure option.

Bosco



On 3/31/15, 10:20 AM, "Gwen Shapira" <gs...@cloudera.com> wrote:

>Related interesting question:
>Since a broker is a consumer (of lead replicas), how do we handle the
>broker level of permissions? Do we hardcode a broker-principal name
>and automatically authorize brokers to do anything? Or is there a
>cleaner way?
>
>
>On Tue, Mar 31, 2015 at 10:17 AM, Don Bosco Durai <bo...@apache.org>
>wrote:
>>>21. Operation: What about other types of requests not covered in the
>>>list,
>> such as committing and fetching offsets, list topics, fetching consumer
>> metadata, heartbeat, join group, etc?
>>
>> Would “CONFIGURE”, “DESCRIBE”, etc take care of this? Or should we add
>> high level grouping like “ADMIN”, “OPERATIONS/MANAGEMENT” to cover
>>related
>> permissions?
>>
>> Bosco
>>
>>
>>
>> On 3/31/15, 9:21 AM, "Jun Rao" <ju...@confluent.io> wrote:
>>
>>>Thanks for the writeup. A few more comments.
>>>
>>>20. I agree that it would be better to do this after KIP-4 (admin
>>>commands)
>>>is done. With KIP-4, all admin operations will be sent as requests to
>>>the
>>>brokers instead of accessing ZK directly. This will make authorization
>>>easier.
>>>
>>>21. Operation: What about other types of requests not covered in the
>>>list,
>>>such as committing and fetching offsets, list topics, fetching consumer
>>>metadata, heartbeat, join group, etc?
>>>
>>>22. TopicConfigCache: We will need such a cache in KIP-4 as well. It
>>>would
>>>be useful to make sure that the implementation can be reused.
>>>
>>>23. Authorizer:
>>>23.1 Do cluster level operations go through authorize() too? If so, what
>>>will be the resource?
>>>23.2 I assume that the authorize() check will be called on every
>>>request.
>>>So, we will have to make sure that the check is cheap.
>>>
>>>24. The acl json string in the config: Should we version this so that we
>>>can evolve it in the future (e.g., adding group support)?
>>>
>>>Jun
>>>
>>>On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt <
>>>pbrahmbhatt@hortonworks.com> wrote:
>>>
>>>> Hi Gwen,
>>>>
>>>> Thanks a lot for taking the time to review this. I have tried to
>>>>address
>>>> all your questions below.
>>>>
>>>> Thanks
>>>> Parth
>>>> On 3/28/15, 8:08 PM, "Gwen Shapira" <gshapira@cloudera.com<mailto:
>>>> gshapira@cloudera.com>> wrote:
>>>>
>>>> Preparing for Tuesday meeting, I went over the KIP :)
>>>>
>>>> First, Parth did an amazing job, the KIP is fantastic - detailed and
>>>> readable. Thank you!
>>>>
>>>> Second, I have a loooong list of questions :) No objections, just some
>>>> things I'm unclear on and random minor comments. In general, I like
>>>> the design, I just feel I'm missing parts of the picture.
>>>>
>>>> 1. "Yes, Create topic will have an optional acls, the output of
>>>> describe will display owner and acls and alter topic will allow to
>>>> modify the acls.”  - will be nice to see what the CLI will look like.
>>>>
>>>>   *   I will modify the KIP but I was going to add “—acl
>>>><acl-file.json>”
>>>> to create-topic and alter-topic.
>>>>
>>>> 2. I like the addition of Topic owner. We made the mistake of
>>>> forgetting about it when adding authorization to Sqoop2. We probably
>>>> want to add “chown” command to the topic commands.
>>>>
>>>>   *   Again we can add “—owner <user-name>” to alter topic.
>>>>
>>>> 3. "Kafka server will read "authorizer.class” config value at startup
>>>> time, create an instance of the specified class and call initialize
>>>> method."
>>>> We’ll need to validate that users specify only one of those.
>>>>
>>>>   *   The config type will be string so type validation should take
>>>>care
>>>> of it.
>>>>
>>>> 4. "One added assumption is that on non-secure connections the session
>>>> will have principal set to an object whose name() method will return
>>>> "Anonymous”."
>>>> Can we keep DrWho? :)
>>>>
>>>>   *   Sure, its up to you actually as you are the owner of the jira
>>>>that
>>>> introduces session concept.
>>>>
>>>> 5. "For cluster actions that do not apply to a specific topic like
>>>> CREATE we have 2 options. We can either add a broker config called
>>>> broker.acls which will point to a json file. This file will be
>>>> available on all broker hosts and authorizer will read the acls on
>>>> initialization and keep refreshing it every X minutes. Any changes
>>>> will require re-distribution of the acl json file. Alternatively we
>>>> can add a zookeeper path /brokers/acls and store the acl json as data.
>>>> Authorizer can refresh the acl from json every X minutes. In absence
>>>> of broker acls the authorizer will fail open, in other words it will
>>>> allow all users from all hosts to perform all cluster actions”
>>>> I prefer a file to ZK - since thats where we store all use-defined
>>>> configurations for now. Everyone knows how to secure a file system :)
>>>>
>>>>   *   I will let everyone vote, file system is fine by me.
>>>>
>>>> 6. "When an Acl is missing , this implementation will always fail open
>>>> for backward compatibility. “ <- agree, but we need to document that
>>>> this makes the default authorizer non-secure
>>>>
>>>>   *   Sure.
>>>>
>>>> 7. "If the value of authorizer.class.name is null, in secure mode the
>>>> cluster will fail with ConfigException. In non secure mode in absence
>>>> of config value forauthorizer.class.name the server will allow all
>>>> requests to all topics that , even if the topic has configured acls”
>>>> <- I don’t think Kafka has “secure mode” - it can support SSL and
>>>> plaintext (un-authenticated) on two different ports simultaneously. I
>>>> don’t object to adding such configuration, but we need to decide
>>>> exactly what it means.
>>>>
>>>>   *   This is one area of confusion so I will add an open question.
>>>>
>>>> 8. "Currently all topic creation/modification/deletion actions are
>>>> performed using KafkaAdminUtil which mostly interacts directly with
>>>> zookeeper instead of forwarding requests to a broker host. Given all
>>>> the code is executed on client side there is no easy way to perform
>>>> authorization. “ - since we are adding the admin protocol requests in
>>>> KIP-4, perhaps addressing those makes sense.
>>>>
>>>>   *   Yes, We will have to wait for KIP-4 to be delivered.
>>>>
>>>> 9. I didn’t see a specification of what is “resource”, I assume its an
>>>> enum with things like Topic and… ?
>>>>
>>>>   *   Resource is not an enum, Right now its just name of the topic.
>>>> Currently we will not be able to support cluster actions like CREATE
>>>>in
>>>> which case the resource can be some constant string like
>>>>“Kafka-Cluster”.
>>>>
>>>> 10. I’m also unclear on where and how “PermissionType” will be used
>>>> and what does “DENY takes precedence” mean here.
>>>>
>>>>   *   I originally did not have the “PermissionType” but someone
>>>>suggested
>>>> in DISCUSS thread that we should add support for ALLOW and DENY acls.
>>>>The
>>>> use case is to support acls like “Allow user U to Perform READ from
>>>>all
>>>> Hosts except from Host1 and Host2”. Deny take precedence only means if
>>>>you
>>>> have define 2 ACLs for the same user/resource/operation. DENY acl will
>>>>take
>>>> effect I.e. “Allow user X to read from *” and “Deny User X to Read
>>>>from
>>>> host1 and host2” will satisfy the use case described in the previous
>>>> statement.
>>>>
>>>> 11. "What does acls on zookeeper node look like given all our admin
>>>> APIs are currently performed directly from client?” <- “secure mode”
>>>> Kafka will need to set ACLs on ZK (using ZK’s model of ACLs) and both
>>>> Kafka and everyone else will need to use them (this is limited to
>>>> Kerberos authentication AFAIK.)
>>>>
>>>>   *   Yes, again I think this can be a separate issue for now and can
>>>>only
>>>> be worked on after KIP-4 is delivered.
>>>>
>>>> 12. "Do we want to support group acls as part of this authorizer? Do
>>>> we want to support principal to local user mapping? If yes we need to
>>>> add plugins for UserToGroupMapper and PrincipalToUserMapper.” <-
>>>> Sentry uses Groups for authorizing, so we need to support that. I
>>>> figured that as long as the API specifies Principal, it typically
>>>> contains both user and group, so nothing else is needed. Did I miss
>>>> anything?
>>>>
>>>>   *   Once we support group acls we will need someway to indicate if a
>>>> principal is of type group or user(as part of acl) or we can have
>>>>group
>>>> acls and user acls stored separately with topic config. We will also
>>>>need a
>>>> way to map an authenticated user to list of groups the user belongs
>>>>to.
>>>>
>>>> 13. It looks like the Authorizer stores the ACLs and not Kafka. So we
>>>> need an API for Kafka to notify Authorizer when a topic is added and
>>>> when ACLs are modified, right? I didn’t see that.
>>>>
>>>>   *   ACLs will be stored under /topic/config at time of topic
>>>>creation
>>>>in
>>>> json format. Authorizer will get these acls using newly introduced
>>>> TopicConfigCache which gets updated anytime topic config changes.
>>>>
>>>> 14. Are we going to have any API for Kafka to give out the ACLs on a
>>>> topic? Or we leave this to the Authorizer?
>>>>
>>>>   *   I think it is better to leave this out side of Kafka. Mainly
>>>>because
>>>> most 3rd party authorizers will have their own ACL stores and
>>>>configuration
>>>> Uis like (Ranger-Argus not sure what are they calling it now).
>>>>
>>>>
>>>> On Wed, Mar 25, 2015 at 9:26 PM, Neha Narkhede
>>>><neha@confluent.io<mailto:
>>>> neha@confluent.io>> wrote:
>>>> Parth,
>>>>
>>>> We can make some 15 mins or so to discuss this at the next KIP
>>>>hangout.
>>>>
>>>> Thanks,
>>>> Neha
>>>>
>>>> On Wed, Mar 25, 2015 at 1:07 PM, Parth Brahmbhatt <
>>>> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>>
>>>>wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I have modified the KIP to reflect the recent change request from the
>>>> reviewers. I have been working on the code and I have the server side
>>>>code
>>>> for authorization ready. I am now modifying the command line
>>>>utilities.
>>>>I
>>>> would really appreciate if some of the committers can spend sometime
>>>>to
>>>> review the KIP so we can make progress on this.
>>>>
>>>> Thanks
>>>> Parth
>>>>
>>>> On 3/18/15, 2:20 PM, "Michael Herstine"
>>>><mherstine@linkedin.com.INVALID
>>>> <ma...@linkedin.com.INVALID>>
>>>> wrote:
>>>>
>>>> >Hi Parth,
>>>> >
>>>> >Thanks! A few questions:
>>>> >
>>>> >1. Do you want to permit rules in your ACLs that DENY access as well
>>>>as
>>>> >ALLOW? This can be handy setting up rules that have exceptions. E.g.
>>>> >“Allow principal P to READ resource R from all hosts” with “Deny
>>>>principal
>>>> >P READ access to resource R from host H1” in combination would allow
>>>>P
>>>>to
>>>> >READ R from all hosts *except* H1.
>>>> >
>>>> >2. When a topic is newly created, will there be an ACL created for
>>>>it?
>>>>If
>>>> >not, would that not deny subsequent access to it?
>>>> >
>>>> >(nit) Maybe use Principal instead of String to represent principals?
>>>> >
>>>> >
>>>> >On 3/9/15, 11:48 AM, "Don Bosco Durai" <bosco@apache.org<mailto:
>>>> bosco@apache.org>> wrote:
>>>> >
>>>> >>Parth
>>>> >>
>>>> >>Overall it is looking good. Couple of questionsŠ
>>>> >>
>>>> >>- Can you give an example how the policies will look like in the
>>>>default
>>>> >>implementation?
>>>> >>- In the operations, can we support ³CONNECT² also? This can be used
>>>> >>during Session connection
>>>> >>- Regarding access control for ³Topic Creation², since we can¹t do
>>>>it
>>>>on
>>>> >>the server side, can we de-scope it for? And plan it as a future
>>>>feature
>>>> >>request?
>>>> >>
>>>> >>Thanks
>>>> >>
>>>> >>Bosco
>>>> >>
>>>> >>
>>>> >>
>>>> >>On 3/6/15, 8:10 AM, "Harsha"
>>>><ka...@harsha.io>>
>>>> wrote:
>>>> >>
>>>> >>>Hi Parth,
>>>> >>>            Thanks for putting this together. Overall it looks good
>>>>to
>>>> >>>            me. Although AdminUtils is a concern KIP-4  can
>>>>probably
>>>>fix
>>>> >>>            that part.
>>>> >>>Thanks,
>>>> >>>Harsha
>>>> >>>
>>>> >>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>>> >>>> Forgot to add links to wiki and jira.
>>>> >>>>
>>>> >>>> Link to wiki:
>>>> >>>>
>>>> >>>>
>>>> 
>>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizatio
>>>> >>>>n
>>>> >>>>+
>>>> >>>>Interface
>>>> >>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>>> >>>>
>>>> >>>> Thanks
>>>> >>>> Parth
>>>> >>>>
>>>> >>>> From: Parth Brahmbhatt
>>>> >>>> <pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
>>>> ><ma...@hortonworks.com>>
>>>> >>>> Date: Thursday, March 5, 2015 at 10:33 AM
>>>> >>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org><mailto:
>>>> dev@kafka.apache.org>"
>>>> >>>> <de...@kafka.apache.org><mailto:
>>>> dev@kafka.apache.org>>
>>>> >>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka
>>>>security
>>>> >>>>
>>>> >>>> Hi,
>>>> >>>>
>>>> >>>> KIP-11 is open for discussion , I have updated the wiki with the
>>>> >>>>design
>>>> >>>> and open questions.
>>>> >>>>
>>>> >>>> Thanks
>>>> >>>> Parth
>>>> >>
>>>> >>
>>>> >
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Neha
>>>>
>>>>
>>
>>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Gwen Shapira <gs...@cloudera.com>]

Related interesting question:
Since a broker is a consumer (of lead replicas), how do we handle the
broker level of permissions? Do we hardcode a broker-principal name
and automatically authorize brokers to do anything? Or is there a
cleaner way?


On Tue, Mar 31, 2015 at 10:17 AM, Don Bosco Durai <bo...@apache.org> wrote:
>>21. Operation: What about other types of requests not covered in the list,
> such as committing and fetching offsets, list topics, fetching consumer
> metadata, heartbeat, join group, etc?
>
> Would “CONFIGURE”, “DESCRIBE”, etc take care of this? Or should we add
> high level grouping like “ADMIN”, “OPERATIONS/MANAGEMENT” to cover related
> permissions?
>
> Bosco
>
>
>
> On 3/31/15, 9:21 AM, "Jun Rao" <ju...@confluent.io> wrote:
>
>>Thanks for the writeup. A few more comments.
>>
>>20. I agree that it would be better to do this after KIP-4 (admin
>>commands)
>>is done. With KIP-4, all admin operations will be sent as requests to the
>>brokers instead of accessing ZK directly. This will make authorization
>>easier.
>>
>>21. Operation: What about other types of requests not covered in the list,
>>such as committing and fetching offsets, list topics, fetching consumer
>>metadata, heartbeat, join group, etc?
>>
>>22. TopicConfigCache: We will need such a cache in KIP-4 as well. It would
>>be useful to make sure that the implementation can be reused.
>>
>>23. Authorizer:
>>23.1 Do cluster level operations go through authorize() too? If so, what
>>will be the resource?
>>23.2 I assume that the authorize() check will be called on every request.
>>So, we will have to make sure that the check is cheap.
>>
>>24. The acl json string in the config: Should we version this so that we
>>can evolve it in the future (e.g., adding group support)?
>>
>>Jun
>>
>>On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt <
>>pbrahmbhatt@hortonworks.com> wrote:
>>
>>> Hi Gwen,
>>>
>>> Thanks a lot for taking the time to review this. I have tried to address
>>> all your questions below.
>>>
>>> Thanks
>>> Parth
>>> On 3/28/15, 8:08 PM, "Gwen Shapira" <gshapira@cloudera.com<mailto:
>>> gshapira@cloudera.com>> wrote:
>>>
>>> Preparing for Tuesday meeting, I went over the KIP :)
>>>
>>> First, Parth did an amazing job, the KIP is fantastic - detailed and
>>> readable. Thank you!
>>>
>>> Second, I have a loooong list of questions :) No objections, just some
>>> things I'm unclear on and random minor comments. In general, I like
>>> the design, I just feel I'm missing parts of the picture.
>>>
>>> 1. "Yes, Create topic will have an optional acls, the output of
>>> describe will display owner and acls and alter topic will allow to
>>> modify the acls.”  - will be nice to see what the CLI will look like.
>>>
>>>   *   I will modify the KIP but I was going to add “—acl
>>><acl-file.json>”
>>> to create-topic and alter-topic.
>>>
>>> 2. I like the addition of Topic owner. We made the mistake of
>>> forgetting about it when adding authorization to Sqoop2. We probably
>>> want to add “chown” command to the topic commands.
>>>
>>>   *   Again we can add “—owner <user-name>” to alter topic.
>>>
>>> 3. "Kafka server will read "authorizer.class” config value at startup
>>> time, create an instance of the specified class and call initialize
>>> method."
>>> We’ll need to validate that users specify only one of those.
>>>
>>>   *   The config type will be string so type validation should take care
>>> of it.
>>>
>>> 4. "One added assumption is that on non-secure connections the session
>>> will have principal set to an object whose name() method will return
>>> "Anonymous”."
>>> Can we keep DrWho? :)
>>>
>>>   *   Sure, its up to you actually as you are the owner of the jira that
>>> introduces session concept.
>>>
>>> 5. "For cluster actions that do not apply to a specific topic like
>>> CREATE we have 2 options. We can either add a broker config called
>>> broker.acls which will point to a json file. This file will be
>>> available on all broker hosts and authorizer will read the acls on
>>> initialization and keep refreshing it every X minutes. Any changes
>>> will require re-distribution of the acl json file. Alternatively we
>>> can add a zookeeper path /brokers/acls and store the acl json as data.
>>> Authorizer can refresh the acl from json every X minutes. In absence
>>> of broker acls the authorizer will fail open, in other words it will
>>> allow all users from all hosts to perform all cluster actions”
>>> I prefer a file to ZK - since thats where we store all use-defined
>>> configurations for now. Everyone knows how to secure a file system :)
>>>
>>>   *   I will let everyone vote, file system is fine by me.
>>>
>>> 6. "When an Acl is missing , this implementation will always fail open
>>> for backward compatibility. “ <- agree, but we need to document that
>>> this makes the default authorizer non-secure
>>>
>>>   *   Sure.
>>>
>>> 7. "If the value of authorizer.class.name is null, in secure mode the
>>> cluster will fail with ConfigException. In non secure mode in absence
>>> of config value forauthorizer.class.name the server will allow all
>>> requests to all topics that , even if the topic has configured acls”
>>> <- I don’t think Kafka has “secure mode” - it can support SSL and
>>> plaintext (un-authenticated) on two different ports simultaneously. I
>>> don’t object to adding such configuration, but we need to decide
>>> exactly what it means.
>>>
>>>   *   This is one area of confusion so I will add an open question.
>>>
>>> 8. "Currently all topic creation/modification/deletion actions are
>>> performed using KafkaAdminUtil which mostly interacts directly with
>>> zookeeper instead of forwarding requests to a broker host. Given all
>>> the code is executed on client side there is no easy way to perform
>>> authorization. “ - since we are adding the admin protocol requests in
>>> KIP-4, perhaps addressing those makes sense.
>>>
>>>   *   Yes, We will have to wait for KIP-4 to be delivered.
>>>
>>> 9. I didn’t see a specification of what is “resource”, I assume its an
>>> enum with things like Topic and… ?
>>>
>>>   *   Resource is not an enum, Right now its just name of the topic.
>>> Currently we will not be able to support cluster actions like CREATE in
>>> which case the resource can be some constant string like
>>>“Kafka-Cluster”.
>>>
>>> 10. I’m also unclear on where and how “PermissionType” will be used
>>> and what does “DENY takes precedence” mean here.
>>>
>>>   *   I originally did not have the “PermissionType” but someone
>>>suggested
>>> in DISCUSS thread that we should add support for ALLOW and DENY acls.
>>>The
>>> use case is to support acls like “Allow user U to Perform READ from all
>>> Hosts except from Host1 and Host2”. Deny take precedence only means if
>>>you
>>> have define 2 ACLs for the same user/resource/operation. DENY acl will
>>>take
>>> effect I.e. “Allow user X to read from *” and “Deny User X to Read from
>>> host1 and host2” will satisfy the use case described in the previous
>>> statement.
>>>
>>> 11. "What does acls on zookeeper node look like given all our admin
>>> APIs are currently performed directly from client?” <- “secure mode”
>>> Kafka will need to set ACLs on ZK (using ZK’s model of ACLs) and both
>>> Kafka and everyone else will need to use them (this is limited to
>>> Kerberos authentication AFAIK.)
>>>
>>>   *   Yes, again I think this can be a separate issue for now and can
>>>only
>>> be worked on after KIP-4 is delivered.
>>>
>>> 12. "Do we want to support group acls as part of this authorizer? Do
>>> we want to support principal to local user mapping? If yes we need to
>>> add plugins for UserToGroupMapper and PrincipalToUserMapper.” <-
>>> Sentry uses Groups for authorizing, so we need to support that. I
>>> figured that as long as the API specifies Principal, it typically
>>> contains both user and group, so nothing else is needed. Did I miss
>>> anything?
>>>
>>>   *   Once we support group acls we will need someway to indicate if a
>>> principal is of type group or user(as part of acl) or we can have group
>>> acls and user acls stored separately with topic config. We will also
>>>need a
>>> way to map an authenticated user to list of groups the user belongs to.
>>>
>>> 13. It looks like the Authorizer stores the ACLs and not Kafka. So we
>>> need an API for Kafka to notify Authorizer when a topic is added and
>>> when ACLs are modified, right? I didn’t see that.
>>>
>>>   *   ACLs will be stored under /topic/config at time of topic creation
>>>in
>>> json format. Authorizer will get these acls using newly introduced
>>> TopicConfigCache which gets updated anytime topic config changes.
>>>
>>> 14. Are we going to have any API for Kafka to give out the ACLs on a
>>> topic? Or we leave this to the Authorizer?
>>>
>>>   *   I think it is better to leave this out side of Kafka. Mainly
>>>because
>>> most 3rd party authorizers will have their own ACL stores and
>>>configuration
>>> Uis like (Ranger-Argus not sure what are they calling it now).
>>>
>>>
>>> On Wed, Mar 25, 2015 at 9:26 PM, Neha Narkhede
>>><neha@confluent.io<mailto:
>>> neha@confluent.io>> wrote:
>>> Parth,
>>>
>>> We can make some 15 mins or so to discuss this at the next KIP hangout.
>>>
>>> Thanks,
>>> Neha
>>>
>>> On Wed, Mar 25, 2015 at 1:07 PM, Parth Brahmbhatt <
>>> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>> wrote:
>>>
>>> Hi all,
>>>
>>> I have modified the KIP to reflect the recent change request from the
>>> reviewers. I have been working on the code and I have the server side
>>>code
>>> for authorization ready. I am now modifying the command line utilities.
>>>I
>>> would really appreciate if some of the committers can spend sometime to
>>> review the KIP so we can make progress on this.
>>>
>>> Thanks
>>> Parth
>>>
>>> On 3/18/15, 2:20 PM, "Michael Herstine" <mherstine@linkedin.com.INVALID
>>> <ma...@linkedin.com.INVALID>>
>>> wrote:
>>>
>>> >Hi Parth,
>>> >
>>> >Thanks! A few questions:
>>> >
>>> >1. Do you want to permit rules in your ACLs that DENY access as well as
>>> >ALLOW? This can be handy setting up rules that have exceptions. E.g.
>>> >“Allow principal P to READ resource R from all hosts” with “Deny
>>>principal
>>> >P READ access to resource R from host H1” in combination would allow P
>>>to
>>> >READ R from all hosts *except* H1.
>>> >
>>> >2. When a topic is newly created, will there be an ACL created for it?
>>>If
>>> >not, would that not deny subsequent access to it?
>>> >
>>> >(nit) Maybe use Principal instead of String to represent principals?
>>> >
>>> >
>>> >On 3/9/15, 11:48 AM, "Don Bosco Durai" <bosco@apache.org<mailto:
>>> bosco@apache.org>> wrote:
>>> >
>>> >>Parth
>>> >>
>>> >>Overall it is looking good. Couple of questionsŠ
>>> >>
>>> >>- Can you give an example how the policies will look like in the
>>>default
>>> >>implementation?
>>> >>- In the operations, can we support ³CONNECT² also? This can be used
>>> >>during Session connection
>>> >>- Regarding access control for ³Topic Creation², since we can¹t do it
>>>on
>>> >>the server side, can we de-scope it for? And plan it as a future
>>>feature
>>> >>request?
>>> >>
>>> >>Thanks
>>> >>
>>> >>Bosco
>>> >>
>>> >>
>>> >>
>>> >>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io>>
>>> wrote:
>>> >>
>>> >>>Hi Parth,
>>> >>>            Thanks for putting this together. Overall it looks good
>>>to
>>> >>>            me. Although AdminUtils is a concern KIP-4  can probably
>>>fix
>>> >>>            that part.
>>> >>>Thanks,
>>> >>>Harsha
>>> >>>
>>> >>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>> >>>> Forgot to add links to wiki and jira.
>>> >>>>
>>> >>>> Link to wiki:
>>> >>>>
>>> >>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizatio
>>> >>>>n
>>> >>>>+
>>> >>>>Interface
>>> >>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>> >>>>
>>> >>>> Thanks
>>> >>>> Parth
>>> >>>>
>>> >>>> From: Parth Brahmbhatt
>>> >>>> <pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
>>> ><ma...@hortonworks.com>>
>>> >>>> Date: Thursday, March 5, 2015 at 10:33 AM
>>> >>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org><mailto:
>>> dev@kafka.apache.org>"
>>> >>>> <de...@kafka.apache.org><mailto:
>>> dev@kafka.apache.org>>
>>> >>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>> >>>>
>>> >>>> Hi,
>>> >>>>
>>> >>>> KIP-11 is open for discussion , I have updated the wiki with the
>>> >>>>design
>>> >>>> and open questions.
>>> >>>>
>>> >>>> Thanks
>>> >>>> Parth
>>> >>
>>> >>
>>> >
>>>
>>>
>>>
>>>
>>> --
>>> Thanks,
>>> Neha
>>>
>>>
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Don Bosco Durai <bo...@apache.org>]

>21. Operation: What about other types of requests not covered in the list,
such as committing and fetching offsets, list topics, fetching consumer
metadata, heartbeat, join group, etc?

Would “CONFIGURE”, “DESCRIBE”, etc take care of this? Or should we add
high level grouping like “ADMIN”, “OPERATIONS/MANAGEMENT” to cover related
permissions?

Bosco



On 3/31/15, 9:21 AM, "Jun Rao" <ju...@confluent.io> wrote:

>Thanks for the writeup. A few more comments.
>
>20. I agree that it would be better to do this after KIP-4 (admin
>commands)
>is done. With KIP-4, all admin operations will be sent as requests to the
>brokers instead of accessing ZK directly. This will make authorization
>easier.
>
>21. Operation: What about other types of requests not covered in the list,
>such as committing and fetching offsets, list topics, fetching consumer
>metadata, heartbeat, join group, etc?
>
>22. TopicConfigCache: We will need such a cache in KIP-4 as well. It would
>be useful to make sure that the implementation can be reused.
>
>23. Authorizer:
>23.1 Do cluster level operations go through authorize() too? If so, what
>will be the resource?
>23.2 I assume that the authorize() check will be called on every request.
>So, we will have to make sure that the check is cheap.
>
>24. The acl json string in the config: Should we version this so that we
>can evolve it in the future (e.g., adding group support)?
>
>Jun
>
>On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt <
>pbrahmbhatt@hortonworks.com> wrote:
>
>> Hi Gwen,
>>
>> Thanks a lot for taking the time to review this. I have tried to address
>> all your questions below.
>>
>> Thanks
>> Parth
>> On 3/28/15, 8:08 PM, "Gwen Shapira" <gshapira@cloudera.com<mailto:
>> gshapira@cloudera.com>> wrote:
>>
>> Preparing for Tuesday meeting, I went over the KIP :)
>>
>> First, Parth did an amazing job, the KIP is fantastic - detailed and
>> readable. Thank you!
>>
>> Second, I have a loooong list of questions :) No objections, just some
>> things I'm unclear on and random minor comments. In general, I like
>> the design, I just feel I'm missing parts of the picture.
>>
>> 1. "Yes, Create topic will have an optional acls, the output of
>> describe will display owner and acls and alter topic will allow to
>> modify the acls.”  - will be nice to see what the CLI will look like.
>>
>>   *   I will modify the KIP but I was going to add “—acl
>><acl-file.json>”
>> to create-topic and alter-topic.
>>
>> 2. I like the addition of Topic owner. We made the mistake of
>> forgetting about it when adding authorization to Sqoop2. We probably
>> want to add “chown” command to the topic commands.
>>
>>   *   Again we can add “—owner <user-name>” to alter topic.
>>
>> 3. "Kafka server will read "authorizer.class” config value at startup
>> time, create an instance of the specified class and call initialize
>> method."
>> We’ll need to validate that users specify only one of those.
>>
>>   *   The config type will be string so type validation should take care
>> of it.
>>
>> 4. "One added assumption is that on non-secure connections the session
>> will have principal set to an object whose name() method will return
>> "Anonymous”."
>> Can we keep DrWho? :)
>>
>>   *   Sure, its up to you actually as you are the owner of the jira that
>> introduces session concept.
>>
>> 5. "For cluster actions that do not apply to a specific topic like
>> CREATE we have 2 options. We can either add a broker config called
>> broker.acls which will point to a json file. This file will be
>> available on all broker hosts and authorizer will read the acls on
>> initialization and keep refreshing it every X minutes. Any changes
>> will require re-distribution of the acl json file. Alternatively we
>> can add a zookeeper path /brokers/acls and store the acl json as data.
>> Authorizer can refresh the acl from json every X minutes. In absence
>> of broker acls the authorizer will fail open, in other words it will
>> allow all users from all hosts to perform all cluster actions”
>> I prefer a file to ZK - since thats where we store all use-defined
>> configurations for now. Everyone knows how to secure a file system :)
>>
>>   *   I will let everyone vote, file system is fine by me.
>>
>> 6. "When an Acl is missing , this implementation will always fail open
>> for backward compatibility. “ <- agree, but we need to document that
>> this makes the default authorizer non-secure
>>
>>   *   Sure.
>>
>> 7. "If the value of authorizer.class.name is null, in secure mode the
>> cluster will fail with ConfigException. In non secure mode in absence
>> of config value forauthorizer.class.name the server will allow all
>> requests to all topics that , even if the topic has configured acls”
>> <- I don’t think Kafka has “secure mode” - it can support SSL and
>> plaintext (un-authenticated) on two different ports simultaneously. I
>> don’t object to adding such configuration, but we need to decide
>> exactly what it means.
>>
>>   *   This is one area of confusion so I will add an open question.
>>
>> 8. "Currently all topic creation/modification/deletion actions are
>> performed using KafkaAdminUtil which mostly interacts directly with
>> zookeeper instead of forwarding requests to a broker host. Given all
>> the code is executed on client side there is no easy way to perform
>> authorization. “ - since we are adding the admin protocol requests in
>> KIP-4, perhaps addressing those makes sense.
>>
>>   *   Yes, We will have to wait for KIP-4 to be delivered.
>>
>> 9. I didn’t see a specification of what is “resource”, I assume its an
>> enum with things like Topic and… ?
>>
>>   *   Resource is not an enum, Right now its just name of the topic.
>> Currently we will not be able to support cluster actions like CREATE in
>> which case the resource can be some constant string like
>>“Kafka-Cluster”.
>>
>> 10. I’m also unclear on where and how “PermissionType” will be used
>> and what does “DENY takes precedence” mean here.
>>
>>   *   I originally did not have the “PermissionType” but someone
>>suggested
>> in DISCUSS thread that we should add support for ALLOW and DENY acls.
>>The
>> use case is to support acls like “Allow user U to Perform READ from all
>> Hosts except from Host1 and Host2”. Deny take precedence only means if
>>you
>> have define 2 ACLs for the same user/resource/operation. DENY acl will
>>take
>> effect I.e. “Allow user X to read from *” and “Deny User X to Read from
>> host1 and host2” will satisfy the use case described in the previous
>> statement.
>>
>> 11. "What does acls on zookeeper node look like given all our admin
>> APIs are currently performed directly from client?” <- “secure mode”
>> Kafka will need to set ACLs on ZK (using ZK’s model of ACLs) and both
>> Kafka and everyone else will need to use them (this is limited to
>> Kerberos authentication AFAIK.)
>>
>>   *   Yes, again I think this can be a separate issue for now and can
>>only
>> be worked on after KIP-4 is delivered.
>>
>> 12. "Do we want to support group acls as part of this authorizer? Do
>> we want to support principal to local user mapping? If yes we need to
>> add plugins for UserToGroupMapper and PrincipalToUserMapper.” <-
>> Sentry uses Groups for authorizing, so we need to support that. I
>> figured that as long as the API specifies Principal, it typically
>> contains both user and group, so nothing else is needed. Did I miss
>> anything?
>>
>>   *   Once we support group acls we will need someway to indicate if a
>> principal is of type group or user(as part of acl) or we can have group
>> acls and user acls stored separately with topic config. We will also
>>need a
>> way to map an authenticated user to list of groups the user belongs to.
>>
>> 13. It looks like the Authorizer stores the ACLs and not Kafka. So we
>> need an API for Kafka to notify Authorizer when a topic is added and
>> when ACLs are modified, right? I didn’t see that.
>>
>>   *   ACLs will be stored under /topic/config at time of topic creation
>>in
>> json format. Authorizer will get these acls using newly introduced
>> TopicConfigCache which gets updated anytime topic config changes.
>>
>> 14. Are we going to have any API for Kafka to give out the ACLs on a
>> topic? Or we leave this to the Authorizer?
>>
>>   *   I think it is better to leave this out side of Kafka. Mainly
>>because
>> most 3rd party authorizers will have their own ACL stores and
>>configuration
>> Uis like (Ranger-Argus not sure what are they calling it now).
>>
>>
>> On Wed, Mar 25, 2015 at 9:26 PM, Neha Narkhede
>><neha@confluent.io<mailto:
>> neha@confluent.io>> wrote:
>> Parth,
>>
>> We can make some 15 mins or so to discuss this at the next KIP hangout.
>>
>> Thanks,
>> Neha
>>
>> On Wed, Mar 25, 2015 at 1:07 PM, Parth Brahmbhatt <
>> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>> wrote:
>>
>> Hi all,
>>
>> I have modified the KIP to reflect the recent change request from the
>> reviewers. I have been working on the code and I have the server side
>>code
>> for authorization ready. I am now modifying the command line utilities.
>>I
>> would really appreciate if some of the committers can spend sometime to
>> review the KIP so we can make progress on this.
>>
>> Thanks
>> Parth
>>
>> On 3/18/15, 2:20 PM, "Michael Herstine" <mherstine@linkedin.com.INVALID
>> <ma...@linkedin.com.INVALID>>
>> wrote:
>>
>> >Hi Parth,
>> >
>> >Thanks! A few questions:
>> >
>> >1. Do you want to permit rules in your ACLs that DENY access as well as
>> >ALLOW? This can be handy setting up rules that have exceptions. E.g.
>> >“Allow principal P to READ resource R from all hosts” with “Deny
>>principal
>> >P READ access to resource R from host H1” in combination would allow P
>>to
>> >READ R from all hosts *except* H1.
>> >
>> >2. When a topic is newly created, will there be an ACL created for it?
>>If
>> >not, would that not deny subsequent access to it?
>> >
>> >(nit) Maybe use Principal instead of String to represent principals?
>> >
>> >
>> >On 3/9/15, 11:48 AM, "Don Bosco Durai" <bosco@apache.org<mailto:
>> bosco@apache.org>> wrote:
>> >
>> >>Parth
>> >>
>> >>Overall it is looking good. Couple of questionsŠ
>> >>
>> >>- Can you give an example how the policies will look like in the
>>default
>> >>implementation?
>> >>- In the operations, can we support ³CONNECT² also? This can be used
>> >>during Session connection
>> >>- Regarding access control for ³Topic Creation², since we can¹t do it
>>on
>> >>the server side, can we de-scope it for? And plan it as a future
>>feature
>> >>request?
>> >>
>> >>Thanks
>> >>
>> >>Bosco
>> >>
>> >>
>> >>
>> >>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io>>
>> wrote:
>> >>
>> >>>Hi Parth,
>> >>>            Thanks for putting this together. Overall it looks good
>>to
>> >>>            me. Although AdminUtils is a concern KIP-4  can probably
>>fix
>> >>>            that part.
>> >>>Thanks,
>> >>>Harsha
>> >>>
>> >>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>> >>>> Forgot to add links to wiki and jira.
>> >>>>
>> >>>> Link to wiki:
>> >>>>
>> >>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizatio
>> >>>>n
>> >>>>+
>> >>>>Interface
>> >>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>> >>>>
>> >>>> Thanks
>> >>>> Parth
>> >>>>
>> >>>> From: Parth Brahmbhatt
>> >>>> <pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
>> ><ma...@hortonworks.com>>
>> >>>> Date: Thursday, March 5, 2015 at 10:33 AM
>> >>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org><mailto:
>> dev@kafka.apache.org>"
>> >>>> <de...@kafka.apache.org><mailto:
>> dev@kafka.apache.org>>
>> >>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>> KIP-11 is open for discussion , I have updated the wiki with the
>> >>>>design
>> >>>> and open questions.
>> >>>>
>> >>>> Thanks
>> >>>> Parth
>> >>
>> >>
>> >
>>
>>
>>
>>
>> --
>> Thanks,
>> Neha
>>
>>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Jun Rao <ju...@confluent.io>]

Thanks for the writeup. A few more comments.

20. I agree that it would be better to do this after KIP-4 (admin commands)
is done. With KIP-4, all admin operations will be sent as requests to the
brokers instead of accessing ZK directly. This will make authorization
easier.

21. Operation: What about other types of requests not covered in the list,
such as committing and fetching offsets, list topics, fetching consumer
metadata, heartbeat, join group, etc?

22. TopicConfigCache: We will need such a cache in KIP-4 as well. It would
be useful to make sure that the implementation can be reused.

23. Authorizer:
23.1 Do cluster level operations go through authorize() too? If so, what
will be the resource?
23.2 I assume that the authorize() check will be called on every request.
So, we will have to make sure that the check is cheap.

24. The acl json string in the config: Should we version this so that we
can evolve it in the future (e.g., adding group support)?

Jun

On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:

> Hi Gwen,
>
> Thanks a lot for taking the time to review this. I have tried to address
> all your questions below.
>
> Thanks
> Parth
> On 3/28/15, 8:08 PM, "Gwen Shapira" <gshapira@cloudera.com<mailto:
> gshapira@cloudera.com>> wrote:
>
> Preparing for Tuesday meeting, I went over the KIP :)
>
> First, Parth did an amazing job, the KIP is fantastic - detailed and
> readable. Thank you!
>
> Second, I have a loooong list of questions :) No objections, just some
> things I'm unclear on and random minor comments. In general, I like
> the design, I just feel I'm missing parts of the picture.
>
> 1. "Yes, Create topic will have an optional acls, the output of
> describe will display owner and acls and alter topic will allow to
> modify the acls.”  - will be nice to see what the CLI will look like.
>
>   *   I will modify the KIP but I was going to add “—acl <acl-file.json>”
> to create-topic and alter-topic.
>
> 2. I like the addition of Topic owner. We made the mistake of
> forgetting about it when adding authorization to Sqoop2. We probably
> want to add “chown” command to the topic commands.
>
>   *   Again we can add “—owner <user-name>” to alter topic.
>
> 3. "Kafka server will read "authorizer.class” config value at startup
> time, create an instance of the specified class and call initialize
> method."
> We’ll need to validate that users specify only one of those.
>
>   *   The config type will be string so type validation should take care
> of it.
>
> 4. "One added assumption is that on non-secure connections the session
> will have principal set to an object whose name() method will return
> "Anonymous”."
> Can we keep DrWho? :)
>
>   *   Sure, its up to you actually as you are the owner of the jira that
> introduces session concept.
>
> 5. "For cluster actions that do not apply to a specific topic like
> CREATE we have 2 options. We can either add a broker config called
> broker.acls which will point to a json file. This file will be
> available on all broker hosts and authorizer will read the acls on
> initialization and keep refreshing it every X minutes. Any changes
> will require re-distribution of the acl json file. Alternatively we
> can add a zookeeper path /brokers/acls and store the acl json as data.
> Authorizer can refresh the acl from json every X minutes. In absence
> of broker acls the authorizer will fail open, in other words it will
> allow all users from all hosts to perform all cluster actions”
> I prefer a file to ZK - since thats where we store all use-defined
> configurations for now. Everyone knows how to secure a file system :)
>
>   *   I will let everyone vote, file system is fine by me.
>
> 6. "When an Acl is missing , this implementation will always fail open
> for backward compatibility. “ <- agree, but we need to document that
> this makes the default authorizer non-secure
>
>   *   Sure.
>
> 7. "If the value of authorizer.class.name is null, in secure mode the
> cluster will fail with ConfigException. In non secure mode in absence
> of config value forauthorizer.class.name the server will allow all
> requests to all topics that , even if the topic has configured acls”
> <- I don’t think Kafka has “secure mode” - it can support SSL and
> plaintext (un-authenticated) on two different ports simultaneously. I
> don’t object to adding such configuration, but we need to decide
> exactly what it means.
>
>   *   This is one area of confusion so I will add an open question.
>
> 8. "Currently all topic creation/modification/deletion actions are
> performed using KafkaAdminUtil which mostly interacts directly with
> zookeeper instead of forwarding requests to a broker host. Given all
> the code is executed on client side there is no easy way to perform
> authorization. “ - since we are adding the admin protocol requests in
> KIP-4, perhaps addressing those makes sense.
>
>   *   Yes, We will have to wait for KIP-4 to be delivered.
>
> 9. I didn’t see a specification of what is “resource”, I assume its an
> enum with things like Topic and… ?
>
>   *   Resource is not an enum, Right now its just name of the topic.
> Currently we will not be able to support cluster actions like CREATE in
> which case the resource can be some constant string like “Kafka-Cluster”.
>
> 10. I’m also unclear on where and how “PermissionType” will be used
> and what does “DENY takes precedence” mean here.
>
>   *   I originally did not have the “PermissionType” but someone suggested
> in DISCUSS thread that we should add support for ALLOW and DENY acls. The
> use case is to support acls like “Allow user U to Perform READ from all
> Hosts except from Host1 and Host2”. Deny take precedence only means if you
> have define 2 ACLs for the same user/resource/operation. DENY acl will take
> effect I.e. “Allow user X to read from *” and “Deny User X to Read from
> host1 and host2” will satisfy the use case described in the previous
> statement.
>
> 11. "What does acls on zookeeper node look like given all our admin
> APIs are currently performed directly from client?” <- “secure mode”
> Kafka will need to set ACLs on ZK (using ZK’s model of ACLs) and both
> Kafka and everyone else will need to use them (this is limited to
> Kerberos authentication AFAIK.)
>
>   *   Yes, again I think this can be a separate issue for now and can only
> be worked on after KIP-4 is delivered.
>
> 12. "Do we want to support group acls as part of this authorizer? Do
> we want to support principal to local user mapping? If yes we need to
> add plugins for UserToGroupMapper and PrincipalToUserMapper.” <-
> Sentry uses Groups for authorizing, so we need to support that. I
> figured that as long as the API specifies Principal, it typically
> contains both user and group, so nothing else is needed. Did I miss
> anything?
>
>   *   Once we support group acls we will need someway to indicate if a
> principal is of type group or user(as part of acl) or we can have group
> acls and user acls stored separately with topic config. We will also need a
> way to map an authenticated user to list of groups the user belongs to.
>
> 13. It looks like the Authorizer stores the ACLs and not Kafka. So we
> need an API for Kafka to notify Authorizer when a topic is added and
> when ACLs are modified, right? I didn’t see that.
>
>   *   ACLs will be stored under /topic/config at time of topic creation in
> json format. Authorizer will get these acls using newly introduced
> TopicConfigCache which gets updated anytime topic config changes.
>
> 14. Are we going to have any API for Kafka to give out the ACLs on a
> topic? Or we leave this to the Authorizer?
>
>   *   I think it is better to leave this out side of Kafka. Mainly because
> most 3rd party authorizers will have their own ACL stores and configuration
> Uis like (Ranger-Argus not sure what are they calling it now).
>
>
> On Wed, Mar 25, 2015 at 9:26 PM, Neha Narkhede <neha@confluent.io<mailto:
> neha@confluent.io>> wrote:
> Parth,
>
> We can make some 15 mins or so to discuss this at the next KIP hangout.
>
> Thanks,
> Neha
>
> On Wed, Mar 25, 2015 at 1:07 PM, Parth Brahmbhatt <
> pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>> wrote:
>
> Hi all,
>
> I have modified the KIP to reflect the recent change request from the
> reviewers. I have been working on the code and I have the server side code
> for authorization ready. I am now modifying the command line utilities. I
> would really appreciate if some of the committers can spend sometime to
> review the KIP so we can make progress on this.
>
> Thanks
> Parth
>
> On 3/18/15, 2:20 PM, "Michael Herstine" <mherstine@linkedin.com.INVALID
> <ma...@linkedin.com.INVALID>>
> wrote:
>
> >Hi Parth,
> >
> >Thanks! A few questions:
> >
> >1. Do you want to permit rules in your ACLs that DENY access as well as
> >ALLOW? This can be handy setting up rules that have exceptions. E.g.
> >“Allow principal P to READ resource R from all hosts” with “Deny principal
> >P READ access to resource R from host H1” in combination would allow P to
> >READ R from all hosts *except* H1.
> >
> >2. When a topic is newly created, will there be an ACL created for it? If
> >not, would that not deny subsequent access to it?
> >
> >(nit) Maybe use Principal instead of String to represent principals?
> >
> >
> >On 3/9/15, 11:48 AM, "Don Bosco Durai" <bosco@apache.org<mailto:
> bosco@apache.org>> wrote:
> >
> >>Parth
> >>
> >>Overall it is looking good. Couple of questionsŠ
> >>
> >>- Can you give an example how the policies will look like in the default
> >>implementation?
> >>- In the operations, can we support ³CONNECT² also? This can be used
> >>during Session connection
> >>- Regarding access control for ³Topic Creation², since we can¹t do it on
> >>the server side, can we de-scope it for? And plan it as a future feature
> >>request?
> >>
> >>Thanks
> >>
> >>Bosco
> >>
> >>
> >>
> >>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io>>
> wrote:
> >>
> >>>Hi Parth,
> >>>            Thanks for putting this together. Overall it looks good to
> >>>            me. Although AdminUtils is a concern KIP-4  can probably fix
> >>>            that part.
> >>>Thanks,
> >>>Harsha
> >>>
> >>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
> >>>> Forgot to add links to wiki and jira.
> >>>>
> >>>> Link to wiki:
> >>>>
> >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizatio
> >>>>n
> >>>>+
> >>>>Interface
> >>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
> >>>>
> >>>> Thanks
> >>>> Parth
> >>>>
> >>>> From: Parth Brahmbhatt
> >>>> <pbrahmbhatt@hortonworks.com<mailto:pbrahmbhatt@hortonworks.com
> ><ma...@hortonworks.com>>
> >>>> Date: Thursday, March 5, 2015 at 10:33 AM
> >>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org><mailto:
> dev@kafka.apache.org>"
> >>>> <de...@kafka.apache.org><mailto:
> dev@kafka.apache.org>>
> >>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
> >>>>
> >>>> Hi,
> >>>>
> >>>> KIP-11 is open for discussion , I have updated the wiki with the
> >>>>design
> >>>> and open questions.
> >>>>
> >>>> Thanks
> >>>> Parth
> >>
> >>
> >
>
>
>
>
> --
> Thanks,
> Neha
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Hi Gwen,

Thanks a lot for taking the time to review this. I have tried to address all your questions below.

Thanks
Parth
On 3/28/15, 8:08 PM, "Gwen Shapira" <gs...@cloudera.com>> wrote:

Preparing for Tuesday meeting, I went over the KIP :)

First, Parth did an amazing job, the KIP is fantastic - detailed and
readable. Thank you!

Second, I have a loooong list of questions :) No objections, just some
things I'm unclear on and random minor comments. In general, I like
the design, I just feel I'm missing parts of the picture.

1. "Yes, Create topic will have an optional acls, the output of
describe will display owner and acls and alter topic will allow to
modify the acls.”  - will be nice to see what the CLI will look like.

  *   I will modify the KIP but I was going to add “—acl <acl-file.json>” to create-topic and alter-topic.

2. I like the addition of Topic owner. We made the mistake of
forgetting about it when adding authorization to Sqoop2. We probably
want to add “chown” command to the topic commands.

  *   Again we can add “—owner <user-name>” to alter topic.

3. "Kafka server will read "authorizer.class” config value at startup
time, create an instance of the specified class and call initialize
method."
We’ll need to validate that users specify only one of those.

  *   The config type will be string so type validation should take care of it.

4. "One added assumption is that on non-secure connections the session
will have principal set to an object whose name() method will return
"Anonymous”."
Can we keep DrWho? :)

  *   Sure, its up to you actually as you are the owner of the jira that introduces session concept.

5. "For cluster actions that do not apply to a specific topic like
CREATE we have 2 options. We can either add a broker config called
broker.acls which will point to a json file. This file will be
available on all broker hosts and authorizer will read the acls on
initialization and keep refreshing it every X minutes. Any changes
will require re-distribution of the acl json file. Alternatively we
can add a zookeeper path /brokers/acls and store the acl json as data.
Authorizer can refresh the acl from json every X minutes. In absence
of broker acls the authorizer will fail open, in other words it will
allow all users from all hosts to perform all cluster actions”
I prefer a file to ZK - since thats where we store all use-defined
configurations for now. Everyone knows how to secure a file system :)

  *   I will let everyone vote, file system is fine by me.

6. "When an Acl is missing , this implementation will always fail open
for backward compatibility. “ <- agree, but we need to document that
this makes the default authorizer non-secure

  *   Sure.

7. "If the value of authorizer.class.name is null, in secure mode the
cluster will fail with ConfigException. In non secure mode in absence
of config value forauthorizer.class.name the server will allow all
requests to all topics that , even if the topic has configured acls”
<- I don’t think Kafka has “secure mode” - it can support SSL and
plaintext (un-authenticated) on two different ports simultaneously. I
don’t object to adding such configuration, but we need to decide
exactly what it means.

  *   This is one area of confusion so I will add an open question.

8. "Currently all topic creation/modification/deletion actions are
performed using KafkaAdminUtil which mostly interacts directly with
zookeeper instead of forwarding requests to a broker host. Given all
the code is executed on client side there is no easy way to perform
authorization. “ - since we are adding the admin protocol requests in
KIP-4, perhaps addressing those makes sense.

  *   Yes, We will have to wait for KIP-4 to be delivered.

9. I didn’t see a specification of what is “resource”, I assume its an
enum with things like Topic and… ?

  *   Resource is not an enum, Right now its just name of the topic. Currently we will not be able to support cluster actions like CREATE in which case the resource can be some constant string like “Kafka-Cluster”.

10. I’m also unclear on where and how “PermissionType” will be used
and what does “DENY takes precedence” mean here.

  *   I originally did not have the “PermissionType” but someone suggested in DISCUSS thread that we should add support for ALLOW and DENY acls. The use case is to support acls like “Allow user U to Perform READ from all Hosts except from Host1 and Host2”. Deny take precedence only means if you have define 2 ACLs for the same user/resource/operation. DENY acl will take effect I.e. “Allow user X to read from *” and “Deny User X to Read from host1 and host2” will satisfy the use case described in the previous statement.

11. "What does acls on zookeeper node look like given all our admin
APIs are currently performed directly from client?” <- “secure mode”
Kafka will need to set ACLs on ZK (using ZK’s model of ACLs) and both
Kafka and everyone else will need to use them (this is limited to
Kerberos authentication AFAIK.)

  *   Yes, again I think this can be a separate issue for now and can only be worked on after KIP-4 is delivered.

12. "Do we want to support group acls as part of this authorizer? Do
we want to support principal to local user mapping? If yes we need to
add plugins for UserToGroupMapper and PrincipalToUserMapper.” <-
Sentry uses Groups for authorizing, so we need to support that. I
figured that as long as the API specifies Principal, it typically
contains both user and group, so nothing else is needed. Did I miss
anything?

  *   Once we support group acls we will need someway to indicate if a principal is of type group or user(as part of acl) or we can have group acls and user acls stored separately with topic config. We will also need a way to map an authenticated user to list of groups the user belongs to.

13. It looks like the Authorizer stores the ACLs and not Kafka. So we
need an API for Kafka to notify Authorizer when a topic is added and
when ACLs are modified, right? I didn’t see that.

  *   ACLs will be stored under /topic/config at time of topic creation in json format. Authorizer will get these acls using newly introduced TopicConfigCache which gets updated anytime topic config changes.

14. Are we going to have any API for Kafka to give out the ACLs on a
topic? Or we leave this to the Authorizer?

  *   I think it is better to leave this out side of Kafka. Mainly because most 3rd party authorizers will have their own ACL stores and configuration Uis like (Ranger-Argus not sure what are they calling it now).


On Wed, Mar 25, 2015 at 9:26 PM, Neha Narkhede <ne...@confluent.io>> wrote:
Parth,

We can make some 15 mins or so to discuss this at the next KIP hangout.

Thanks,
Neha

On Wed, Mar 25, 2015 at 1:07 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com<ma...@hortonworks.com>> wrote:

Hi all,

I have modified the KIP to reflect the recent change request from the
reviewers. I have been working on the code and I have the server side code
for authorization ready. I am now modifying the command line utilities. I
would really appreciate if some of the committers can spend sometime to
review the KIP so we can make progress on this.

Thanks
Parth

On 3/18/15, 2:20 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>>
wrote:

>Hi Parth,
>
>Thanks! A few questions:
>
>1. Do you want to permit rules in your ACLs that DENY access as well as
>ALLOW? This can be handy setting up rules that have exceptions. E.g.
>“Allow principal P to READ resource R from all hosts” with “Deny principal
>P READ access to resource R from host H1” in combination would allow P to
>READ R from all hosts *except* H1.
>
>2. When a topic is newly created, will there be an ACL created for it? If
>not, would that not deny subsequent access to it?
>
>(nit) Maybe use Principal instead of String to represent principals?
>
>
>On 3/9/15, 11:48 AM, "Don Bosco Durai" <bo...@apache.org>> wrote:
>
>>Parth
>>
>>Overall it is looking good. Couple of questionsŠ
>>
>>- Can you give an example how the policies will look like in the default
>>implementation?
>>- In the operations, can we support ³CONNECT² also? This can be used
>>during Session connection
>>- Regarding access control for ³Topic Creation², since we can¹t do it on
>>the server side, can we de-scope it for? And plan it as a future feature
>>request?
>>
>>Thanks
>>
>>Bosco
>>
>>
>>
>>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io>> wrote:
>>
>>>Hi Parth,
>>>            Thanks for putting this together. Overall it looks good to
>>>            me. Although AdminUtils is a concern KIP-4  can probably fix
>>>            that part.
>>>Thanks,
>>>Harsha
>>>
>>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>>> Forgot to add links to wiki and jira.
>>>>
>>>> Link to wiki:
>>>>
>>>>
https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizatio
>>>>n
>>>>+
>>>>Interface
>>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>>>
>>>> Thanks
>>>> Parth
>>>>
>>>> From: Parth Brahmbhatt
>>>> <pb...@hortonworks.com>>
>>>> Date: Thursday, March 5, 2015 at 10:33 AM
>>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org>"
>>>> <de...@kafka.apache.org>>
>>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>>>
>>>> Hi,
>>>>
>>>> KIP-11 is open for discussion , I have updated the wiki with the
>>>>design
>>>> and open questions.
>>>>
>>>> Thanks
>>>> Parth
>>
>>
>




--
Thanks,
Neha

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Gwen Shapira <gs...@cloudera.com>]

Preparing for Tuesday meeting, I went over the KIP :)

First, Parth did an amazing job, the KIP is fantastic - detailed and
readable. Thank you!

Second, I have a loooong list of questions :) No objections, just some
things I'm unclear on and random minor comments. In general, I like
the design, I just feel I'm missing parts of the picture.

1. "Yes, Create topic will have an optional acls, the output of
describe will display owner and acls and alter topic will allow to
modify the acls.”  - will be nice to see what the CLI will look like.

2. I like the addition of Topic owner. We made the mistake of
forgetting about it when adding authorization to Sqoop2. We probably
want to add “chown” command to the topic commands.

3. "Kafka server will read "authorizer.class” config value at startup
time, create an instance of the specified class and call initialize
method."
We’ll need to validate that users specify only one of those.

4. "One added assumption is that on non-secure connections the session
will have principal set to an object whose name() method will return
"Anonymous”."
Can we keep DrWho? :)

5. "For cluster actions that do not apply to a specific topic like
CREATE we have 2 options. We can either add a broker config called
broker.acls which will point to a json file. This file will be
available on all broker hosts and authorizer will read the acls on
initialization and keep refreshing it every X minutes. Any changes
will require re-distribution of the acl json file. Alternatively we
can add a zookeeper path /brokers/acls and store the acl json as data.
Authorizer can refresh the acl from json every X minutes. In absence
of broker acls the authorizer will fail open, in other words it will
allow all users from all hosts to perform all cluster actions”
I prefer a file to ZK - since thats where we store all use-defined
configurations for now. Everyone knows how to secure a file system :)

6. "When an Acl is missing , this implementation will always fail open
for backward compatibility. “ <- agree, but we need to document that
this makes the default authorizer non-secure

7. "If the value of authorizer.class.name is null, in secure mode the
cluster will fail with ConfigException. In non secure mode in absence
of config value forauthorizer.class.name the server will allow all
requests to all topics that , even if the topic has configured acls”
<- I don’t think Kafka has “secure mode” - it can support SSL and
plaintext (un-authenticated) on two different ports simultaneously. I
don’t object to adding such configuration, but we need to decide
exactly what it means.

8. "Currently all topic creation/modification/deletion actions are
performed using KafkaAdminUtil which mostly interacts directly with
zookeeper instead of forwarding requests to a broker host. Given all
the code is executed on client side there is no easy way to perform
authorization. “ - since we are adding the admin protocol requests in
KIP-4, perhaps addressing those makes sense.

9. I didn’t see a specification of what is “resource”, I assume its an
enum with things like Topic and… ?

10. I’m also unclear on where and how “PermissionType” will be used
and what does “DENY takes precedence” mean here.

11. "What does acls on zookeeper node look like given all our admin
APIs are currently performed directly from client?” <- “secure mode”
Kafka will need to set ACLs on ZK (using ZK’s model of ACLs) and both
Kafka and everyone else will need to use them (this is limited to
Kerberos authentication AFAIK.)

12. "Do we want to support group acls as part of this authorizer? Do
we want to support principal to local user mapping? If yes we need to
add plugins for UserToGroupMapper and PrincipalToUserMapper.” <-
Sentry uses Groups for authorizing, so we need to support that. I
figured that as long as the API specifies Principal, it typically
contains both user and group, so nothing else is needed. Did I miss
anything?

13. It looks like the Authorizer stores the ACLs and not Kafka. So we
need an API for Kafka to notify Authorizer when a topic is added and
when ACLs are modified, right? I didn’t see that.

14. Are we going to have any API for Kafka to give out the ACLs on a
topic? Or we leave this to the Authorizer?



On Wed, Mar 25, 2015 at 9:26 PM, Neha Narkhede <ne...@confluent.io> wrote:
> Parth,
>
> We can make some 15 mins or so to discuss this at the next KIP hangout.
>
> Thanks,
> Neha
>
> On Wed, Mar 25, 2015 at 1:07 PM, Parth Brahmbhatt <
> pbrahmbhatt@hortonworks.com> wrote:
>
>> Hi all,
>>
>> I have modified the KIP to reflect the recent change request from the
>> reviewers. I have been working on the code and I have the server side code
>> for authorization ready. I am now modifying the command line utilities. I
>> would really appreciate if some of the committers can spend sometime to
>> review the KIP so we can make progress on this.
>>
>> Thanks
>> Parth
>>
>> On 3/18/15, 2:20 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
>> wrote:
>>
>> >Hi Parth,
>> >
>> >Thanks! A few questions:
>> >
>> >1. Do you want to permit rules in your ACLs that DENY access as well as
>> >ALLOW? This can be handy setting up rules that have exceptions. E.g.
>> >“Allow principal P to READ resource R from all hosts” with “Deny principal
>> >P READ access to resource R from host H1” in combination would allow P to
>> >READ R from all hosts *except* H1.
>> >
>> >2. When a topic is newly created, will there be an ACL created for it? If
>> >not, would that not deny subsequent access to it?
>> >
>> >(nit) Maybe use Principal instead of String to represent principals?
>> >
>> >
>> >On 3/9/15, 11:48 AM, "Don Bosco Durai" <bo...@apache.org> wrote:
>> >
>> >>Parth
>> >>
>> >>Overall it is looking good. Couple of questionsŠ
>> >>
>> >>- Can you give an example how the policies will look like in the default
>> >>implementation?
>> >>- In the operations, can we support ³CONNECT² also? This can be used
>> >>during Session connection
>> >>- Regarding access control for ³Topic Creation², since we can¹t do it on
>> >>the server side, can we de-scope it for? And plan it as a future feature
>> >>request?
>> >>
>> >>Thanks
>> >>
>> >>Bosco
>> >>
>> >>
>> >>
>> >>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io> wrote:
>> >>
>> >>>Hi Parth,
>> >>>            Thanks for putting this together. Overall it looks good to
>> >>>            me. Although AdminUtils is a concern KIP-4  can probably fix
>> >>>            that part.
>> >>>Thanks,
>> >>>Harsha
>> >>>
>> >>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>> >>>> Forgot to add links to wiki and jira.
>> >>>>
>> >>>> Link to wiki:
>> >>>>
>> >>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizatio
>> >>>>n
>> >>>>+
>> >>>>Interface
>> >>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>> >>>>
>> >>>> Thanks
>> >>>> Parth
>> >>>>
>> >>>> From: Parth Brahmbhatt
>> >>>> <pb...@hortonworks.com>>
>> >>>> Date: Thursday, March 5, 2015 at 10:33 AM
>> >>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org>"
>> >>>> <de...@kafka.apache.org>>
>> >>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>> KIP-11 is open for discussion , I have updated the wiki with the
>> >>>>design
>> >>>> and open questions.
>> >>>>
>> >>>> Thanks
>> >>>> Parth
>> >>
>> >>
>> >
>>
>>
>
>
> --
> Thanks,
> Neha

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Neha Narkhede <ne...@confluent.io>]

Parth,

We can make some 15 mins or so to discuss this at the next KIP hangout.

Thanks,
Neha

On Wed, Mar 25, 2015 at 1:07 PM, Parth Brahmbhatt <
pbrahmbhatt@hortonworks.com> wrote:

> Hi all,
>
> I have modified the KIP to reflect the recent change request from the
> reviewers. I have been working on the code and I have the server side code
> for authorization ready. I am now modifying the command line utilities. I
> would really appreciate if some of the committers can spend sometime to
> review the KIP so we can make progress on this.
>
> Thanks
> Parth
>
> On 3/18/15, 2:20 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
> wrote:
>
> >Hi Parth,
> >
> >Thanks! A few questions:
> >
> >1. Do you want to permit rules in your ACLs that DENY access as well as
> >ALLOW? This can be handy setting up rules that have exceptions. E.g.
> >“Allow principal P to READ resource R from all hosts” with “Deny principal
> >P READ access to resource R from host H1” in combination would allow P to
> >READ R from all hosts *except* H1.
> >
> >2. When a topic is newly created, will there be an ACL created for it? If
> >not, would that not deny subsequent access to it?
> >
> >(nit) Maybe use Principal instead of String to represent principals?
> >
> >
> >On 3/9/15, 11:48 AM, "Don Bosco Durai" <bo...@apache.org> wrote:
> >
> >>Parth
> >>
> >>Overall it is looking good. Couple of questionsŠ
> >>
> >>- Can you give an example how the policies will look like in the default
> >>implementation?
> >>- In the operations, can we support ³CONNECT² also? This can be used
> >>during Session connection
> >>- Regarding access control for ³Topic Creation², since we can¹t do it on
> >>the server side, can we de-scope it for? And plan it as a future feature
> >>request?
> >>
> >>Thanks
> >>
> >>Bosco
> >>
> >>
> >>
> >>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io> wrote:
> >>
> >>>Hi Parth,
> >>>            Thanks for putting this together. Overall it looks good to
> >>>            me. Although AdminUtils is a concern KIP-4  can probably fix
> >>>            that part.
> >>>Thanks,
> >>>Harsha
> >>>
> >>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
> >>>> Forgot to add links to wiki and jira.
> >>>>
> >>>> Link to wiki:
> >>>>
> >>>>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizatio
> >>>>n
> >>>>+
> >>>>Interface
> >>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
> >>>>
> >>>> Thanks
> >>>> Parth
> >>>>
> >>>> From: Parth Brahmbhatt
> >>>> <pb...@hortonworks.com>>
> >>>> Date: Thursday, March 5, 2015 at 10:33 AM
> >>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org>"
> >>>> <de...@kafka.apache.org>>
> >>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
> >>>>
> >>>> Hi,
> >>>>
> >>>> KIP-11 is open for discussion , I have updated the wiki with the
> >>>>design
> >>>> and open questions.
> >>>>
> >>>> Thanks
> >>>> Parth
> >>
> >>
> >
>
>


-- 
Thanks,
Neha

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Hi all,

I have modified the KIP to reflect the recent change request from the
reviewers. I have been working on the code and I have the server side code
for authorization ready. I am now modifying the command line utilities. I
would really appreciate if some of the committers can spend sometime to
review the KIP so we can make progress on this.

Thanks
Parth

On 3/18/15, 2:20 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
wrote:

>Hi Parth,
>
>Thanks! A few questions:
>
>1. Do you want to permit rules in your ACLs that DENY access as well as
>ALLOW? This can be handy setting up rules that have exceptions. E.g.
>“Allow principal P to READ resource R from all hosts” with “Deny principal
>P READ access to resource R from host H1” in combination would allow P to
>READ R from all hosts *except* H1.
>
>2. When a topic is newly created, will there be an ACL created for it? If
>not, would that not deny subsequent access to it?
>
>(nit) Maybe use Principal instead of String to represent principals?
>
>
>On 3/9/15, 11:48 AM, "Don Bosco Durai" <bo...@apache.org> wrote:
>
>>Parth
>>
>>Overall it is looking good. Couple of questionsŠ
>>
>>- Can you give an example how the policies will look like in the default
>>implementation?
>>- In the operations, can we support ³CONNECT² also? This can be used
>>during Session connection
>>- Regarding access control for ³Topic Creation², since we can¹t do it on
>>the server side, can we de-scope it for? And plan it as a future feature
>>request?
>>
>>Thanks
>>
>>Bosco
>>
>> 
>>
>>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io> wrote:
>>
>>>Hi Parth,
>>>            Thanks for putting this together. Overall it looks good to
>>>            me. Although AdminUtils is a concern KIP-4  can probably fix
>>>            that part.
>>>Thanks,
>>>Harsha
>>>
>>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>>> Forgot to add links to wiki and jira.
>>>> 
>>>> Link to wiki:
>>>> 
>>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizatio
>>>>n
>>>>+
>>>>Interface
>>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>>> 
>>>> Thanks
>>>> Parth
>>>> 
>>>> From: Parth Brahmbhatt
>>>> <pb...@hortonworks.com>>
>>>> Date: Thursday, March 5, 2015 at 10:33 AM
>>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org>"
>>>> <de...@kafka.apache.org>>
>>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>>> 
>>>> Hi,
>>>> 
>>>> KIP-11 is open for discussion , I have updated the wiki with the
>>>>design
>>>> and open questions.
>>>> 
>>>> Thanks
>>>> Parth
>>
>>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Hi Michael,

Thanks for taking the time to review. Currently I did not plan on adding
“Deny” but I guess it can’t hurt except for adding more constructs would
probably make the acls more complex.

When a topic is created with no acls provided , I was planning to add a
default ACL which would allow access to everyone from all hosts.

I am assuming you are referring to principal in Acl and acls were supposed
to be provided in property files, stored in zk so I thought it is better
to just refer to a string. We will always be using
session.principal.getName to get the actual principal name.

Thanks
Parth

On 3/18/15, 2:20 PM, "Michael Herstine" <mh...@linkedin.com.INVALID>
wrote:

>Hi Parth,
>
>Thanks! A few questions:
>
>1. Do you want to permit rules in your ACLs that DENY access as well as
>ALLOW? This can be handy setting up rules that have exceptions. E.g.
>“Allow principal P to READ resource R from all hosts” with “Deny principal
>P READ access to resource R from host H1” in combination would allow P to
>READ R from all hosts *except* H1.
>
>2. When a topic is newly created, will there be an ACL created for it? If
>not, would that not deny subsequent access to it?
>
>(nit) Maybe use Principal instead of String to represent principals?
>
>
>On 3/9/15, 11:48 AM, "Don Bosco Durai" <bo...@apache.org> wrote:
>
>>Parth
>>
>>Overall it is looking good. Couple of questionsŠ
>>
>>- Can you give an example how the policies will look like in the default
>>implementation?
>>- In the operations, can we support ³CONNECT² also? This can be used
>>during Session connection
>>- Regarding access control for ³Topic Creation², since we can¹t do it on
>>the server side, can we de-scope it for? And plan it as a future feature
>>request?
>>
>>Thanks
>>
>>Bosco
>>
>> 
>>
>>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io> wrote:
>>
>>>Hi Parth,
>>>            Thanks for putting this together. Overall it looks good to
>>>            me. Although AdminUtils is a concern KIP-4  can probably fix
>>>            that part.
>>>Thanks,
>>>Harsha
>>>
>>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>>> Forgot to add links to wiki and jira.
>>>> 
>>>> Link to wiki:
>>>> 
>>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorizatio
>>>>n
>>>>+
>>>>Interface
>>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>>> 
>>>> Thanks
>>>> Parth
>>>> 
>>>> From: Parth Brahmbhatt
>>>> <pb...@hortonworks.com>>
>>>> Date: Thursday, March 5, 2015 at 10:33 AM
>>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org>"
>>>> <de...@kafka.apache.org>>
>>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>>> 
>>>> Hi,
>>>> 
>>>> KIP-11 is open for discussion , I have updated the wiki with the
>>>>design
>>>> and open questions.
>>>> 
>>>> Thanks
>>>> Parth
>>
>>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Michael Herstine <mh...@linkedin.com.INVALID>]

Hi Parth,

Thanks! A few questions:

1. Do you want to permit rules in your ACLs that DENY access as well as
ALLOW? This can be handy setting up rules that have exceptions. E.g.
“Allow principal P to READ resource R from all hosts” with “Deny principal
P READ access to resource R from host H1” in combination would allow P to
READ R from all hosts *except* H1.

2. When a topic is newly created, will there be an ACL created for it? If
not, would that not deny subsequent access to it?

(nit) Maybe use Principal instead of String to represent principals?


On 3/9/15, 11:48 AM, "Don Bosco Durai" <bo...@apache.org> wrote:

>Parth
>
>Overall it is looking good. Couple of questionsŠ
>
>- Can you give an example how the policies will look like in the default
>implementation?
>- In the operations, can we support ³CONNECT² also? This can be used
>during Session connection
>- Regarding access control for ³Topic Creation², since we can¹t do it on
>the server side, can we de-scope it for? And plan it as a future feature
>request?
>
>Thanks
>
>Bosco
>
> 
>
>On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io> wrote:
>
>>Hi Parth,
>>            Thanks for putting this together. Overall it looks good to
>>            me. Although AdminUtils is a concern KIP-4  can probably fix
>>            that part.
>>Thanks,
>>Harsha
>>
>>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>>> Forgot to add links to wiki and jira.
>>> 
>>> Link to wiki:
>>> 
>>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization
>>>+
>>>Interface
>>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>>> 
>>> Thanks
>>> Parth
>>> 
>>> From: Parth Brahmbhatt
>>> <pb...@hortonworks.com>>
>>> Date: Thursday, March 5, 2015 at 10:33 AM
>>> To: "dev@kafka.apache.org<ma...@kafka.apache.org>"
>>> <de...@kafka.apache.org>>
>>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>>> 
>>> Hi,
>>> 
>>> KIP-11 is open for discussion , I have updated the wiki with the design
>>> and open questions.
>>> 
>>> Thanks
>>> Parth
>
>

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Don Bosco Durai <bo...@apache.org>]

Parth

Overall it is looking good. Couple of questionsŠ

- Can you give an example how the policies will look like in the default
implementation?
- In the operations, can we support ³CONNECT² also? This can be used
during Session connection
- Regarding access control for ³Topic Creation², since we can¹t do it on
the server side, can we de-scope it for? And plan it as a future feature
request?

Thanks

Bosco

 

On 3/6/15, 8:10 AM, "Harsha" <ka...@harsha.io> wrote:

>Hi Parth,
>            Thanks for putting this together. Overall it looks good to
>            me. Although AdminUtils is a concern KIP-4  can probably fix
>            that part.
>Thanks,
>Harsha
>
>On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
>> Forgot to add links to wiki and jira.
>> 
>> Link to wiki:
>> 
>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+
>>Interface
>> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
>> 
>> Thanks
>> Parth
>> 
>> From: Parth Brahmbhatt
>> <pb...@hortonworks.com>>
>> Date: Thursday, March 5, 2015 at 10:33 AM
>> To: "dev@kafka.apache.org<ma...@kafka.apache.org>"
>> <de...@kafka.apache.org>>
>> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
>> 
>> Hi,
>> 
>> KIP-11 is open for discussion , I have updated the wiki with the design
>> and open questions.
>> 
>> Thanks
>> Parth

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Harsha <ka...@harsha.io>]

Hi Parth,
            Thanks for putting this together. Overall it looks good to
            me. Although AdminUtils is a concern KIP-4  can probably fix
            that part. 
Thanks,
Harsha

On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
> Forgot to add links to wiki and jira.
> 
> Link to wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface
> Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688
> 
> Thanks
> Parth
> 
> From: Parth Brahmbhatt
> <pb...@hortonworks.com>>
> Date: Thursday, March 5, 2015 at 10:33 AM
> To: "dev@kafka.apache.org<ma...@kafka.apache.org>"
> <de...@kafka.apache.org>>
> Subject: [DISCUSS] KIP-11- Authorization design for kafka security
> 
> Hi,
> 
> KIP-11 is open for discussion , I have updated the wiki with the design
> and open questions.
> 
> Thanks
> Parth

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Parth Brahmbhatt <pb...@hortonworks.com>]

Forgot to add links to wiki and jira.

Link to wiki: https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface
Link to Jira: https://issues.apache.org/jira/browse/KAFKA-1688

Thanks
Parth

From: Parth Brahmbhatt <pb...@hortonworks.com>>
Date: Thursday, March 5, 2015 at 10:33 AM
To: "dev@kafka.apache.org<ma...@kafka.apache.org>" <de...@kafka.apache.org>>
Subject: [DISCUSS] KIP-11- Authorization design for kafka security

Hi,

KIP-11 is open for discussion , I have updated the wiki with the design and open questions.

Thanks
Parth