You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Rajini Sivaram <ra...@gmail.com> on 2019/08/06 21:31:43 UTC
[DISCUSS] KIP-504 - Add new Java Authorizer Interface
Hi all,
I have created a KIP to replace the Scala Authorizer API with a new Java
API:
-
https://cwiki.apache.org/confluence/display/KAFKA/KIP-504+-+Add+new+Java+Authorizer+Interface
This is replacement for KIP-50 which was accepted but never merged. Apart
from moving to a Java API consistent with other pluggable interfaces in the
broker, KIP-504 also attempts to address known limitations in the
authorizer. If you have come across other limitations that you would like
to see addressed in the new API, please raise these on the discussion
thread so that we can consider those too. All suggestions and feedback are
welcome.
Thank you,
Rajini
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Colin McCabe <cm...@apache.org>.
Hi Rajini,
Thanks for the KIP. This will be a great improvement.
Why not just pass the cluster ID directly to Authorizer#start, rather than dealing with the ClusterResourceListener interface? That seems like it would be simpler. If authorizers don't need that information, they can ignore that parameter.
Do we really need to change the configuration key name from authorizer.class.name to authorizer.class? We should be able to use introspection to see whether it's a subclass of the old or the new interface, right? This would avoid having two configuration keys that were very similar.
The Authorizer JavaDoc should have a comment about thread safety. I assume that all of the Authorizer methods need to be thread-safe, right?
For CreateAcls and DeleteAcls, it seems like we would be better off just using AclBinding objects. An AclBinding object contains a ResourcePattern and an AccessControlEntry. There is no need to pass them separately.
It seems like we don't need all the different overloads of deleteAcls. We can just pass in an AclBindingFilter, and that can delete whatever is needed. For example, if you want to delete all the ACLs for a specific resource, you could pass in an AclBindingFilter with AccessControlEntryFilter.ANY. It would be nice to have deleteAcls return the number of entries that were deleted.
I like the way KafkaRequestContext is split from Action. That will be helpful.
We need a way of listing all the ACLs, because that will be needed for the AdminClient. The simplest API would be just passing in an AclBindingFilter. Listing the ACLs for a specific resource is just a special case, which doesn't need its own function. Since the number of ACLs could be quite large, we probably want to return an iterator rather than a full collection. We can eventually make the iterator load results lazily if needed.
AuthorizationMode doesn't really make sense to me. It seems to be designed to allow us to log different kinds of authorization failures differently. But it would be simpler to just split the API into two parts: getting an authorization result, and logging it. The Authorizer#authorize method should return a result that we can later pass to an Authorizer#logFailure method. Then, when code wants to try to authorize against X, and then if that fails authorize against Y, it can simply only pass the second failure result to Authorizer#logFailure to avoid creating extra log messages. Or we could create a method AuthorizationResult#logFaliure().
For requestType, I don't think this should be a string. The 16-bit API key is clearer and more concise. The request names currently aren't part of the stable API, and it would be nice to keep it that way. To distinguish between follower fetches and consumer fetches, we should just have a separate way of knowing that the broker made the request. Maybe something like KafkaRequestContext#Originator. Then we can have Originator#CLIENT and Originator#BROKER.
It's hard to understand what Action#count is for. If we want to log something like "failed to authorize deleting 123 partitions", it would be better to just let the caller supply some custom text in Authorizer#logFailure.
Thanks again for the KIP.
best,
Colin
On Thu, Aug 8, 2019, at 02:03, Rajini Sivaram wrote:
> Hi Don,
>
> Thanks for reviewing the KIP.
>
> 1. I had this originally as a single Action, but thought it may be useful
> to support batched authorize calls as well and keep it consistent with
> other methods. Single requests can contain multiple topics. For example a
> produce request can contain records for several partitions of different
> topics. Broker could potentially authorize these together. For
> SimpleAclAuthorizer, batched authorize methods don't provide any
> optimisation since lookup is based on resources followed by the matching
> logic. But some authorizers may manage ACLs by user principal rather than
> resource and may be able to optimize batched requests. I am ok with using
> single Action if this is likely to cause issues.
> 2. If you have two listeners, one for inter-broker traffic and another for
> external clients, start method is invoked twice, once for each listener. On
> second thought, that may be confusing and a single start() invocation that
> provides all listener information and returns multiple futures would be
> better. Will update the KIP.
> 3. A typical example is a consumer subscribing to a regex pattern. We
> request all topic metadata from the broker in order to decide whether the
> pattern matches, expecting to receive a list of authorised topics. The user
> is not asking to subscribe to an unauthorized topic. If there are 10000
> topics in the cluster and the user has access to 100 of them, at the moment
> we log 9900 DENIED log entries at INFO level in SimpleAclAuthorizer. The
> proposal is to authorize this request with AuthorizationMode.FILTER, so
> that authorizers can log resources that are filtered out at lower level
> like DEBUG since this is not an attempt to access unauthorized resources.
> Brokers already handle these differently since no authorization error is
> returned to the client in these cases. Providing authorization mode to
> authorizers enables authorizer implementations to generate better audit
> logs.
> 4. Each request may contain multiple instances of the same authorizable
> resource. For example a produce request may contain records for 10
> partitions of the same topic. At the moment, we invoke authorize method 10
> times. The proposal is to invoke it once with count=10. The count is
> provided to authorizer just for audit logging purposes.
> 5. Authorizer implements Closeable, so you could use close() to flush
> audits?
>
> On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <bo...@apache.org> wrote:
>
> > Rajini
> >
> > Thanks for putting this together. It is looking good. I have few
> > questions...
> >
> > 1. List<AuthorizationResult> authorize(..., List<Action> actions). Do you
> > see a scenario where the broker will call authorize for multiple topics at
> > the same time? I can understand that during creating/deleting ACLS,
> > multiple permissions for multiple resources might be done. For authorize
> > call, would this be a case? And does the Authorize implementation will be
> > able to do performance optimization because of this? Or should we just keep
> > it simple? I don't see it as an issue from Apache Ranger side, but just
> > checking to see whether we need to be aware of something.
> > 2. Should I assume that the SecurityProtocol passed during start and the
> > one return by KafkaRequestContext.securityProtocol() will be the same?
> > CompletableFuture<Void> start(String listenerName, SecurityProtocol
> > securityProtocol);
> > KafkaRequestContext.securityProtocol()
> > 3. What is the purpose of AuthorizationMode? How does the broker decide
> > what mode to use when the authorize() method is called?
> > 4. Can we clarify "count" in Action a bit more? How is it used?
> > 5. Do you feel having "stop" along with "start" be helpful? E.g. In Ranger
> > we try to optimize the Audit writing by caching the logs for a fixed
> > interval. But when the Broker terminates, we do a forced flush. Having an
> > explicit "stop" might give us a formal way to flush our audits.
> >
> > Thanks
> >
> > Bosco
> >
> > On 8/7/19, 3:59 PM, "Rajini Sivaram" <ra...@gmail.com> wrote:
> >
> > Hi Ron/Harsha/Satish,
> >
> > Thanks for reviewing the KIP!
> >
> > We should perhaps have a wider discussion outside this KIP for
> > refactoring
> > clients so that others who are not following this KIP also notice the
> > discussion. Satish, would you like to start a discussion thread on dev?
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <
> > satish.duggana@gmail.com>
> > wrote:
> >
> > > I felt the same need when we want to add a pluggable API for core
> > > server functionality. This does not need to be part of this KIP, it
> > > can be a separate KIP. I can contribute those refactoring changes if
> > > others are OK with that.
> > >
> > > It is better to have a structure like below.
> > >
> > > kafka-common:
> > > common classes which can be used in any of the other modules in Kafka
> > > like client, Kafka-server-common and server etc.
> > >
> > > kafka-client-common:
> > > common classes which can be used in the client module. This can be
> > > part of client module itself.
> > >
> > > kafka-server-common:
> > > classes required only for kafka-server.
> > >
> > > Thanks.
> > > Satish.
> > >
> > > On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani <ka...@harsha.io>
> > > wrote:
> > > >
> > > > Thanks for the KIP Rajini.
> > > > Quick thought, it would be good to have a common module outside of
> > > clients
> > > > that only applies to server side interfaces & changes. It looks
> > like we
> > > are
> > > > increasingly in favor of using Java interface for pluggable
> > modules on
> > > the
> > > > broker side.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > >
> > > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <
> > rajinisivaram@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I have created a KIP to replace the Scala Authorizer API with a
> > new
> > > Java
> > > > > API:
> > > > >
> > > > > -
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > >
> > > > > This is replacement for KIP-50 which was accepted but never
> > merged.
> > > Apart
> > > > > from moving to a Java API consistent with other pluggable
> > interfaces
> > > in the
> > > > > broker, KIP-504 also attempts to address known limitations in the
> > > > > authorizer. If you have come across other limitations that you
> > would
> > > like
> > > > > to see addressed in the new API, please raise these on the
> > discussion
> > > > > thread so that we can consider those too. All suggestions and
> > feedback
> > > are
> > > > > welcome.
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Rajini
> > > > >
> > >
> >
> >
> >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Colin,
Thanks for the reviewing the KIP and the suggestions to improve.
*1. Why not just pass the cluster ID directly to Authorizer#start*
I was thinking that we already have APIs that use ClusterResourceListener,
so kept it consistent. But you are right, since we have a start method, we
could just pass it in. Updated the KIP.
*2. Do we really need to change the configuration key name
from authorizer.class.name <http://authorizer.class.name/> to
authorizer.class?*
I changed it thinking other classes like principal.builder.class didn't use
the `.name` suffix, so it was a good time to make config names consistent.
But I suppose it is ok to use the suffix since policy classes use that.
Updated the KIP.
*3. The Authorizer JavaDoc should have a comment about thread safety.*
Added.
*4. For CreateAcls and DeleteAcls, it seems like we would be better off
just using AclBinding objects.*
I looked at AclBinding and thought update operations worked better as
(ResourcePattern,
Set<AccessControlEntry>) rather than AclBinding which has (ResourcePattern,
AccessControlEntry). Using AclBinding means that authorization
implementations have to group the bindings by resource pattern first in
order to perform batched updates. But I suppose that is ok too, so updated.
*5. It seems like we don't need all the different overloads of deleteAcls.
We can just pass in an AclBindingFilter*
I was trying to avoid using the filter in the authorizer API since the
filtering logic is quite involved. But I suppose it is ok since we have the
matching logic in public classes, so authorizer implementations can use
those. Updated the KIP.
*6. We need a way of listing all the ACLs, because that will be needed for
the AdminClient. The simplest API would be just passing in an
AclBindingFilter. *
Same as delete, I was trying to avoid filter in the API, thinking we could
filter in the broker in common code. Have updated to use filter and return
Iterable.
*7. AuthorizationMode doesn't really make sense to me... **The
Authorizer#authorize method should return a result that we can later pass
to an Authorizer#logFailure method.*
Audit logging is not just about failures, it is about access that was
allowed too. We need to enable authorizers to generate appropriate audit
logs when authorizing. We already do, but this is trying to improve the
contents of the generated log. At the time of authorization, authorizers
have a lot of information that may be included in audit logs, for example,
how something was authorized or why some operation was denied. Separating
out the APIs removes the ability to include these in a single audit log
entry.
*8. For requestType, I don't think this should be a string. The 16-bit API
key is clearer and more concise. The request names currently aren't part
of the stable API, and it would be nice to keep it that way. To
distinguish between follower fetches and consumer fetches, we should just
have a separate way of knowing that the broker made the request. Maybe
something like KafkaRequestContext#Originator. Then we can have
Originator#CLIENT and Originator#BROKER.*
Since the request type is only used for logging, String feels like the
better option. The names are identical to those we use in metrics, so they
are externally visible. We could always add the 16-bit API key as well if
this interface is used for other pluggable APIs (I can change the name of
the method to avoid confusion). The follower fetch request is identified by
a flag in the body of the fetch request. Don't think we have a reliable way
of providing originator by looking at the header of any request.
*9. It's hard to understand what Action#count is for. If we want to log
something like "failed to authorize deleting 123 partitions", it would be
better to just let the caller supply some custom text in
Authorizer#logFailure.*
Basically, I would like to remove duplicate authorize calls for authorizing
topics from a single request. Today, for a Produce request to 10 partitions
of a topic, we get 10 log entries saying `Principal User:Alice is Allowed
Operation = Write from host = 127.0.0.1 for resource = Topic:sometopic`. I
wanted to retain the ability to generate that in authorizers, but invoke
authorize() method only once. Also wanted authorizers to be able to generate
a single line that says `Principal User:Alice is Allowed Operation = Write
from host = 127.0.0.1 for resource = Topic:sometopic authorizeCount=10`. Or
ignore count altogether. Authorizer doesn't know about partitions, so it is
more about how many times the resource was accessed in a compound request.
On Fri, Aug 9, 2019 at 1:08 PM Rajini Sivaram <ra...@gmail.com>
wrote:
> Hi Don,
>
> Thanks for the suggestions. A few responses below:
>
> 3.1 Can rename and improve docs if we keep this. Let's finish the
> discussion on Colin's suggestions regarding this first.
> 3.2 No, I was thinking of some requests that have an old way of
> authorizing and a new way where we have retained the old way for backward
> compatibility. One example is Cluster:Create permission to create topics.
> We have replaced this with fine-grained topic create access using Topic:Create
> for topic patterns. But we still check if user has Cluster:Create first.
> If Denied, the deny is ignored and we check Topic:Create. We dont want to
> log DENY for Cluster:Create at INFO level for this, since this is not a
> mandatory ACL for creating topics. We will get appropriate logs from the
> subsequent Topic:Create in this case.
> 3.3 They are not quite the same. FILTER implies that user actually
> requested access to and performed some operation on the filtered resources.
> LIST_AUTHORZED did not result in any actual access. User simply wanted to
> know what they are allowed to access.
> 3.4 Request types are Produce, JoinGroup, OffsetCommit etc. So that is
> different from authorization mode, operation etc.
>
>
> On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <bo...@apache.org> wrote:
>
>> Hi Rajini
>>
>> Thanks for clarifying. This is very helpful...
>>
>> #1 - On the Ranger side, we should be able to handle multiple requests at
>> the same time. I was just not sure how much processing overhead will be
>> there on the Broker side to split and then consolidate the results. If it
>> is negligible, then this is the better way. It will make it future proof.
>> #2 - I agree, having a single "start" call makes it cleaner. The
>> Authorization implementation will only have to do the initialization only
>> once.
>> #3.1 - Thanks for the clarification. I think it makes sense to have this.
>> The term "Mode" might not be explicit enough. Essentially it seems you want
>> the Authorizer to know the Intent/Purpose of the authorize call and let the
>> Authorizer decide what to log as Audit event. Changing the class/field name
>> or giving more documentation will do.
>> #3.2 - Regarding the option "OPTIONAL". Are you thinking from chaining
>> multiple Authorizer? If so, I am not sure whether the Broker would have
>> enough information to make that decision. I feel the Authorizer will be the
>> one who would have that knowledge. E.g. in Ranger we have explicit deny,
>> which means no matter what, the request should be denied for the user/group
>> or condition. So if you are thinking of chaining Authorizers, then
>> responses should have the third state, e.g. "DENIED_FINAL", in which case
>> if there is an Authorization chain, it will be stop and the request will be
>> denied and if it is just denied, then you might fall back to next
>> authorizer. If we don't have chaining of Authorizing in mind, then we
>> should reconsider OPTIONAL for now. Or clarify under which scenario
>> OPTIONAL will be called.
>> #3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't LIST_AUTHORIZED a
>> special case for "FILTER"?
>> #3.4 KafkaRequestContext. requestType() v/s Action. authorizationMode. I
>> am not sure about the overlap or ambiguity.
>> #4 - Cool. This is good, it will be less stress on the Authorizer. Ranger
>> already supports the "count" concept and also has batching capability to
>> aggregate similar requests to reduce the number of audit logs to write. We
>> should be able to pass this through.
>> #5 - Assuming if the object instance is going out of scope, we should be
>> fine. Not a super important ask. Ranger is already catching KILL signal for
>> clean up.
>>
>> Thanks again. These are good enhancements. Appreciate your efforts here.
>>
>> Bosco
>>
>>
>>
>> On 8/8/19, 2:03 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
>>
>> Hi Don,
>>
>> Thanks for reviewing the KIP.
>>
>> 1. I had this originally as a single Action, but thought it may be
>> useful
>> to support batched authorize calls as well and keep it consistent with
>> other methods. Single requests can contain multiple topics. For
>> example a
>> produce request can contain records for several partitions of
>> different
>> topics. Broker could potentially authorize these together. For
>> SimpleAclAuthorizer, batched authorize methods don't provide any
>> optimisation since lookup is based on resources followed by the
>> matching
>> logic. But some authorizers may manage ACLs by user principal rather
>> than
>> resource and may be able to optimize batched requests. I am ok with
>> using
>> single Action if this is likely to cause issues.
>> 2. If you have two listeners, one for inter-broker traffic and
>> another for
>> external clients, start method is invoked twice, once for each
>> listener. On
>> second thought, that may be confusing and a single start() invocation
>> that
>> provides all listener information and returns multiple futures would
>> be
>> better. Will update the KIP.
>> 3. A typical example is a consumer subscribing to a regex pattern. We
>> request all topic metadata from the broker in order to decide whether
>> the
>> pattern matches, expecting to receive a list of authorised topics.
>> The user
>> is not asking to subscribe to an unauthorized topic. If there are
>> 10000
>> topics in the cluster and the user has access to 100 of them, at the
>> moment
>> we log 9900 DENIED log entries at INFO level in SimpleAclAuthorizer.
>> The
>> proposal is to authorize this request with AuthorizationMode.FILTER,
>> so
>> that authorizers can log resources that are filtered out at lower
>> level
>> like DEBUG since this is not an attempt to access unauthorized
>> resources.
>> Brokers already handle these differently since no authorization error
>> is
>> returned to the client in these cases. Providing authorization mode to
>> authorizers enables authorizer implementations to generate better
>> audit
>> logs.
>> 4. Each request may contain multiple instances of the same
>> authorizable
>> resource. For example a produce request may contain records for 10
>> partitions of the same topic. At the moment, we invoke authorize
>> method 10
>> times. The proposal is to invoke it once with count=10. The count is
>> provided to authorizer just for audit logging purposes.
>> 5. Authorizer implements Closeable, so you could use close() to flush
>> audits?
>>
>> On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <bo...@apache.org>
>> wrote:
>>
>> > Rajini
>> >
>> > Thanks for putting this together. It is looking good. I have few
>> > questions...
>> >
>> > 1. List<AuthorizationResult> authorize(..., List<Action> actions).
>> Do you
>> > see a scenario where the broker will call authorize for multiple
>> topics at
>> > the same time? I can understand that during creating/deleting ACLS,
>> > multiple permissions for multiple resources might be done. For
>> authorize
>> > call, would this be a case? And does the Authorize implementation
>> will be
>> > able to do performance optimization because of this? Or should we
>> just keep
>> > it simple? I don't see it as an issue from Apache Ranger side, but
>> just
>> > checking to see whether we need to be aware of something.
>> > 2. Should I assume that the SecurityProtocol passed during start
>> and the
>> > one return by KafkaRequestContext.securityProtocol() will be the
>> same?
>> > CompletableFuture<Void> start(String listenerName, SecurityProtocol
>> > securityProtocol);
>> > KafkaRequestContext.securityProtocol()
>> > 3. What is the purpose of AuthorizationMode? How does the broker
>> decide
>> > what mode to use when the authorize() method is called?
>> > 4. Can we clarify "count" in Action a bit more? How is it used?
>> > 5. Do you feel having "stop" along with "start" be helpful? E.g. In
>> Ranger
>> > we try to optimize the Audit writing by caching the logs for a fixed
>> > interval. But when the Broker terminates, we do a forced flush.
>> Having an
>> > explicit "stop" might give us a formal way to flush our audits.
>> >
>> > Thanks
>> >
>> > Bosco
>> >
>> > On 8/7/19, 3:59 PM, "Rajini Sivaram" <ra...@gmail.com>
>> wrote:
>> >
>> > Hi Ron/Harsha/Satish,
>> >
>> > Thanks for reviewing the KIP!
>> >
>> > We should perhaps have a wider discussion outside this KIP for
>> > refactoring
>> > clients so that others who are not following this KIP also
>> notice the
>> > discussion. Satish, would you like to start a discussion thread
>> on dev?
>> >
>> > Regards,
>> >
>> > Rajini
>> >
>> >
>> > On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <
>> > satish.duggana@gmail.com>
>> > wrote:
>> >
>> > > I felt the same need when we want to add a pluggable API for
>> core
>> > > server functionality. This does not need to be part of this
>> KIP, it
>> > > can be a separate KIP. I can contribute those refactoring
>> changes if
>> > > others are OK with that.
>> > >
>> > > It is better to have a structure like below.
>> > >
>> > > kafka-common:
>> > > common classes which can be used in any of the other modules
>> in Kafka
>> > > like client, Kafka-server-common and server etc.
>> > >
>> > > kafka-client-common:
>> > > common classes which can be used in the client module. This
>> can be
>> > > part of client module itself.
>> > >
>> > > kafka-server-common:
>> > > classes required only for kafka-server.
>> > >
>> > > Thanks.
>> > > Satish.
>> > >
>> > > On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani <
>> kafka@harsha.io>
>> > > wrote:
>> > > >
>> > > > Thanks for the KIP Rajini.
>> > > > Quick thought, it would be good to have a common module
>> outside of
>> > > clients
>> > > > that only applies to server side interfaces & changes. It
>> looks
>> > like we
>> > > are
>> > > > increasingly in favor of using Java interface for pluggable
>> > modules on
>> > > the
>> > > > broker side.
>> > > >
>> > > > Thanks,
>> > > > Harsha
>> > > >
>> > > >
>> > > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <
>> > rajinisivaram@gmail.com
>> > > >
>> > > > wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > > I have created a KIP to replace the Scala Authorizer API
>> with a
>> > new
>> > > Java
>> > > > > API:
>> > > > >
>> > > > > -
>> > > > > https://cwiki.apache.org/confluence/display/KAFKA/
>> > > > > KIP-504+-+Add+new+Java+Authorizer+Interface
>> > > > >
>> > > > > This is replacement for KIP-50 which was accepted but
>> never
>> > merged.
>> > > Apart
>> > > > > from moving to a Java API consistent with other pluggable
>> > interfaces
>> > > in the
>> > > > > broker, KIP-504 also attempts to address known
>> limitations in the
>> > > > > authorizer. If you have come across other limitations
>> that you
>> > would
>> > > like
>> > > > > to see addressed in the new API, please raise these on the
>> > discussion
>> > > > > thread so that we can consider those too. All suggestions
>> and
>> > feedback
>> > > are
>> > > > > welcome.
>> > > > >
>> > > > > Thank you,
>> > > > >
>> > > > > Rajini
>> > > > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Don Bosco Durai <bo...@apache.org>.
Hi Rajini
Overall, it looks good.
Thanks
Bosco
On 9/4/19, 3:39 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
Hi Colin & Don,
I have submitted a PR to help with reviewing the new API:
https://github.com/apache/kafka/pull/7293/files.
- Authorizer.java contains the updated API and includes javadoc
update including the threading model.
- AclAuthorizer.scala changes in the PR should give an idea of the
implementation changes for custom authorizers that want to continue to be
synchronous.
- KafkaApis.scala continues to use the API synchronously for now. It can
be updated later.
Thank you,
Rajini
On Wed, Sep 4, 2019 at 9:38 AM Rajini Sivaram <ra...@gmail.com>
wrote:
> Hi Don,
>
> Thanks for your note.
>
> 1) The intention is to avoid blocking in the calling thread. We already
> have several requests that are put into a purgatory when waiting for remote
> communication, for example produce request waiting for replication. Since
> we have a limited number of request threads in the broker, we want to make
> progress with other requests, while one is waiting on any form of remote
> communication.
>
> 2) Async management calls will be useful with the default authorizer when
> KIP-500 removes ZK and we rely on Kafka instead. Our current ZK-based
> implementation as well as any custom implementations that don't want to be
> async will just need to return a sync'ed value. So instead of returning `
> value`, the code would just return `CompletableFuture.completedFuture
> (value)`. So it would be just a single line change in the implementation
> with the new API. The caller would treat completedFuture exactly as it
> does today, processing the request synchronously without using a purgatory.
>
> 3) For implementations that return a completedFuture as described in 2),
> the behaviour would remain exactly the same. No additional threads or
> purgatory will be used for this case. So there would be no performance
> penalty. For implementations that return a future that is not complete, we
> prioritise running more requests concurrently. So in a deployment with a
> large number of clients, we would improve performance by allowing other
> requests to be processed on the request threads while some are waiting for
> authorization metadata.
>
> 4) I was concerned about this too. The goal is to make the API flexible
> enough to handle large scale deployments in future when caching all
> authorization metadata in each broker is not viable. Using an async API
> that returns CompletionStage, the caller has the option to handle the
> result synchronously or asynchronously, so we don't necessarily need to
> update the calling code right away. Custom authorizers using the async API
> have full control over whether authorization is performed in-line since
> completedFuture will always be handled synchronously. We do need to
> update KafkaApis to take advantage of the asynchronous API to improve
> scale. Even though this is a big change, since we will be doing the same
> for all requests, it shouldn't be too hard to maintain since the same
> pattern will be used for all requests.
>
> Regards,
>
> Rajini
>
> On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <bo...@apache.org> wrote:
>
>> Hi Rajini
>>
>> Help me understand this a bit more.
>>
>> 1. For all practical purpose, without authorization you can't go to the
>> next step. The calling code needs to block anyway. So should we just let
>> the implementation code do the async part?
>> 2. If you feel management calls need to be async, then we should consider
>> another set of async APIs. I don't feel we should complicate it further (
>> 3. Another concern I have is wrt performance. Kafka has been built to
>> handle 1000s per second requests. Not sure whether making it async will add
>> any unnecessary overhead.
>> 4. How much complication would this add on the calling side? And is it
>> worth it?
>>
>> Thanks
>>
>> Bosco
>>
>>
>> On 9/3/19, 8:50 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
>>
>> Hi all,
>>
>> Ismael brought up a point that it will be good to make the Authorizer
>> interface asynchronous to avoid blocking request threads during remote
>> operations.
>>
>> 1) Since we want to support different backends for authorization
>> metadata,
>> making createAcls() and deleteAcls() asynchronous makes sense since
>> these
>> always involve remote operations. When KIP-500 removes ZooKeeper, we
>> would
>> want to move ACLs to Kafka and async updates will avoid unnecessary
>> blocking.
>> 2) For authorize() method, we currently use cached ACLs in the
>> built-in
>> authorizers, so synchronous authorise operations work well now. But
>> async
>> authorize() would support this scenario as well as authorizers in
>> large
>> organisations where an LRU cache would enable a smaller cache even
>> when the
>> backend holds a large amount of ACLs for infrequently used resources
>> or
>> users who don't use the system frequently.
>>
>> For both cases, the built-in authorizer will continue to be
>> synchronous,
>> using CompletableFuture.completedFuture() to return the actual
>> result. But
>> async API will make custom authorizer implementations more flexible. I
>> would like to know if there are any concerns with these changes before
>> updating the KIP.
>>
>> *Proposed API:*
>> public interface Authorizer extends Configurable, Closeable {
>>
>> Map<Endpoint, CompletionStage<Void>> start(AuthorizerServerInfo
>> serverInfo);
>> List<CompletionStage<AuthorizationResult>>
>> authorize(AuthorizableRequestContext requestContext, List<Action>
>> actions);
>> List<CompletionStage<AclCreateResult>>
>> createAcls(AuthorizableRequestContext requestContext, List<AclBinding>
>> aclBindings);
>> List<CompletionStage<AclDeleteResult>>
>> deleteAcls(AuthorizableRequestContext requestContext,
>> List<AclBindingFilter> aclBindingFilters);
>> CompletionStage<Collection<AclBinding>> acls(AclBindingFilter
>> filter);
>> }
>>
>>
>> Thank you,
>>
>> Rajini
>>
>> On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <bo...@apache.org>
>> wrote:
>>
>> > Hi Rajini
>> >
>> > Thanks for clarifying. I am good for now.
>> >
>> > Regards
>> >
>> > Bosco
>> >
>> >
>> > On 8/16/19, 11:30 AM, "Rajini Sivaram" <ra...@gmail.com>
>> wrote:
>> >
>> > Hi Don,
>> >
>> > That should be fine. I guess Ranger loads policies from the
>> database
>> > synchronously when the authorizer is configured, similar to
>> > SimpleAclAuthorizer loading from ZooKeeper. Ranger can continue
>> to load
>> > synchronously from `configure()` or `start()` and return an
>> empty map
>> > from
>> > `start()`. That would retain the existing behaviour.. When the
>> same
>> > database stores policies for all listeners and the policies are
>> not
>> > stored
>> > in Kafka, there is no value in making the load asynchronous.
>> >
>> > Regards,
>> >
>> > Rajini
>> >
>> >
>> > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai <
>> bosco@apache.org>
>> > wrote:
>> >
>> > > Hi Rajini
>> > >
>> > > Assuming this doesn't affect custom plugins, I don't have any
>> > concerns
>> > > regarding this.
>> > >
>> > > I do have one question regarding:
>> > >
>> > > "For authorizers that don’t store metadata in ZooKeeper,
>> ensure that
>> > > authorizer metadata for each listener is available before
>> starting
>> > up the
>> > > listener. This enables different authorization metadata
>> stores for
>> > > different listeners."
>> > >
>> > > Since Ranger uses its own database for storing policies, do
>> you
>> > anticipate
>> > > any issues?
>> > >
>> > > Thanks
>> > >
>> > > Bosco
>> > >
>> > >
>> > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
>> rajinisivaram@gmail.com>
>> > wrote:
>> > >
>> > > Hi all,
>> > >
>> > > I made another change to the KIP. The KIP was originally
>> > proposing to
>> > > extend SimpleAclAuthorizer to also implement the new API
>> (in
>> > addition
>> > > to
>> > > the existing API). But since we use the new API when
>> available,
>> > this
>> > > can
>> > > break custom authorizers that extend this class and
>> override
>> > specific
>> > > methods of the old API. To avoid breaking any existing
>> custom
>> > > implementations that extend this class, particularly
>> because it
>> > is in
>> > > the
>> > > public package kafka.security.auth, the KIP now proposes
>> to
>> > retain the
>> > > old
>> > > authorizer as-is. SimpleAclAuthorizer will continue to
>> > implement the
>> > > old
>> > > API, but will be deprecated. A new authorizer
>> implementation
>> > > kafka.security.authorizer.AclAuthorizer will be added for
>> the
>> > new API
>> > > (this
>> > > will not be in the public package).
>> > >
>> > > Please let me know if you have any concerns.
>> > >
>> > > Regards,
>> > >
>> > > Rajini
>> > >
>> > >
>> > > On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
>> > > rajinisivaram@gmail.com>
>> > > wrote:
>> > >
>> > > > Thanks Colin.
>> > > >
>> > > > If there are no other concerns, I will start vote later
>> today.
>> > Many
>> > > thanks
>> > > > to every one for the feedback.
>> > > >
>> > > > Regards,
>> > > >
>> > > > Rajini
>> > > >
>> > > >
>> > > > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <
>> > cmccabe@apache.org>
>> > > wrote:
>> > > >
>> > > >> Thanks, Rajini. It looks good to me.
>> > > >>
>> > > >> best,
>> > > >> Colin
>> > > >>
>> > > >>
>> > > >> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram wrote:
>> > > >> > Hi Colin,
>> > > >> >
>> > > >> > Thanks for the review. I have updated the KIP to
>> move the
>> > > interfaces for
>> > > >> > request context and server info to the authorizer
>> package.
>> > These
>> > > are now
>> > > >> > called AuthorizableRequestContext and
>> AuthorizerServerInfo.
>> > > Endpoint is
>> > > >> now
>> > > >> > a class in org.apache.kafka.common to make it
>> reusable
>> > since we
>> > > already
>> > > >> > have multiple implementations of it. I have removed
>> > requestName
>> > > from the
>> > > >> > request context interface since authorizers can
>> distinguish
>> > > follower
>> > > >> fetch
>> > > >> > and consumer fetch from the operation being
>> authorized. So
>> > 16-bit
>> > > >> request
>> > > >> > type should be sufficient for audit logging. Also
>> replaced
>> > > AuditFlag
>> > > >> with
>> > > >> > two booleans as you suggested.
>> > > >> >
>> > > >> > Can you take another look and see if the KIP is
>> ready for
>> > voting?
>> > > >> >
>> > > >> > Thanks for all your help!
>> > > >> >
>> > > >> > Regards,
>> > > >> >
>> > > >> > Rajini
>> > > >> >
>> > > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <
>> > cmccabe@apache.org>
>> > > >> wrote:
>> > > >> >
>> > > >> > > Hi Rajini,
>> > > >> > >
>> > > >> > > I think it would be good to rename
>> KafkaRequestContext to
>> > > something
>> > > >> like
>> > > >> > > AuthorizableRequestContext, and put it in the
>> > > >> > > org.apache.kafka.server.authorizer namespace. If
>> we put
>> > it in
>> > > the
>> > > >> > > org.apache.kafka.common namespace, then it's not
>> really
>> > clear
>> > > that
>> > > >> it's
>> > > >> > > part of the Authorizer API. Since this class is
>> purely an
>> > > interface,
>> > > >> with
>> > > >> > > no concrete implementation of anything, there's
>> nothing
>> > common
>> > > to
>> > > >> really
>> > > >> > > reuse in any case. We definitely don't want
>> someone to
>> > > accidentally
>> > > >> add or
>> > > >> > > remove methods because they think this is just
>> another
>> > internal
>> > > class
>> > > >> used
>> > > >> > > for requests.
>> > > >> > >
>> > > >> > > The BrokerInfo class is a nice improvement. It
>> looks
>> > like it
>> > > will be
>> > > >> > > useful for passing in information about the
>> context we're
>> > > running
>> > > >> in. It
>> > > >> > > would be nice to call this class ServerInfo rather
>> than
>> > > BrokerInfo,
>> > > >> since
>> > > >> > > we will want to run the authorizer on controllers
>> as well
>> > as on
>> > > >> brokers,
>> > > >> > > and the controller may run as a separate process
>> post
>> > KIP-500.
>> > > I also
>> > > >> > > think that this class should be in the
>> > > >> org.apache.kafka.server.authorizer
>> > > >> > > namespace. Again, it is an interface, not a
>> concrete
>> > > implementation,
>> > > >> and
>> > > >> > > it's an interface that is very specifically for the
>> > authorizer.
>> > > >> > >
>> > > >> > > I agree that we probably don't have enough
>> information
>> > > preserved for
>> > > >> > > requests currently to always know what entity made
>> them.
>> > So we
>> > > can
>> > > >> leave
>> > > >> > > that out for now (except in the special case of
>> Fetch).
>> > > Perhaps we
>> > > >> can add
>> > > >> > > this later if it's needed.
>> > > >> > >
>> > > >> > > I understand the intention behind AuthorizationMode
>> > (which is
>> > > now
>> > > >> called
>> > > >> > > AuditFlag in the latest revision). But it still
>> feels
>> > > complex. What
>> > > >> if we
>> > > >> > > just had two booleans in Action: logSuccesses and
>> > logFailures?
>> > > That
>> > > >> seems
>> > > >> > > to cover all the cases here. MANDATORY_AUTHORIZE
>> = true,
>> > true.
>> > > >> > > OPTIONAL_AUTHORIZE = true, false. FILTER = true,
>> false.
>> > > >> LIST_AUTHORIZED =
>> > > >> > > false, false. Would there be anything lost versus
>> having
>> > the
>> > > enum?
>> > > >> > >
>> > > >> > > best,
>> > > >> > > Colin
>> > > >> > >
>> > > >> > >
>> > > >> > > On Wed, Aug 14, 2019, at 06:29, Mickael Maison
>> wrote:
>> > > >> > > > Hi Rajini,
>> > > >> > > >
>> > > >> > > > Thanks for the KIP!
>> > > >> > > > I really like that authorize() will be able to
>> take a
>> > batch of
>> > > >> > > > requests, this will speed up many
>> implementations!
>> > > >> > > >
>> > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <
>> > > >> rajinisivaram@gmail.com>
>> > > >> > > wrote:
>> > > >> > > > >
>> > > >> > > > > Thanks David! I have fixed the typo.
>> > > >> > > > >
>> > > >> > > > > Also made a couple of changes to make the
>> context
>> > > interfaces more
>> > > >> > > generic.
>> > > >> > > > > KafkaRequestContext now returns the 16-bit API
>> key as
>> > Colin
>> > > >> suggested
>> > > >> > > as
>> > > >> > > > > well as the friendly name used in metrics
>> which are
>> > useful
>> > > in
>> > > >> audit
>> > > >> > > logs.
>> > > >> > > > > `Authorizer#start` is now provided a generic
>> > `BrokerInfo`
>> > > >> interface
>> > > >> > > that
>> > > >> > > > > gives cluster id, broker id and endpoint
>> information.
>> > The
>> > > generic
>> > > >> > > interface
>> > > >> > > > > can potentially be used in other broker
>> plugins in
>> > future
>> > > and
>> > > >> provides
>> > > >> > > > > dynamically generated configs like broker id
>> and ports
>> > > which are
>> > > >> > > currently
>> > > >> > > > > not available to plugins unless these configs
>> are
>> > statically
>> > > >> > > configured.
>> > > >> > > > > Please let me know if there are any concerns.
>> > > >> > > > >
>> > > >> > > > > Regards,
>> > > >> > > > >
>> > > >> > > > > Rajini
>> > > >> > > > >
>> > > >> > > > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot <
>> > > djacot@confluent.io>
>> > > >> > > wrote:
>> > > >> > > > >
>> > > >> > > > > > Hi Rajini,
>> > > >> > > > > >
>> > > >> > > > > > Thank you for the update! It looks good to
>> me.
>> > There is a
>> > > typo
>> > > >> in the
>> > > >> > > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` ->
>> > > >> `MANDATORY_AUTHORIZE`.
>> > > >> > > > > >
>> > > >> > > > > > Regards,
>> > > >> > > > > > David
>> > > >> > > > > >
>> > > >> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini
>> Sivaram <
>> > > >> > > rajinisivaram@gmail.com>
>> > > >> > > > > > wrote:
>> > > >> > > > > >
>> > > >> > > > > > > Hi David,
>> > > >> > > > > > >
>> > > >> > > > > > > Thanks for reviewing the KIP! Since
>> questions
>> > about
>> > > >> `authorization
>> > > >> > > mode`
>> > > >> > > > > > > and `count` have come up multiple times, I
>> have
>> > renamed
>> > > both.
>> > > >> > > > > > >
>> > > >> > > > > > > 1) Renamed `count` to
>> `resourceReferenceCount`.
>> > It is
>> > > the
>> > > >> number
>> > > >> > > of times
>> > > >> > > > > > > the resource being authorized is referenced
>> > within the
>> > > >> request.
>> > > >> > > > > > >
>> > > >> > > > > > > 2) Renamed `AuthorizationMode` to
>> `AuditFlag`. It
>> > is
>> > > provided
>> > > >> to
>> > > >> > > improve
>> > > >> > > > > > > audit logging in the authorizer. The enum
>> values
>> > have
>> > > javadoc
>> > > >> which
>> > > >> > > > > > > indicate how the authorization result is
>> used in
>> > each
>> > > of the
>> > > >> modes
>> > > >> > > to
>> > > >> > > > > > > enable authorizers to log audit messages
>> at the
>> > right
>> > > severity
>> > > >> > > level.
>> > > >> > > > > > >
>> > > >> > > > > > > Regards,
>> > > >> > > > > > >
>> > > >> > > > > > > Rajini
>> > > >> > > > > > >
>> > > >> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM David
>> Jacot <
>> > > >> djacot@confluent.io>
>> > > >> > > wrote:
>> > > >> > > > > > >
>> > > >> > > > > > > > Hi Rajini,
>> > > >> > > > > > > >
>> > > >> > > > > > > > Thank you for the KIP. Overall, it looks
>> good
>> > to me.
>> > > I have
>> > > >> few
>> > > >> > > > > > > > questions/suggestions:
>> > > >> > > > > > > >
>> > > >> > > > > > > > 1. It is hard to grasp what
>> `Action#count` is
>> > for. I
>> > > guess I
>> > > >> > > understand
>> > > >> > > > > > > > where you want to go with it but it took
>> me a
>> > while to
>> > > >> figure it
>> > > >> > > out.
>> > > >> > > > > > > > Perhaps, we could come up with a better
>> name
>> > than
>> > > `count`?
>> > > >> > > > > > > >
>> > > >> > > > > > > > 2. I had a hard time trying to
>> understand the
>> > > >> `AuthorizationMode`
>> > > >> > > > > > > concept,
>> > > >> > > > > > > > especially wrt. the OPTIONAL one. My
>> > understanding is
>> > > that
>> > > >> an
>> > > >> > > ACL is
>> > > >> > > > > > > either
>> > > >> > > > > > > > defined or not. Could you elaborate a
>> bit more
>> > on
>> > > that?
>> > > >> > > > > > > >
>> > > >> > > > > > > > Thanks,
>> > > >> > > > > > > > David
>> > > >> > > > > > > >
>> > > >> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don
>> Bosco Durai
>> > <
>> > > >> > > bosco@apache.org>
>> > > >> > > > > > > wrote:
>> > > >> > > > > > > >
>> > > >> > > > > > > > > Hi Rajini
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > 3.2 - This makes sense. Thanks for
>> clarifying.
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > Rest looks fine. Once the
>> implementations are
>> > done,
>> > > it
>> > > >> will be
>> > > >> > > more
>> > > >> > > > > > > clear
>> > > >> > > > > > > > > on the different values RequestType
>> and Mode.
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > Thanks
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > Bosco
>> > > >> > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <
>> > > >> rajinisivaram@gmail.com
>> > > >> > > >
>> > > >> > > > > > wrote:
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > Hi Don,
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > Thanks for the suggestions. A few
>> > responses
>> > > below:
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > 3.1 Can rename and improve docs if
>> we keep
>> > > this. Let's
>> > > >> > > finish the
>> > > >> > > > > > > > > discussion on Colin's suggestions
>> > regarding this
>> > > >> first.
>> > > >> > > > > > > > > 3.2 No, I was thinking of some
>> requests
>> > that
>> > > have an
>> > > >> old
>> > > >> > > way of
>> > > >> > > > > > > > > authorizing
>> > > >> > > > > > > > > and a new way where we have
>> retained the
>> > old
>> > > way for
>> > > >> > > backward
>> > > >> > > > > > > > > compatibility. One example is
>> > Cluster:Create
>> > > >> permission to
>> > > >> > > create
>> > > >> > > > > > > > > topics.
>> > > >> > > > > > > > > We have replaced this with
>> fine-grained
>> > topic
>> > > create
>> > > >> > > access using
>> > > >> > > > > > > > > Topic:Create
>> > > >> > > > > > > > > for topic patterns. But we still
>> check if
>> > user
>> > > has
>> > > >> > > Cluster:Create
>> > > >> > > > > > > > > first. If
>> > > >> > > > > > > > > Denied, the deny is ignored and we
>> check
>> > > >> Topic:Create. We
>> > > >> > > dont
>> > > >> > > > > > want
>> > > >> > > > > > > > to
>> > > >> > > > > > > > > log
>> > > >> > > > > > > > > DENY for Cluster:Create at INFO
>> level for
>> > this,
>> > > since
>> > > >> this
>> > > >> > > is
>> > > >> > > > > > not a
>> > > >> > > > > > > > > mandatory ACL for creating topics.
>> We
>> > will get
>> > > >> appropriate
>> > > >> > > logs
>> > > >> > > > > > > from
>> > > >> > > > > > > > > the
>> > > >> > > > > > > > > subsequent Topic:Create in this
>> case.
>> > > >> > > > > > > > > 3.3 They are not quite the same.
>> FILTER
>> > implies
>> > > that
>> > > >> user
>> > > >> > > > > > actually
>> > > >> > > > > > > > > requested access to and performed
>> some
>> > > operation on
>> > > >> the
>> > > >> > > filtered
>> > > >> > > > > > > > > resources.
>> > > >> > > > > > > > > LIST_AUTHORZED did not result in
>> any
>> > actual
>> > > access.
>> > > >> User
>> > > >> > > simply
>> > > >> > > > > > > > wanted
>> > > >> > > > > > > > > to
>> > > >> > > > > > > > > know what they are allowed to
>> access.
>> > > >> > > > > > > > > 3.4 Request types are Produce,
>> JoinGroup,
>> > > OffsetCommit
>> > > >> > > etc. So
>> > > >> > > > > > that
>> > > >> > > > > > > > is
>> > > >> > > > > > > > > different from authorization mode,
>> > operation
>> > > etc.
>> > > >> > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > On Thu, Aug 8, 2019 at 11:36 PM
>> Don Bosco
>> > Durai
>> > > <
>> > > >> > > > > > bosco@apache.org>
>> > > >> > > > > > > > > wrote:
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > > Hi Rajini
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > Thanks for clarifying. This is
>> very
>> > helpful...
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > #1 - On the Ranger side, we
>> should be
>> > able to
>> > > handle
>> > > >> > > multiple
>> > > >> > > > > > > > > requests at
>> > > >> > > > > > > > > > the same time. I was just not
>> sure how
>> > much
>> > > >> processing
>> > > >> > > overhead
>> > > >> > > > > > > > will
>> > > >> > > > > > > > > be
>> > > >> > > > > > > > > > there on the Broker side to
>> split and
>> > then
>> > > >> consolidate
>> > > >> > > the
>> > > >> > > > > > > results.
>> > > >> > > > > > > > > If it
>> > > >> > > > > > > > > > is negligible, then this is the
>> better
>> > way.
>> > > It will
>> > > >> > > make it
>> > > >> > > > > > > future
>> > > >> > > > > > > > > proof.
>> > > >> > > > > > > > > > #2 - I agree, having a single
>> "start"
>> > call
>> > > makes it
>> > > >> > > cleaner.
>> > > >> > > > > > The
>> > > >> > > > > > > > > > Authorization implementation
>> will only
>> > have
>> > > to do
>> > > >> the
>> > > >> > > > > > > > initialization
>> > > >> > > > > > > > > only
>> > > >> > > > > > > > > > once.
>> > > >> > > > > > > > > > #3.1 - Thanks for the
>> clarification. I
>> > think
>> > > it
>> > > >> makes
>> > > >> > > sense to
>> > > >> > > > > > > have
>> > > >> > > > > > > > > this.
>> > > >> > > > > > > > > > The term "Mode" might not be
>> explicit
>> > enough.
>> > > >> > > Essentially it
>> > > >> > > > > > > seems
>> > > >> > > > > > > > > you want
>> > > >> > > > > > > > > > the Authorizer to know the
>> > Intent/Purpose of
>> > > the
>> > > >> > > authorize call
>> > > >> > > > > > > and
>> > > >> > > > > > > > > let the
>> > > >> > > > > > > > > > Authorizer decide what to log as
>> Audit
>> > event.
>> > > >> Changing
>> > > >> > > the
>> > > >> > > > > > > > > class/field name
>> > > >> > > > > > > > > > or giving more documentation
>> will do.
>> > > >> > > > > > > > > > #3.2 - Regarding the option
>> "OPTIONAL".
>> > Are
>> > > you
>> > > >> thinking
>> > > >> > > from
>> > > >> > > > > > > > > chaining
>> > > >> > > > > > > > > > multiple Authorizer? If so, I
>> am not
>> > sure
>> > > whether
>> > > >> the
>> > > >> > > Broker
>> > > >> > > > > > > would
>> > > >> > > > > > > > > have
>> > > >> > > > > > > > > > enough information to make that
>> > decision. I
>> > > feel the
>> > > >> > > Authorizer
>> > > >> > > > > > > > will
>> > > >> > > > > > > > > be the
>> > > >> > > > > > > > > > one who would have that
>> knowledge. E.g.
>> > in
>> > > Ranger
>> > > >> we have
>> > > >> > > > > > > explicit
>> > > >> > > > > > > > > deny,
>> > > >> > > > > > > > > > which means no matter what, the
>> request
>> > > should be
>> > > >> denied
>> > > >> > > for
>> > > >> > > > > > the
>> > > >> > > > > > > > > user/group
>> > > >> > > > > > > > > > or condition. So if you are
>> thinking of
>> > > chaining
>> > > >> > > Authorizers,
>> > > >> > > > > > > then
>> > > >> > > > > > > > > > responses should have the third
>> state,
>> > e.g.
>> > > >> > > "DENIED_FINAL", in
>> > > >> > > > > > > > which
>> > > >> > > > > > > > > case
>> > > >> > > > > > > > > > if there is an Authorization
>> chain, it
>> > will
>> > > be stop
>> > > >> and
>> > > >> > > the
>> > > >> > > > > > > request
>> > > >> > > > > > > > > will be
>> > > >> > > > > > > > > > denied and if it is just denied,
>> then
>> > you
>> > > might fall
>> > > >> > > back to
>> > > >> > > > > > next
>> > > >> > > > > > > > > > authorizer. If we don't have
>> chaining of
>> > > >> Authorizing in
>> > > >> > > mind,
>> > > >> > > > > > > then
>> > > >> > > > > > > > we
>> > > >> > > > > > > > > > should reconsider OPTIONAL for
>> now. Or
>> > > clarify under
>> > > >> > > which
>> > > >> > > > > > > scenario
>> > > >> > > > > > > > > > OPTIONAL will be called.
>> > > >> > > > > > > > > > #3.3 Regarding, FILTER v/s
>> > LIST_AUTHORIZED,
>> > > isn't
>> > > >> > > > > > > LIST_AUTHORIZED a
>> > > >> > > > > > > > > > special case for "FILTER"?
>> > > >> > > > > > > > > > #3.4 KafkaRequestContext.
>> requestType()
>> > v/s
>> > > Action.
>> > > >> > > > > > > > > authorizationMode. I
>> > > >> > > > > > > > > > am not sure about the overlap or
>> > ambiguity.
>> > > >> > > > > > > > > > #4 - Cool. This is good, it will
>> be less
>> > > stress on
>> > > >> the
>> > > >> > > > > > > Authorizer.
>> > > >> > > > > > > > > Ranger
>> > > >> > > > > > > > > > already supports the "count"
>> concept
>> > and also
>> > > has
>> > > >> > > batching
>> > > >> > > > > > > > > capability to
>> > > >> > > > > > > > > > aggregate similar requests to
>> reduce the
>> > > number of
>> > > >> audit
>> > > >> > > logs
>> > > >> > > > > > to
>> > > >> > > > > > > > > write. We
>> > > >> > > > > > > > > > should be able to pass this
>> through.
>> > > >> > > > > > > > > > #5 - Assuming if the object
>> instance is
>> > going
>> > > out of
>> > > >> > > scope, we
>> > > >> > > > > > > > > should be
>> > > >> > > > > > > > > > fine. Not a super important ask.
>> Ranger
>> > is
>> > > already
>> > > >> > > catching
>> > > >> > > > > > KILL
>> > > >> > > > > > > > > signal for
>> > > >> > > > > > > > > > clean up.
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > Thanks again. These are good
>> > enhancements.
>> > > >> Appreciate
>> > > >> > > your
>> > > >> > > > > > > efforts
>> > > >> > > > > > > > > here.
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > Bosco
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > On 8/8/19, 2:03 AM, "Rajini
>> Sivaram" <
>> > > >> > > rajinisivaram@gmail.com
>> > > >> > > > > > >
>> > > >> > > > > > > > > wrote:
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > Hi Don,
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > Thanks for reviewing the KIP.
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > 1. I had this originally as
>> a single
>> > > Action, but
>> > > >> > > thought it
>> > > >> > > > > > > may
>> > > >> > > > > > > > > be
>> > > >> > > > > > > > > > useful
>> > > >> > > > > > > > > > to support batched authorize
>> calls
>> > as
>> > > well and
>> > > >> keep
>> > > >> > > it
>> > > >> > > > > > > > > consistent with
>> > > >> > > > > > > > > > other methods. Single
>> requests can
>> > contain
>> > > >> multiple
>> > > >> > > topics.
>> > > >> > > > > > > For
>> > > >> > > > > > > > > > example a
>> > > >> > > > > > > > > > produce request can contain
>> records
>> > for
>> > > several
>> > > >> > > partitions
>> > > >> > > > > > of
>> > > >> > > > > > > > > different
>> > > >> > > > > > > > > > topics. Broker could
>> potentially
>> > > authorize these
>> > > >> > > together.
>> > > >> > > > > > > For
>> > > >> > > > > > > > > > SimpleAclAuthorizer, batched
>> > authorize
>> > > methods
>> > > >> don't
>> > > >> > > > > > provide
>> > > >> > > > > > > > any
>> > > >> > > > > > > > > > optimisation since lookup is
>> based
>> > on
>> > > resources
>> > > >> > > followed by
>> > > >> > > > > > > the
>> > > >> > > > > > > > > > matching
>> > > >> > > > > > > > > > logic. But some authorizers
>> may
>> > manage
>> > > ACLs by
>> > > >> user
>> > > >> > > > > > principal
>> > > >> > > > > > > > > rather
>> > > >> > > > > > > > > > than
>> > > >> > > > > > > > > > resource and may be able to
>> optimize
>> > > batched
>> > > >> > > requests. I am
>> > > >> > > > > > > ok
>> > > >> > > > > > > > > with
>> > > >> > > > > > > > > > using
>> > > >> > > > > > > > > > single Action if this is
>> likely to
>> > cause
>> > > issues.
>> > > >> > > > > > > > > > 2. If you have two
>> listeners, one
>> > for
>> > > >> inter-broker
>> > > >> > > traffic
>> > > >> > > > > > > and
>> > > >> > > > > > > > > another
>> > > >> > > > > > > > > > for
>> > > >> > > > > > > > > > external clients, start
>> method is
>> > invoked
>> > > twice,
>> > > >> > > once for
>> > > >> > > > > > > each
>> > > >> > > > > > > > > > listener. On
>> > > >> > > > > > > > > > second thought, that may be
>> > confusing and
>> > > a
>> > > >> single
>> > > >> > > start()
>> > > >> > > > > > > > > invocation
>> > > >> > > > > > > > > > that
>> > > >> > > > > > > > > > provides all listener
>> information
>> > and
>> > > returns
>> > > >> > > multiple
>> > > >> > > > > > > futures
>> > > >> > > > > > > > > would be
>> > > >> > > > > > > > > > better. Will update the KIP.
>> > > >> > > > > > > > > > 3. A typical example is a
>> consumer
>> > > subscribing
>> > > >> to a
>> > > >> > > regex
>> > > >> > > > > > > > > pattern. We
>> > > >> > > > > > > > > > request all topic metadata
>> from the
>> > > broker in
>> > > >> order
>> > > >> > > to
>> > > >> > > > > > decide
>> > > >> > > > > > > > > whether
>> > > >> > > > > > > > > > the
>> > > >> > > > > > > > > > pattern matches, expecting to
>> > receive a
>> > > list of
>> > > >> > > authorised
>> > > >> > > > > > > > > topics. The
>> > > >> > > > > > > > > > user
>> > > >> > > > > > > > > > is not asking to subscribe
>> to an
>> > > unauthorized
>> > > >> topic.
>> > > >> > > If
>> > > >> > > > > > there
>> > > >> > > > > > > > > are 10000
>> > > >> > > > > > > > > > topics in the cluster and
>> the user
>> > has
>> > > access
>> > > >> to 100
>> > > >> > > of
>> > > >> > > > > > them,
>> > > >> > > > > > > > at
>> > > >> > > > > > > > > the
>> > > >> > > > > > > > > > moment
>> > > >> > > > > > > > > > we log 9900 DENIED log
>> entries at
>> > INFO
>> > > level in
>> > > >> > > > > > > > > SimpleAclAuthorizer.
>> > > >> > > > > > > > > > The
>> > > >> > > > > > > > > > proposal is to authorize this
>> > request with
>> > > >> > > > > > > > > AuthorizationMode.FILTER, so
>> > > >> > > > > > > > > > that authorizers can log
>> resources
>> > that
>> > > are
>> > > >> filtered
>> > > >> > > out at
>> > > >> > > > > > > > > lower level
>> > > >> > > > > > > > > > like DEBUG since this is not
>> an
>> > attempt to
>> > > >> access
>> > > >> > > > > > > unauthorized
>> > > >> > > > > > > > > > resources.
>> > > >> > > > > > > > > > Brokers already handle these
>> > differently
>> > > since
>> > > >> no
>> > > >> > > > > > > authorization
>> > > >> > > > > > > > > error
>> > > >> > > > > > > > > > is
>> > > >> > > > > > > > > > returned to the client in
>> these
>> > cases.
>> > > Providing
>> > > >> > > > > > > authorization
>> > > >> > > > > > > > > mode to
>> > > >> > > > > > > > > > authorizers enables
>> authorizer
>> > > implementations
>> > > >> to
>> > > >> > > generate
>> > > >> > > > > > > > > better audit
>> > > >> > > > > > > > > > logs.
>> > > >> > > > > > > > > > 4. Each request may contain
>> multiple
>> > > instances
>> > > >> of
>> > > >> > > the same
>> > > >> > > > > > > > > authorizable
>> > > >> > > > > > > > > > resource. For example a
>> produce
>> > request
>> > > may
>> > > >> contain
>> > > >> > > records
>> > > >> > > > > > > for
>> > > >> > > > > > > > > 10
>> > > >> > > > > > > > > > partitions of the same
>> topic. At the
>> > > moment, we
>> > > >> > > invoke
>> > > >> > > > > > > > authorize
>> > > >> > > > > > > > > > method 10
>> > > >> > > > > > > > > > times. The proposal is to
>> invoke it
>> > once
>> > > with
>> > > >> > > count=10. The
>> > > >> > > > > > > > > count is
>> > > >> > > > > > > > > > provided to authorizer just
>> for
>> > audit
>> > > logging
>> > > >> > > purposes.
>> > > >> > > > > > > > > > 5. Authorizer implements
>> Closeable,
>> > so you
>> > > >> could use
>> > > >> > > > > > close()
>> > > >> > > > > > > to
>> > > >> > > > > > > > > flush
>> > > >> > > > > > > > > > audits?
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > On Thu, Aug 8, 2019 at 7:01
>> AM Don
>> > Bosco
>> > > Durai <
>> > > >> > > > > > > > bosco@apache.org
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > wrote:
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > > Rajini
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Thanks for putting this
>> together.
>> > It is
>> > > >> looking
>> > > >> > > good. I
>> > > >> > > > > > > have
>> > > >> > > > > > > > > few
>> > > >> > > > > > > > > > > questions...
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > 1.
>> List<AuthorizationResult>
>> > > authorize(...,
>> > > >> > > List<Action>
>> > > >> > > > > > > > > actions).
>> > > >> > > > > > > > > > Do you
>> > > >> > > > > > > > > > > see a scenario where the
>> broker
>> > will
>> > > call
>> > > >> > > authorize for
>> > > >> > > > > > > > > multiple
>> > > >> > > > > > > > > > topics at
>> > > >> > > > > > > > > > > the same time? I can
>> understand
>> > that
>> > > during
>> > > >> > > > > > > creating/deleting
>> > > >> > > > > > > > > ACLS,
>> > > >> > > > > > > > > > > multiple permissions for
>> multiple
>> > > resources
>> > > >> might
>> > > >> > > be
>> > > >> > > > > > done.
>> > > >> > > > > > > > For
>> > > >> > > > > > > > > > authorize
>> > > >> > > > > > > > > > > call, would this be a
>> case? And
>> > does the
>> > > >> Authorize
>> > > >> > > > > > > > > implementation
>> > > >> > > > > > > > > > will be
>> > > >> > > > > > > > > > > able to do performance
>> > optimization
>> > > because of
>> > > >> > > this? Or
>> > > >> > > > > > > > should
>> > > >> > > > > > > > > we
>> > > >> > > > > > > > > > just keep
>> > > >> > > > > > > > > > > it simple? I don't see it
>> as an
>> > issue
>> > > from
>> > > >> Apache
>> > > >> > > Ranger
>> > > >> > > > > > > > side,
>> > > >> > > > > > > > > but
>> > > >> > > > > > > > > > just
>> > > >> > > > > > > > > > > checking to see whether we
>> need
>> > to be
>> > > aware of
>> > > >> > > something.
>> > > >> > > > > > > > > > > 2. Should I assume that the
>> > > SecurityProtocol
>> > > >> passed
>> > > >> > > > > > during
>> > > >> > > > > > > > > start and
>> > > >> > > > > > > > > > the
>> > > >> > > > > > > > > > > one return by
>> > > >> > > KafkaRequestContext.securityProtocol() will
>> > > >> > > > > > > be
>> > > >> > > > > > > > > the
>> > > >> > > > > > > > > > same?
>> > > >> > > > > > > > > > > CompletableFuture<Void>
>> > start(String
>> > > >> listenerName,
>> > > >> > > > > > > > > SecurityProtocol
>> > > >> > > > > > > > > > > securityProtocol);
>> > > >> > > > > > > > > > >
>> > KafkaRequestContext.securityProtocol()
>> > > >> > > > > > > > > > > 3. What is the purpose of
>> > > AuthorizationMode?
>> > > >> How
>> > > >> > > does the
>> > > >> > > > > > > > > broker
>> > > >> > > > > > > > > > decide
>> > > >> > > > > > > > > > > what mode to use when the
>> > authorize()
>> > > method
>> > > >> is
>> > > >> > > called?
>> > > >> > > > > > > > > > > 4. Can we clarify "count"
>> in
>> > Action a
>> > > bit
>> > > >> more?
>> > > >> > > How is it
>> > > >> > > > > > > > used?
>> > > >> > > > > > > > > > > 5. Do you feel having
>> "stop"
>> > along with
>> > > >> "start" be
>> > > >> > > > > > helpful?
>> > > >> > > > > > > > > E.g. In
>> > > >> > > > > > > > > > Ranger
>> > > >> > > > > > > > > > > we try to optimize the
>> Audit
>> > writing by
>> > > >> caching
>> > > >> > > the logs
>> > > >> > > > > > > for
>> > > >> > > > > > > > a
>> > > >> > > > > > > > > fixed
>> > > >> > > > > > > > > > > interval. But when the
>> Broker
>> > > terminates, we
>> > > >> do a
>> > > >> > > forced
>> > > >> > > > > > > > flush.
>> > > >> > > > > > > > > > Having an
>> > > >> > > > > > > > > > > explicit "stop" might give
>> us a
>> > formal
>> > > way to
>> > > >> > > flush our
>> > > >> > > > > > > > audits.
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Thanks
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Bosco
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > On 8/7/19, 3:59 PM,
>> "Rajini
>> > Sivaram" <
>> > > >> > > > > > > > rajinisivaram@gmail.com
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > wrote:
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Hi Ron/Harsha/Satish,
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Thanks for reviewing
>> the KIP!
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > We should perhaps have
>> a wider
>> > > discussion
>> > > >> > > outside
>> > > >> > > > > > this
>> > > >> > > > > > > > KIP
>> > > >> > > > > > > > > for
>> > > >> > > > > > > > > > > refactoring
>> > > >> > > > > > > > > > > clients so that others
>> who
>> > are not
>> > > >> following
>> > > >> > > this KIP
>> > > >> > > > > > > > also
>> > > >> > > > > > > > > > notice the
>> > > >> > > > > > > > > > > discussion. Satish,
>> would you
>> > like
>> > > to
>> > > >> start a
>> > > >> > > > > > > discussion
>> > > >> > > > > > > > > thread
>> > > >> > > > > > > > > > on dev?
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Regards,
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Rajini
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > On Wed, Aug 7, 2019 at
>> 6:21 PM
>> > > Satish
>> > > >> Duggana <
>> > > >> > > > > > > > > > > satish.duggana@gmail.com>
>> > > >> > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > > I felt the same need
>> when
>> > we want
>> > > to
>> > > >> add a
>> > > >> > > > > > pluggable
>> > > >> > > > > > > > API
>> > > >> > > > > > > > > for
>> > > >> > > > > > > > > > core
>> > > >> > > > > > > > > > > > server
>> functionality. This
>> > does
>> > > not
>> > > >> need to
>> > > >> > > be part
>> > > >> > > > > > > of
>> > > >> > > > > > > > > this
>> > > >> > > > > > > > > > KIP, it
>> > > >> > > > > > > > > > > > can be a separate
>> KIP. I can
>> > > contribute
>> > > >> those
>> > > >> > > > > > > > refactoring
>> > > >> > > > > > > > > > changes if
>> > > >> > > > > > > > > > > > others are OK with
>> that.
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > It is better to have
>> a
>> > structure
>> > > like
>> > > >> below.
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > kafka-common:
>> > > >> > > > > > > > > > > > common classes which
>> can be
>> > used
>> > > in any
>> > > >> of
>> > > >> > > the
>> > > >> > > > > > other
>> > > >> > > > > > > > > modules
>> > > >> > > > > > > > > > in Kafka
>> > > >> > > > > > > > > > > > like client,
>> > Kafka-server-common
>> > > and
>> > > >> server
>> > > >> > > etc.
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > kafka-client-common:
>> > > >> > > > > > > > > > > > common classes which
>> can be
>> > used
>> > > in the
>> > > >> > > client
>> > > >> > > > > > > module.
>> > > >> > > > > > > > > This
>> > > >> > > > > > > > > > can be
>> > > >> > > > > > > > > > > > part of client module
>> > itself.
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > kafka-server-common:
>> > > >> > > > > > > > > > > > classes required
>> only for
>> > > kafka-server.
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > Thanks.
>> > > >> > > > > > > > > > > > Satish.
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > On Wed, Aug 7, 2019
>> at 9:28
>> > PM
>> > > Harsha
>> > > >> > > Chintalapani
>> > > >> > > > > > <
>> > > >> > > > > > > > > > kafka@harsha.io>
>> > > >> > > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > Thanks for the KIP
>> Rajini.
>> > > >> > > > > > > > > > > > > Quick thought, it
>> would
>> > be good
>> > > to
>> > > >> have a
>> > > >> > > common
>> > > >> > > > > > > > module
>> > > >> > > > > > > > > > outside of
>> > > >> > > > > > > > > > > > clients
>> > > >> > > > > > > > > > > > > that only applies
>> to
>> > server side
>> > > >> > > interfaces &
>> > > >> > > > > > > > changes.
>> > > >> > > > > > > > > It
>> > > >> > > > > > > > > > looks
>> > > >> > > > > > > > > > > like we
>> > > >> > > > > > > > > > > > are
>> > > >> > > > > > > > > > > > > increasingly in
>> favor of
>> > using
>> > > Java
>> > > >> > > interface for
>> > > >> > > > > > > > > pluggable
>> > > >> > > > > > > > > > > modules on
>> > > >> > > > > > > > > > > > the
>> > > >> > > > > > > > > > > > > broker side.
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > Thanks,
>> > > >> > > > > > > > > > > > > Harsha
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > On Tue, Aug 06,
>> 2019 at
>> > 2:31 PM,
>> > > >> Rajini
>> > > >> > > Sivaram <
>> > > >> > > > > > > > > > > rajinisivaram@gmail.com
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > Hi all,
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > I have created a
>> KIP to
>> > > replace the
>> > > >> Scala
>> > > >> > > > > > > > Authorizer
>> > > >> > > > > > > > > API
>> > > >> > > > > > > > > > with a
>> > > >> > > > > > > > > > > new
>> > > >> > > > > > > > > > > > Java
>> > > >> > > > > > > > > > > > > > API:
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > -
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > >
>> > https://cwiki.apache.org/confluence/display/KAFKA/
>> > > >> > > > > > > > > > > > > >
>> > > >> > > KIP-504+-+Add+new+Java+Authorizer+Interface
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > This is
>> replacement for
>> > KIP-50
>> > > >> which was
>> > > >> > > > > > accepted
>> > > >> > > > > > > > > but never
>> > > >> > > > > > > > > > > merged.
>> > > >> > > > > > > > > > > > Apart
>> > > >> > > > > > > > > > > > > > from moving to a
>> Java
>> > API
>> > > consistent
>> > > >> > > with other
>> > > >> > > > > > > > > pluggable
>> > > >> > > > > > > > > > > interfaces
>> > > >> > > > > > > > > > > > in the
>> > > >> > > > > > > > > > > > > > broker, KIP-504
>> also
>> > attempts
>> > > to
>> > > >> address
>> > > >> > > known
>> > > >> > > > > > > > > limitations
>> > > >> > > > > > > > > > in the
>> > > >> > > > > > > > > > > > > > authorizer. If
>> you have
>> > come
>> > > across
>> > > >> other
>> > > >> > > > > > > > > limitations that
>> > > >> > > > > > > > > > you
>> > > >> > > > > > > > > > > would
>> > > >> > > > > > > > > > > > like
>> > > >> > > > > > > > > > > > > > to see addressed
>> in the
>> > new
>> > > API,
>> > > >> please
>> > > >> > > raise
>> > > >> > > > > > > these
>> > > >> > > > > > > > > on the
>> > > >> > > > > > > > > > > discussion
>> > > >> > > > > > > > > > > > > > thread so that
>> we can
>> > > consider those
>> > > >> > > too. All
>> > > >> > > > > > > > > suggestions
>> > > >> > > > > > > > > > and
>> > > >> > > > > > > > > > > feedback
>> > > >> > > > > > > > > > > > are
>> > > >> > > > > > > > > > > > > > welcome.
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > Thank you,
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > Rajini
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > >
>> > > >> > > > > > >
>> > > >> > > > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Colin & Don,
I have submitted a PR to help with reviewing the new API:
https://github.com/apache/kafka/pull/7293/files.
- Authorizer.java contains the updated API and includes javadoc
update including the threading model.
- AclAuthorizer.scala changes in the PR should give an idea of the
implementation changes for custom authorizers that want to continue to be
synchronous.
- KafkaApis.scala continues to use the API synchronously for now. It can
be updated later.
Thank you,
Rajini
On Wed, Sep 4, 2019 at 9:38 AM Rajini Sivaram <ra...@gmail.com>
wrote:
> Hi Don,
>
> Thanks for your note.
>
> 1) The intention is to avoid blocking in the calling thread. We already
> have several requests that are put into a purgatory when waiting for remote
> communication, for example produce request waiting for replication. Since
> we have a limited number of request threads in the broker, we want to make
> progress with other requests, while one is waiting on any form of remote
> communication.
>
> 2) Async management calls will be useful with the default authorizer when
> KIP-500 removes ZK and we rely on Kafka instead. Our current ZK-based
> implementation as well as any custom implementations that don't want to be
> async will just need to return a sync'ed value. So instead of returning `
> value`, the code would just return `CompletableFuture.completedFuture
> (value)`. So it would be just a single line change in the implementation
> with the new API. The caller would treat completedFuture exactly as it
> does today, processing the request synchronously without using a purgatory.
>
> 3) For implementations that return a completedFuture as described in 2),
> the behaviour would remain exactly the same. No additional threads or
> purgatory will be used for this case. So there would be no performance
> penalty. For implementations that return a future that is not complete, we
> prioritise running more requests concurrently. So in a deployment with a
> large number of clients, we would improve performance by allowing other
> requests to be processed on the request threads while some are waiting for
> authorization metadata.
>
> 4) I was concerned about this too. The goal is to make the API flexible
> enough to handle large scale deployments in future when caching all
> authorization metadata in each broker is not viable. Using an async API
> that returns CompletionStage, the caller has the option to handle the
> result synchronously or asynchronously, so we don't necessarily need to
> update the calling code right away. Custom authorizers using the async API
> have full control over whether authorization is performed in-line since
> completedFuture will always be handled synchronously. We do need to
> update KafkaApis to take advantage of the asynchronous API to improve
> scale. Even though this is a big change, since we will be doing the same
> for all requests, it shouldn't be too hard to maintain since the same
> pattern will be used for all requests.
>
> Regards,
>
> Rajini
>
> On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <bo...@apache.org> wrote:
>
>> Hi Rajini
>>
>> Help me understand this a bit more.
>>
>> 1. For all practical purpose, without authorization you can't go to the
>> next step. The calling code needs to block anyway. So should we just let
>> the implementation code do the async part?
>> 2. If you feel management calls need to be async, then we should consider
>> another set of async APIs. I don't feel we should complicate it further (
>> 3. Another concern I have is wrt performance. Kafka has been built to
>> handle 1000s per second requests. Not sure whether making it async will add
>> any unnecessary overhead.
>> 4. How much complication would this add on the calling side? And is it
>> worth it?
>>
>> Thanks
>>
>> Bosco
>>
>>
>> On 9/3/19, 8:50 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
>>
>> Hi all,
>>
>> Ismael brought up a point that it will be good to make the Authorizer
>> interface asynchronous to avoid blocking request threads during remote
>> operations.
>>
>> 1) Since we want to support different backends for authorization
>> metadata,
>> making createAcls() and deleteAcls() asynchronous makes sense since
>> these
>> always involve remote operations. When KIP-500 removes ZooKeeper, we
>> would
>> want to move ACLs to Kafka and async updates will avoid unnecessary
>> blocking.
>> 2) For authorize() method, we currently use cached ACLs in the
>> built-in
>> authorizers, so synchronous authorise operations work well now. But
>> async
>> authorize() would support this scenario as well as authorizers in
>> large
>> organisations where an LRU cache would enable a smaller cache even
>> when the
>> backend holds a large amount of ACLs for infrequently used resources
>> or
>> users who don't use the system frequently.
>>
>> For both cases, the built-in authorizer will continue to be
>> synchronous,
>> using CompletableFuture.completedFuture() to return the actual
>> result. But
>> async API will make custom authorizer implementations more flexible. I
>> would like to know if there are any concerns with these changes before
>> updating the KIP.
>>
>> *Proposed API:*
>> public interface Authorizer extends Configurable, Closeable {
>>
>> Map<Endpoint, CompletionStage<Void>> start(AuthorizerServerInfo
>> serverInfo);
>> List<CompletionStage<AuthorizationResult>>
>> authorize(AuthorizableRequestContext requestContext, List<Action>
>> actions);
>> List<CompletionStage<AclCreateResult>>
>> createAcls(AuthorizableRequestContext requestContext, List<AclBinding>
>> aclBindings);
>> List<CompletionStage<AclDeleteResult>>
>> deleteAcls(AuthorizableRequestContext requestContext,
>> List<AclBindingFilter> aclBindingFilters);
>> CompletionStage<Collection<AclBinding>> acls(AclBindingFilter
>> filter);
>> }
>>
>>
>> Thank you,
>>
>> Rajini
>>
>> On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <bo...@apache.org>
>> wrote:
>>
>> > Hi Rajini
>> >
>> > Thanks for clarifying. I am good for now.
>> >
>> > Regards
>> >
>> > Bosco
>> >
>> >
>> > On 8/16/19, 11:30 AM, "Rajini Sivaram" <ra...@gmail.com>
>> wrote:
>> >
>> > Hi Don,
>> >
>> > That should be fine. I guess Ranger loads policies from the
>> database
>> > synchronously when the authorizer is configured, similar to
>> > SimpleAclAuthorizer loading from ZooKeeper. Ranger can continue
>> to load
>> > synchronously from `configure()` or `start()` and return an
>> empty map
>> > from
>> > `start()`. That would retain the existing behaviour.. When the
>> same
>> > database stores policies for all listeners and the policies are
>> not
>> > stored
>> > in Kafka, there is no value in making the load asynchronous.
>> >
>> > Regards,
>> >
>> > Rajini
>> >
>> >
>> > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai <
>> bosco@apache.org>
>> > wrote:
>> >
>> > > Hi Rajini
>> > >
>> > > Assuming this doesn't affect custom plugins, I don't have any
>> > concerns
>> > > regarding this.
>> > >
>> > > I do have one question regarding:
>> > >
>> > > "For authorizers that don’t store metadata in ZooKeeper,
>> ensure that
>> > > authorizer metadata for each listener is available before
>> starting
>> > up the
>> > > listener. This enables different authorization metadata
>> stores for
>> > > different listeners."
>> > >
>> > > Since Ranger uses its own database for storing policies, do
>> you
>> > anticipate
>> > > any issues?
>> > >
>> > > Thanks
>> > >
>> > > Bosco
>> > >
>> > >
>> > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
>> rajinisivaram@gmail.com>
>> > wrote:
>> > >
>> > > Hi all,
>> > >
>> > > I made another change to the KIP. The KIP was originally
>> > proposing to
>> > > extend SimpleAclAuthorizer to also implement the new API
>> (in
>> > addition
>> > > to
>> > > the existing API). But since we use the new API when
>> available,
>> > this
>> > > can
>> > > break custom authorizers that extend this class and
>> override
>> > specific
>> > > methods of the old API. To avoid breaking any existing
>> custom
>> > > implementations that extend this class, particularly
>> because it
>> > is in
>> > > the
>> > > public package kafka.security.auth, the KIP now proposes
>> to
>> > retain the
>> > > old
>> > > authorizer as-is. SimpleAclAuthorizer will continue to
>> > implement the
>> > > old
>> > > API, but will be deprecated. A new authorizer
>> implementation
>> > > kafka.security.authorizer.AclAuthorizer will be added for
>> the
>> > new API
>> > > (this
>> > > will not be in the public package).
>> > >
>> > > Please let me know if you have any concerns.
>> > >
>> > > Regards,
>> > >
>> > > Rajini
>> > >
>> > >
>> > > On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
>> > > rajinisivaram@gmail.com>
>> > > wrote:
>> > >
>> > > > Thanks Colin.
>> > > >
>> > > > If there are no other concerns, I will start vote later
>> today.
>> > Many
>> > > thanks
>> > > > to every one for the feedback.
>> > > >
>> > > > Regards,
>> > > >
>> > > > Rajini
>> > > >
>> > > >
>> > > > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <
>> > cmccabe@apache.org>
>> > > wrote:
>> > > >
>> > > >> Thanks, Rajini. It looks good to me.
>> > > >>
>> > > >> best,
>> > > >> Colin
>> > > >>
>> > > >>
>> > > >> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram wrote:
>> > > >> > Hi Colin,
>> > > >> >
>> > > >> > Thanks for the review. I have updated the KIP to
>> move the
>> > > interfaces for
>> > > >> > request context and server info to the authorizer
>> package.
>> > These
>> > > are now
>> > > >> > called AuthorizableRequestContext and
>> AuthorizerServerInfo.
>> > > Endpoint is
>> > > >> now
>> > > >> > a class in org.apache.kafka.common to make it
>> reusable
>> > since we
>> > > already
>> > > >> > have multiple implementations of it. I have removed
>> > requestName
>> > > from the
>> > > >> > request context interface since authorizers can
>> distinguish
>> > > follower
>> > > >> fetch
>> > > >> > and consumer fetch from the operation being
>> authorized. So
>> > 16-bit
>> > > >> request
>> > > >> > type should be sufficient for audit logging. Also
>> replaced
>> > > AuditFlag
>> > > >> with
>> > > >> > two booleans as you suggested.
>> > > >> >
>> > > >> > Can you take another look and see if the KIP is
>> ready for
>> > voting?
>> > > >> >
>> > > >> > Thanks for all your help!
>> > > >> >
>> > > >> > Regards,
>> > > >> >
>> > > >> > Rajini
>> > > >> >
>> > > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <
>> > cmccabe@apache.org>
>> > > >> wrote:
>> > > >> >
>> > > >> > > Hi Rajini,
>> > > >> > >
>> > > >> > > I think it would be good to rename
>> KafkaRequestContext to
>> > > something
>> > > >> like
>> > > >> > > AuthorizableRequestContext, and put it in the
>> > > >> > > org.apache.kafka.server.authorizer namespace. If
>> we put
>> > it in
>> > > the
>> > > >> > > org.apache.kafka.common namespace, then it's not
>> really
>> > clear
>> > > that
>> > > >> it's
>> > > >> > > part of the Authorizer API. Since this class is
>> purely an
>> > > interface,
>> > > >> with
>> > > >> > > no concrete implementation of anything, there's
>> nothing
>> > common
>> > > to
>> > > >> really
>> > > >> > > reuse in any case. We definitely don't want
>> someone to
>> > > accidentally
>> > > >> add or
>> > > >> > > remove methods because they think this is just
>> another
>> > internal
>> > > class
>> > > >> used
>> > > >> > > for requests.
>> > > >> > >
>> > > >> > > The BrokerInfo class is a nice improvement. It
>> looks
>> > like it
>> > > will be
>> > > >> > > useful for passing in information about the
>> context we're
>> > > running
>> > > >> in. It
>> > > >> > > would be nice to call this class ServerInfo rather
>> than
>> > > BrokerInfo,
>> > > >> since
>> > > >> > > we will want to run the authorizer on controllers
>> as well
>> > as on
>> > > >> brokers,
>> > > >> > > and the controller may run as a separate process
>> post
>> > KIP-500.
>> > > I also
>> > > >> > > think that this class should be in the
>> > > >> org.apache.kafka.server.authorizer
>> > > >> > > namespace. Again, it is an interface, not a
>> concrete
>> > > implementation,
>> > > >> and
>> > > >> > > it's an interface that is very specifically for the
>> > authorizer.
>> > > >> > >
>> > > >> > > I agree that we probably don't have enough
>> information
>> > > preserved for
>> > > >> > > requests currently to always know what entity made
>> them.
>> > So we
>> > > can
>> > > >> leave
>> > > >> > > that out for now (except in the special case of
>> Fetch).
>> > > Perhaps we
>> > > >> can add
>> > > >> > > this later if it's needed.
>> > > >> > >
>> > > >> > > I understand the intention behind AuthorizationMode
>> > (which is
>> > > now
>> > > >> called
>> > > >> > > AuditFlag in the latest revision). But it still
>> feels
>> > > complex. What
>> > > >> if we
>> > > >> > > just had two booleans in Action: logSuccesses and
>> > logFailures?
>> > > That
>> > > >> seems
>> > > >> > > to cover all the cases here. MANDATORY_AUTHORIZE
>> = true,
>> > true.
>> > > >> > > OPTIONAL_AUTHORIZE = true, false. FILTER = true,
>> false.
>> > > >> LIST_AUTHORIZED =
>> > > >> > > false, false. Would there be anything lost versus
>> having
>> > the
>> > > enum?
>> > > >> > >
>> > > >> > > best,
>> > > >> > > Colin
>> > > >> > >
>> > > >> > >
>> > > >> > > On Wed, Aug 14, 2019, at 06:29, Mickael Maison
>> wrote:
>> > > >> > > > Hi Rajini,
>> > > >> > > >
>> > > >> > > > Thanks for the KIP!
>> > > >> > > > I really like that authorize() will be able to
>> take a
>> > batch of
>> > > >> > > > requests, this will speed up many
>> implementations!
>> > > >> > > >
>> > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <
>> > > >> rajinisivaram@gmail.com>
>> > > >> > > wrote:
>> > > >> > > > >
>> > > >> > > > > Thanks David! I have fixed the typo.
>> > > >> > > > >
>> > > >> > > > > Also made a couple of changes to make the
>> context
>> > > interfaces more
>> > > >> > > generic.
>> > > >> > > > > KafkaRequestContext now returns the 16-bit API
>> key as
>> > Colin
>> > > >> suggested
>> > > >> > > as
>> > > >> > > > > well as the friendly name used in metrics
>> which are
>> > useful
>> > > in
>> > > >> audit
>> > > >> > > logs.
>> > > >> > > > > `Authorizer#start` is now provided a generic
>> > `BrokerInfo`
>> > > >> interface
>> > > >> > > that
>> > > >> > > > > gives cluster id, broker id and endpoint
>> information.
>> > The
>> > > generic
>> > > >> > > interface
>> > > >> > > > > can potentially be used in other broker
>> plugins in
>> > future
>> > > and
>> > > >> provides
>> > > >> > > > > dynamically generated configs like broker id
>> and ports
>> > > which are
>> > > >> > > currently
>> > > >> > > > > not available to plugins unless these configs
>> are
>> > statically
>> > > >> > > configured.
>> > > >> > > > > Please let me know if there are any concerns.
>> > > >> > > > >
>> > > >> > > > > Regards,
>> > > >> > > > >
>> > > >> > > > > Rajini
>> > > >> > > > >
>> > > >> > > > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot <
>> > > djacot@confluent.io>
>> > > >> > > wrote:
>> > > >> > > > >
>> > > >> > > > > > Hi Rajini,
>> > > >> > > > > >
>> > > >> > > > > > Thank you for the update! It looks good to
>> me.
>> > There is a
>> > > typo
>> > > >> in the
>> > > >> > > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` ->
>> > > >> `MANDATORY_AUTHORIZE`.
>> > > >> > > > > >
>> > > >> > > > > > Regards,
>> > > >> > > > > > David
>> > > >> > > > > >
>> > > >> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini
>> Sivaram <
>> > > >> > > rajinisivaram@gmail.com>
>> > > >> > > > > > wrote:
>> > > >> > > > > >
>> > > >> > > > > > > Hi David,
>> > > >> > > > > > >
>> > > >> > > > > > > Thanks for reviewing the KIP! Since
>> questions
>> > about
>> > > >> `authorization
>> > > >> > > mode`
>> > > >> > > > > > > and `count` have come up multiple times, I
>> have
>> > renamed
>> > > both.
>> > > >> > > > > > >
>> > > >> > > > > > > 1) Renamed `count` to
>> `resourceReferenceCount`.
>> > It is
>> > > the
>> > > >> number
>> > > >> > > of times
>> > > >> > > > > > > the resource being authorized is referenced
>> > within the
>> > > >> request.
>> > > >> > > > > > >
>> > > >> > > > > > > 2) Renamed `AuthorizationMode` to
>> `AuditFlag`. It
>> > is
>> > > provided
>> > > >> to
>> > > >> > > improve
>> > > >> > > > > > > audit logging in the authorizer. The enum
>> values
>> > have
>> > > javadoc
>> > > >> which
>> > > >> > > > > > > indicate how the authorization result is
>> used in
>> > each
>> > > of the
>> > > >> modes
>> > > >> > > to
>> > > >> > > > > > > enable authorizers to log audit messages
>> at the
>> > right
>> > > severity
>> > > >> > > level.
>> > > >> > > > > > >
>> > > >> > > > > > > Regards,
>> > > >> > > > > > >
>> > > >> > > > > > > Rajini
>> > > >> > > > > > >
>> > > >> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM David
>> Jacot <
>> > > >> djacot@confluent.io>
>> > > >> > > wrote:
>> > > >> > > > > > >
>> > > >> > > > > > > > Hi Rajini,
>> > > >> > > > > > > >
>> > > >> > > > > > > > Thank you for the KIP. Overall, it looks
>> good
>> > to me.
>> > > I have
>> > > >> few
>> > > >> > > > > > > > questions/suggestions:
>> > > >> > > > > > > >
>> > > >> > > > > > > > 1. It is hard to grasp what
>> `Action#count` is
>> > for. I
>> > > guess I
>> > > >> > > understand
>> > > >> > > > > > > > where you want to go with it but it took
>> me a
>> > while to
>> > > >> figure it
>> > > >> > > out.
>> > > >> > > > > > > > Perhaps, we could come up with a better
>> name
>> > than
>> > > `count`?
>> > > >> > > > > > > >
>> > > >> > > > > > > > 2. I had a hard time trying to
>> understand the
>> > > >> `AuthorizationMode`
>> > > >> > > > > > > concept,
>> > > >> > > > > > > > especially wrt. the OPTIONAL one. My
>> > understanding is
>> > > that
>> > > >> an
>> > > >> > > ACL is
>> > > >> > > > > > > either
>> > > >> > > > > > > > defined or not. Could you elaborate a
>> bit more
>> > on
>> > > that?
>> > > >> > > > > > > >
>> > > >> > > > > > > > Thanks,
>> > > >> > > > > > > > David
>> > > >> > > > > > > >
>> > > >> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don
>> Bosco Durai
>> > <
>> > > >> > > bosco@apache.org>
>> > > >> > > > > > > wrote:
>> > > >> > > > > > > >
>> > > >> > > > > > > > > Hi Rajini
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > 3.2 - This makes sense. Thanks for
>> clarifying.
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > Rest looks fine. Once the
>> implementations are
>> > done,
>> > > it
>> > > >> will be
>> > > >> > > more
>> > > >> > > > > > > clear
>> > > >> > > > > > > > > on the different values RequestType
>> and Mode.
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > Thanks
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > Bosco
>> > > >> > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <
>> > > >> rajinisivaram@gmail.com
>> > > >> > > >
>> > > >> > > > > > wrote:
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > Hi Don,
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > Thanks for the suggestions. A few
>> > responses
>> > > below:
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > 3.1 Can rename and improve docs if
>> we keep
>> > > this. Let's
>> > > >> > > finish the
>> > > >> > > > > > > > > discussion on Colin's suggestions
>> > regarding this
>> > > >> first.
>> > > >> > > > > > > > > 3.2 No, I was thinking of some
>> requests
>> > that
>> > > have an
>> > > >> old
>> > > >> > > way of
>> > > >> > > > > > > > > authorizing
>> > > >> > > > > > > > > and a new way where we have
>> retained the
>> > old
>> > > way for
>> > > >> > > backward
>> > > >> > > > > > > > > compatibility. One example is
>> > Cluster:Create
>> > > >> permission to
>> > > >> > > create
>> > > >> > > > > > > > > topics.
>> > > >> > > > > > > > > We have replaced this with
>> fine-grained
>> > topic
>> > > create
>> > > >> > > access using
>> > > >> > > > > > > > > Topic:Create
>> > > >> > > > > > > > > for topic patterns. But we still
>> check if
>> > user
>> > > has
>> > > >> > > Cluster:Create
>> > > >> > > > > > > > > first. If
>> > > >> > > > > > > > > Denied, the deny is ignored and we
>> check
>> > > >> Topic:Create. We
>> > > >> > > dont
>> > > >> > > > > > want
>> > > >> > > > > > > > to
>> > > >> > > > > > > > > log
>> > > >> > > > > > > > > DENY for Cluster:Create at INFO
>> level for
>> > this,
>> > > since
>> > > >> this
>> > > >> > > is
>> > > >> > > > > > not a
>> > > >> > > > > > > > > mandatory ACL for creating topics.
>> We
>> > will get
>> > > >> appropriate
>> > > >> > > logs
>> > > >> > > > > > > from
>> > > >> > > > > > > > > the
>> > > >> > > > > > > > > subsequent Topic:Create in this
>> case.
>> > > >> > > > > > > > > 3.3 They are not quite the same.
>> FILTER
>> > implies
>> > > that
>> > > >> user
>> > > >> > > > > > actually
>> > > >> > > > > > > > > requested access to and performed
>> some
>> > > operation on
>> > > >> the
>> > > >> > > filtered
>> > > >> > > > > > > > > resources.
>> > > >> > > > > > > > > LIST_AUTHORZED did not result in
>> any
>> > actual
>> > > access.
>> > > >> User
>> > > >> > > simply
>> > > >> > > > > > > > wanted
>> > > >> > > > > > > > > to
>> > > >> > > > > > > > > know what they are allowed to
>> access.
>> > > >> > > > > > > > > 3.4 Request types are Produce,
>> JoinGroup,
>> > > OffsetCommit
>> > > >> > > etc. So
>> > > >> > > > > > that
>> > > >> > > > > > > > is
>> > > >> > > > > > > > > different from authorization mode,
>> > operation
>> > > etc.
>> > > >> > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > On Thu, Aug 8, 2019 at 11:36 PM
>> Don Bosco
>> > Durai
>> > > <
>> > > >> > > > > > bosco@apache.org>
>> > > >> > > > > > > > > wrote:
>> > > >> > > > > > > > >
>> > > >> > > > > > > > > > Hi Rajini
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > Thanks for clarifying. This is
>> very
>> > helpful...
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > #1 - On the Ranger side, we
>> should be
>> > able to
>> > > handle
>> > > >> > > multiple
>> > > >> > > > > > > > > requests at
>> > > >> > > > > > > > > > the same time. I was just not
>> sure how
>> > much
>> > > >> processing
>> > > >> > > overhead
>> > > >> > > > > > > > will
>> > > >> > > > > > > > > be
>> > > >> > > > > > > > > > there on the Broker side to
>> split and
>> > then
>> > > >> consolidate
>> > > >> > > the
>> > > >> > > > > > > results.
>> > > >> > > > > > > > > If it
>> > > >> > > > > > > > > > is negligible, then this is the
>> better
>> > way.
>> > > It will
>> > > >> > > make it
>> > > >> > > > > > > future
>> > > >> > > > > > > > > proof.
>> > > >> > > > > > > > > > #2 - I agree, having a single
>> "start"
>> > call
>> > > makes it
>> > > >> > > cleaner.
>> > > >> > > > > > The
>> > > >> > > > > > > > > > Authorization implementation
>> will only
>> > have
>> > > to do
>> > > >> the
>> > > >> > > > > > > > initialization
>> > > >> > > > > > > > > only
>> > > >> > > > > > > > > > once.
>> > > >> > > > > > > > > > #3.1 - Thanks for the
>> clarification. I
>> > think
>> > > it
>> > > >> makes
>> > > >> > > sense to
>> > > >> > > > > > > have
>> > > >> > > > > > > > > this.
>> > > >> > > > > > > > > > The term "Mode" might not be
>> explicit
>> > enough.
>> > > >> > > Essentially it
>> > > >> > > > > > > seems
>> > > >> > > > > > > > > you want
>> > > >> > > > > > > > > > the Authorizer to know the
>> > Intent/Purpose of
>> > > the
>> > > >> > > authorize call
>> > > >> > > > > > > and
>> > > >> > > > > > > > > let the
>> > > >> > > > > > > > > > Authorizer decide what to log as
>> Audit
>> > event.
>> > > >> Changing
>> > > >> > > the
>> > > >> > > > > > > > > class/field name
>> > > >> > > > > > > > > > or giving more documentation
>> will do.
>> > > >> > > > > > > > > > #3.2 - Regarding the option
>> "OPTIONAL".
>> > Are
>> > > you
>> > > >> thinking
>> > > >> > > from
>> > > >> > > > > > > > > chaining
>> > > >> > > > > > > > > > multiple Authorizer? If so, I
>> am not
>> > sure
>> > > whether
>> > > >> the
>> > > >> > > Broker
>> > > >> > > > > > > would
>> > > >> > > > > > > > > have
>> > > >> > > > > > > > > > enough information to make that
>> > decision. I
>> > > feel the
>> > > >> > > Authorizer
>> > > >> > > > > > > > will
>> > > >> > > > > > > > > be the
>> > > >> > > > > > > > > > one who would have that
>> knowledge. E.g.
>> > in
>> > > Ranger
>> > > >> we have
>> > > >> > > > > > > explicit
>> > > >> > > > > > > > > deny,
>> > > >> > > > > > > > > > which means no matter what, the
>> request
>> > > should be
>> > > >> denied
>> > > >> > > for
>> > > >> > > > > > the
>> > > >> > > > > > > > > user/group
>> > > >> > > > > > > > > > or condition. So if you are
>> thinking of
>> > > chaining
>> > > >> > > Authorizers,
>> > > >> > > > > > > then
>> > > >> > > > > > > > > > responses should have the third
>> state,
>> > e.g.
>> > > >> > > "DENIED_FINAL", in
>> > > >> > > > > > > > which
>> > > >> > > > > > > > > case
>> > > >> > > > > > > > > > if there is an Authorization
>> chain, it
>> > will
>> > > be stop
>> > > >> and
>> > > >> > > the
>> > > >> > > > > > > request
>> > > >> > > > > > > > > will be
>> > > >> > > > > > > > > > denied and if it is just denied,
>> then
>> > you
>> > > might fall
>> > > >> > > back to
>> > > >> > > > > > next
>> > > >> > > > > > > > > > authorizer. If we don't have
>> chaining of
>> > > >> Authorizing in
>> > > >> > > mind,
>> > > >> > > > > > > then
>> > > >> > > > > > > > we
>> > > >> > > > > > > > > > should reconsider OPTIONAL for
>> now. Or
>> > > clarify under
>> > > >> > > which
>> > > >> > > > > > > scenario
>> > > >> > > > > > > > > > OPTIONAL will be called.
>> > > >> > > > > > > > > > #3.3 Regarding, FILTER v/s
>> > LIST_AUTHORIZED,
>> > > isn't
>> > > >> > > > > > > LIST_AUTHORIZED a
>> > > >> > > > > > > > > > special case for "FILTER"?
>> > > >> > > > > > > > > > #3.4 KafkaRequestContext.
>> requestType()
>> > v/s
>> > > Action.
>> > > >> > > > > > > > > authorizationMode. I
>> > > >> > > > > > > > > > am not sure about the overlap or
>> > ambiguity.
>> > > >> > > > > > > > > > #4 - Cool. This is good, it will
>> be less
>> > > stress on
>> > > >> the
>> > > >> > > > > > > Authorizer.
>> > > >> > > > > > > > > Ranger
>> > > >> > > > > > > > > > already supports the "count"
>> concept
>> > and also
>> > > has
>> > > >> > > batching
>> > > >> > > > > > > > > capability to
>> > > >> > > > > > > > > > aggregate similar requests to
>> reduce the
>> > > number of
>> > > >> audit
>> > > >> > > logs
>> > > >> > > > > > to
>> > > >> > > > > > > > > write. We
>> > > >> > > > > > > > > > should be able to pass this
>> through.
>> > > >> > > > > > > > > > #5 - Assuming if the object
>> instance is
>> > going
>> > > out of
>> > > >> > > scope, we
>> > > >> > > > > > > > > should be
>> > > >> > > > > > > > > > fine. Not a super important ask.
>> Ranger
>> > is
>> > > already
>> > > >> > > catching
>> > > >> > > > > > KILL
>> > > >> > > > > > > > > signal for
>> > > >> > > > > > > > > > clean up.
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > Thanks again. These are good
>> > enhancements.
>> > > >> Appreciate
>> > > >> > > your
>> > > >> > > > > > > efforts
>> > > >> > > > > > > > > here.
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > Bosco
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > On 8/8/19, 2:03 AM, "Rajini
>> Sivaram" <
>> > > >> > > rajinisivaram@gmail.com
>> > > >> > > > > > >
>> > > >> > > > > > > > > wrote:
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > Hi Don,
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > Thanks for reviewing the KIP.
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > 1. I had this originally as
>> a single
>> > > Action, but
>> > > >> > > thought it
>> > > >> > > > > > > may
>> > > >> > > > > > > > > be
>> > > >> > > > > > > > > > useful
>> > > >> > > > > > > > > > to support batched authorize
>> calls
>> > as
>> > > well and
>> > > >> keep
>> > > >> > > it
>> > > >> > > > > > > > > consistent with
>> > > >> > > > > > > > > > other methods. Single
>> requests can
>> > contain
>> > > >> multiple
>> > > >> > > topics.
>> > > >> > > > > > > For
>> > > >> > > > > > > > > > example a
>> > > >> > > > > > > > > > produce request can contain
>> records
>> > for
>> > > several
>> > > >> > > partitions
>> > > >> > > > > > of
>> > > >> > > > > > > > > different
>> > > >> > > > > > > > > > topics. Broker could
>> potentially
>> > > authorize these
>> > > >> > > together.
>> > > >> > > > > > > For
>> > > >> > > > > > > > > > SimpleAclAuthorizer, batched
>> > authorize
>> > > methods
>> > > >> don't
>> > > >> > > > > > provide
>> > > >> > > > > > > > any
>> > > >> > > > > > > > > > optimisation since lookup is
>> based
>> > on
>> > > resources
>> > > >> > > followed by
>> > > >> > > > > > > the
>> > > >> > > > > > > > > > matching
>> > > >> > > > > > > > > > logic. But some authorizers
>> may
>> > manage
>> > > ACLs by
>> > > >> user
>> > > >> > > > > > principal
>> > > >> > > > > > > > > rather
>> > > >> > > > > > > > > > than
>> > > >> > > > > > > > > > resource and may be able to
>> optimize
>> > > batched
>> > > >> > > requests. I am
>> > > >> > > > > > > ok
>> > > >> > > > > > > > > with
>> > > >> > > > > > > > > > using
>> > > >> > > > > > > > > > single Action if this is
>> likely to
>> > cause
>> > > issues.
>> > > >> > > > > > > > > > 2. If you have two
>> listeners, one
>> > for
>> > > >> inter-broker
>> > > >> > > traffic
>> > > >> > > > > > > and
>> > > >> > > > > > > > > another
>> > > >> > > > > > > > > > for
>> > > >> > > > > > > > > > external clients, start
>> method is
>> > invoked
>> > > twice,
>> > > >> > > once for
>> > > >> > > > > > > each
>> > > >> > > > > > > > > > listener. On
>> > > >> > > > > > > > > > second thought, that may be
>> > confusing and
>> > > a
>> > > >> single
>> > > >> > > start()
>> > > >> > > > > > > > > invocation
>> > > >> > > > > > > > > > that
>> > > >> > > > > > > > > > provides all listener
>> information
>> > and
>> > > returns
>> > > >> > > multiple
>> > > >> > > > > > > futures
>> > > >> > > > > > > > > would be
>> > > >> > > > > > > > > > better. Will update the KIP.
>> > > >> > > > > > > > > > 3. A typical example is a
>> consumer
>> > > subscribing
>> > > >> to a
>> > > >> > > regex
>> > > >> > > > > > > > > pattern. We
>> > > >> > > > > > > > > > request all topic metadata
>> from the
>> > > broker in
>> > > >> order
>> > > >> > > to
>> > > >> > > > > > decide
>> > > >> > > > > > > > > whether
>> > > >> > > > > > > > > > the
>> > > >> > > > > > > > > > pattern matches, expecting to
>> > receive a
>> > > list of
>> > > >> > > authorised
>> > > >> > > > > > > > > topics. The
>> > > >> > > > > > > > > > user
>> > > >> > > > > > > > > > is not asking to subscribe
>> to an
>> > > unauthorized
>> > > >> topic.
>> > > >> > > If
>> > > >> > > > > > there
>> > > >> > > > > > > > > are 10000
>> > > >> > > > > > > > > > topics in the cluster and
>> the user
>> > has
>> > > access
>> > > >> to 100
>> > > >> > > of
>> > > >> > > > > > them,
>> > > >> > > > > > > > at
>> > > >> > > > > > > > > the
>> > > >> > > > > > > > > > moment
>> > > >> > > > > > > > > > we log 9900 DENIED log
>> entries at
>> > INFO
>> > > level in
>> > > >> > > > > > > > > SimpleAclAuthorizer.
>> > > >> > > > > > > > > > The
>> > > >> > > > > > > > > > proposal is to authorize this
>> > request with
>> > > >> > > > > > > > > AuthorizationMode.FILTER, so
>> > > >> > > > > > > > > > that authorizers can log
>> resources
>> > that
>> > > are
>> > > >> filtered
>> > > >> > > out at
>> > > >> > > > > > > > > lower level
>> > > >> > > > > > > > > > like DEBUG since this is not
>> an
>> > attempt to
>> > > >> access
>> > > >> > > > > > > unauthorized
>> > > >> > > > > > > > > > resources.
>> > > >> > > > > > > > > > Brokers already handle these
>> > differently
>> > > since
>> > > >> no
>> > > >> > > > > > > authorization
>> > > >> > > > > > > > > error
>> > > >> > > > > > > > > > is
>> > > >> > > > > > > > > > returned to the client in
>> these
>> > cases.
>> > > Providing
>> > > >> > > > > > > authorization
>> > > >> > > > > > > > > mode to
>> > > >> > > > > > > > > > authorizers enables
>> authorizer
>> > > implementations
>> > > >> to
>> > > >> > > generate
>> > > >> > > > > > > > > better audit
>> > > >> > > > > > > > > > logs.
>> > > >> > > > > > > > > > 4. Each request may contain
>> multiple
>> > > instances
>> > > >> of
>> > > >> > > the same
>> > > >> > > > > > > > > authorizable
>> > > >> > > > > > > > > > resource. For example a
>> produce
>> > request
>> > > may
>> > > >> contain
>> > > >> > > records
>> > > >> > > > > > > for
>> > > >> > > > > > > > > 10
>> > > >> > > > > > > > > > partitions of the same
>> topic. At the
>> > > moment, we
>> > > >> > > invoke
>> > > >> > > > > > > > authorize
>> > > >> > > > > > > > > > method 10
>> > > >> > > > > > > > > > times. The proposal is to
>> invoke it
>> > once
>> > > with
>> > > >> > > count=10. The
>> > > >> > > > > > > > > count is
>> > > >> > > > > > > > > > provided to authorizer just
>> for
>> > audit
>> > > logging
>> > > >> > > purposes.
>> > > >> > > > > > > > > > 5. Authorizer implements
>> Closeable,
>> > so you
>> > > >> could use
>> > > >> > > > > > close()
>> > > >> > > > > > > to
>> > > >> > > > > > > > > flush
>> > > >> > > > > > > > > > audits?
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > On Thu, Aug 8, 2019 at 7:01
>> AM Don
>> > Bosco
>> > > Durai <
>> > > >> > > > > > > > bosco@apache.org
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > wrote:
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > > Rajini
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Thanks for putting this
>> together.
>> > It is
>> > > >> looking
>> > > >> > > good. I
>> > > >> > > > > > > have
>> > > >> > > > > > > > > few
>> > > >> > > > > > > > > > > questions...
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > 1.
>> List<AuthorizationResult>
>> > > authorize(...,
>> > > >> > > List<Action>
>> > > >> > > > > > > > > actions).
>> > > >> > > > > > > > > > Do you
>> > > >> > > > > > > > > > > see a scenario where the
>> broker
>> > will
>> > > call
>> > > >> > > authorize for
>> > > >> > > > > > > > > multiple
>> > > >> > > > > > > > > > topics at
>> > > >> > > > > > > > > > > the same time? I can
>> understand
>> > that
>> > > during
>> > > >> > > > > > > creating/deleting
>> > > >> > > > > > > > > ACLS,
>> > > >> > > > > > > > > > > multiple permissions for
>> multiple
>> > > resources
>> > > >> might
>> > > >> > > be
>> > > >> > > > > > done.
>> > > >> > > > > > > > For
>> > > >> > > > > > > > > > authorize
>> > > >> > > > > > > > > > > call, would this be a
>> case? And
>> > does the
>> > > >> Authorize
>> > > >> > > > > > > > > implementation
>> > > >> > > > > > > > > > will be
>> > > >> > > > > > > > > > > able to do performance
>> > optimization
>> > > because of
>> > > >> > > this? Or
>> > > >> > > > > > > > should
>> > > >> > > > > > > > > we
>> > > >> > > > > > > > > > just keep
>> > > >> > > > > > > > > > > it simple? I don't see it
>> as an
>> > issue
>> > > from
>> > > >> Apache
>> > > >> > > Ranger
>> > > >> > > > > > > > side,
>> > > >> > > > > > > > > but
>> > > >> > > > > > > > > > just
>> > > >> > > > > > > > > > > checking to see whether we
>> need
>> > to be
>> > > aware of
>> > > >> > > something.
>> > > >> > > > > > > > > > > 2. Should I assume that the
>> > > SecurityProtocol
>> > > >> passed
>> > > >> > > > > > during
>> > > >> > > > > > > > > start and
>> > > >> > > > > > > > > > the
>> > > >> > > > > > > > > > > one return by
>> > > >> > > KafkaRequestContext.securityProtocol() will
>> > > >> > > > > > > be
>> > > >> > > > > > > > > the
>> > > >> > > > > > > > > > same?
>> > > >> > > > > > > > > > > CompletableFuture<Void>
>> > start(String
>> > > >> listenerName,
>> > > >> > > > > > > > > SecurityProtocol
>> > > >> > > > > > > > > > > securityProtocol);
>> > > >> > > > > > > > > > >
>> > KafkaRequestContext.securityProtocol()
>> > > >> > > > > > > > > > > 3. What is the purpose of
>> > > AuthorizationMode?
>> > > >> How
>> > > >> > > does the
>> > > >> > > > > > > > > broker
>> > > >> > > > > > > > > > decide
>> > > >> > > > > > > > > > > what mode to use when the
>> > authorize()
>> > > method
>> > > >> is
>> > > >> > > called?
>> > > >> > > > > > > > > > > 4. Can we clarify "count"
>> in
>> > Action a
>> > > bit
>> > > >> more?
>> > > >> > > How is it
>> > > >> > > > > > > > used?
>> > > >> > > > > > > > > > > 5. Do you feel having
>> "stop"
>> > along with
>> > > >> "start" be
>> > > >> > > > > > helpful?
>> > > >> > > > > > > > > E.g. In
>> > > >> > > > > > > > > > Ranger
>> > > >> > > > > > > > > > > we try to optimize the
>> Audit
>> > writing by
>> > > >> caching
>> > > >> > > the logs
>> > > >> > > > > > > for
>> > > >> > > > > > > > a
>> > > >> > > > > > > > > fixed
>> > > >> > > > > > > > > > > interval. But when the
>> Broker
>> > > terminates, we
>> > > >> do a
>> > > >> > > forced
>> > > >> > > > > > > > flush.
>> > > >> > > > > > > > > > Having an
>> > > >> > > > > > > > > > > explicit "stop" might give
>> us a
>> > formal
>> > > way to
>> > > >> > > flush our
>> > > >> > > > > > > > audits.
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Thanks
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Bosco
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > On 8/7/19, 3:59 PM,
>> "Rajini
>> > Sivaram" <
>> > > >> > > > > > > > rajinisivaram@gmail.com
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > > wrote:
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Hi Ron/Harsha/Satish,
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Thanks for reviewing
>> the KIP!
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > We should perhaps have
>> a wider
>> > > discussion
>> > > >> > > outside
>> > > >> > > > > > this
>> > > >> > > > > > > > KIP
>> > > >> > > > > > > > > for
>> > > >> > > > > > > > > > > refactoring
>> > > >> > > > > > > > > > > clients so that others
>> who
>> > are not
>> > > >> following
>> > > >> > > this KIP
>> > > >> > > > > > > > also
>> > > >> > > > > > > > > > notice the
>> > > >> > > > > > > > > > > discussion. Satish,
>> would you
>> > like
>> > > to
>> > > >> start a
>> > > >> > > > > > > discussion
>> > > >> > > > > > > > > thread
>> > > >> > > > > > > > > > on dev?
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Regards,
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > Rajini
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > On Wed, Aug 7, 2019 at
>> 6:21 PM
>> > > Satish
>> > > >> Duggana <
>> > > >> > > > > > > > > > > satish.duggana@gmail.com>
>> > > >> > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > > > I felt the same need
>> when
>> > we want
>> > > to
>> > > >> add a
>> > > >> > > > > > pluggable
>> > > >> > > > > > > > API
>> > > >> > > > > > > > > for
>> > > >> > > > > > > > > > core
>> > > >> > > > > > > > > > > > server
>> functionality. This
>> > does
>> > > not
>> > > >> need to
>> > > >> > > be part
>> > > >> > > > > > > of
>> > > >> > > > > > > > > this
>> > > >> > > > > > > > > > KIP, it
>> > > >> > > > > > > > > > > > can be a separate
>> KIP. I can
>> > > contribute
>> > > >> those
>> > > >> > > > > > > > refactoring
>> > > >> > > > > > > > > > changes if
>> > > >> > > > > > > > > > > > others are OK with
>> that.
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > It is better to have
>> a
>> > structure
>> > > like
>> > > >> below.
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > kafka-common:
>> > > >> > > > > > > > > > > > common classes which
>> can be
>> > used
>> > > in any
>> > > >> of
>> > > >> > > the
>> > > >> > > > > > other
>> > > >> > > > > > > > > modules
>> > > >> > > > > > > > > > in Kafka
>> > > >> > > > > > > > > > > > like client,
>> > Kafka-server-common
>> > > and
>> > > >> server
>> > > >> > > etc.
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > kafka-client-common:
>> > > >> > > > > > > > > > > > common classes which
>> can be
>> > used
>> > > in the
>> > > >> > > client
>> > > >> > > > > > > module.
>> > > >> > > > > > > > > This
>> > > >> > > > > > > > > > can be
>> > > >> > > > > > > > > > > > part of client module
>> > itself.
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > kafka-server-common:
>> > > >> > > > > > > > > > > > classes required
>> only for
>> > > kafka-server.
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > Thanks.
>> > > >> > > > > > > > > > > > Satish.
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > > > On Wed, Aug 7, 2019
>> at 9:28
>> > PM
>> > > Harsha
>> > > >> > > Chintalapani
>> > > >> > > > > > <
>> > > >> > > > > > > > > > kafka@harsha.io>
>> > > >> > > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > Thanks for the KIP
>> Rajini.
>> > > >> > > > > > > > > > > > > Quick thought, it
>> would
>> > be good
>> > > to
>> > > >> have a
>> > > >> > > common
>> > > >> > > > > > > > module
>> > > >> > > > > > > > > > outside of
>> > > >> > > > > > > > > > > > clients
>> > > >> > > > > > > > > > > > > that only applies
>> to
>> > server side
>> > > >> > > interfaces &
>> > > >> > > > > > > > changes.
>> > > >> > > > > > > > > It
>> > > >> > > > > > > > > > looks
>> > > >> > > > > > > > > > > like we
>> > > >> > > > > > > > > > > > are
>> > > >> > > > > > > > > > > > > increasingly in
>> favor of
>> > using
>> > > Java
>> > > >> > > interface for
>> > > >> > > > > > > > > pluggable
>> > > >> > > > > > > > > > > modules on
>> > > >> > > > > > > > > > > > the
>> > > >> > > > > > > > > > > > > broker side.
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > Thanks,
>> > > >> > > > > > > > > > > > > Harsha
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > On Tue, Aug 06,
>> 2019 at
>> > 2:31 PM,
>> > > >> Rajini
>> > > >> > > Sivaram <
>> > > >> > > > > > > > > > > rajinisivaram@gmail.com
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > wrote:
>> > > >> > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > Hi all,
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > I have created a
>> KIP to
>> > > replace the
>> > > >> Scala
>> > > >> > > > > > > > Authorizer
>> > > >> > > > > > > > > API
>> > > >> > > > > > > > > > with a
>> > > >> > > > > > > > > > > new
>> > > >> > > > > > > > > > > > Java
>> > > >> > > > > > > > > > > > > > API:
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > -
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > >
>> > https://cwiki.apache.org/confluence/display/KAFKA/
>> > > >> > > > > > > > > > > > > >
>> > > >> > > KIP-504+-+Add+new+Java+Authorizer+Interface
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > This is
>> replacement for
>> > KIP-50
>> > > >> which was
>> > > >> > > > > > accepted
>> > > >> > > > > > > > > but never
>> > > >> > > > > > > > > > > merged.
>> > > >> > > > > > > > > > > > Apart
>> > > >> > > > > > > > > > > > > > from moving to a
>> Java
>> > API
>> > > consistent
>> > > >> > > with other
>> > > >> > > > > > > > > pluggable
>> > > >> > > > > > > > > > > interfaces
>> > > >> > > > > > > > > > > > in the
>> > > >> > > > > > > > > > > > > > broker, KIP-504
>> also
>> > attempts
>> > > to
>> > > >> address
>> > > >> > > known
>> > > >> > > > > > > > > limitations
>> > > >> > > > > > > > > > in the
>> > > >> > > > > > > > > > > > > > authorizer. If
>> you have
>> > come
>> > > across
>> > > >> other
>> > > >> > > > > > > > > limitations that
>> > > >> > > > > > > > > > you
>> > > >> > > > > > > > > > > would
>> > > >> > > > > > > > > > > > like
>> > > >> > > > > > > > > > > > > > to see addressed
>> in the
>> > new
>> > > API,
>> > > >> please
>> > > >> > > raise
>> > > >> > > > > > > these
>> > > >> > > > > > > > > on the
>> > > >> > > > > > > > > > > discussion
>> > > >> > > > > > > > > > > > > > thread so that
>> we can
>> > > consider those
>> > > >> > > too. All
>> > > >> > > > > > > > > suggestions
>> > > >> > > > > > > > > > and
>> > > >> > > > > > > > > > > feedback
>> > > >> > > > > > > > > > > > are
>> > > >> > > > > > > > > > > > > > welcome.
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > Thank you,
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > > > > Rajini
>> > > >> > > > > > > > > > > > > >
>> > > >> > > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > > >
>> > > >> > > > > > > >
>> > > >> > > > > > >
>> > > >> > > > > >
>> > > >> > > >
>> > > >> > >
>> > > >> >
>> > > >>
>> > > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>> >
>> >
>> >
>> >
>>
>>
>>
>>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Jun,
Thanks for your note. Yes, agree that sync authorize() will be sufficient
in most cases based on the numbers we expect to see in the foreseeable
future.
For async CreateAcls/DeleteAcls, we will need to add a callback in
KafkaApis and add to purgatory if future is not complete. Since we don't
expect large numbers of these requests, we should be able to use the
existing purgatory implementation. Since we support pluggable ACL storage
(e.g. a database), this may be worth doing. I do agree that we don't expect
a large amount of ACL updates, but we could still have multiple concurrent
requests updating ACLs when topics are created or deleted. If the same user
issues large numbers of ACL update requests, we could use throttling to
delay these, enabling other requests to progress, but this still blocks the
request thread during the actual ACL updates.
Since the underlying API to update ACLs in the backend may be async anyway
(e.g. when ACLs are stored in a Kafka topic), an async API to avoid request
threads being blocked seems useful even though it adds some additional
complexity to the Kafka code. Since we don't currently support pluggable
backends for CreateTopic/AlterTopic or AlterConfgs, perhaps we don't need
to process these asynchronously.
Regards,
Rajini
On Mon, Sep 9, 2019 at 5:52 AM Jun Rao <ju...@confluent.io> wrote:
> Hi, Rajini,
>
> Thanks for the reply. The 4-step approach that you outlined seems to work.
> Overall, I can see that the async authorize() api could lead to an overall
> more efficient implementation. The tradeoff is that we have to code every
> request with an extra stage. To me, this optimization seems too early. With
> 100K topics, each with 1KB worth of users, the required ACL space is about
> 100MB and we should be able to cache everything. Beyond that, we still have
> the option of dividing up the topic and ACL metadata such that every broker
> is only required to cache a subset of all metadata.
>
> Regarding making create/delete api async, I still have mixed feelings on
> this. I am wondering if the benefit is worth the added complexity. To me,
> both operations are rare. If one request thread is blocked occasionally for
> a few millisecs, it's probably ok since it mostly just affects one client.
> In the case that a particular user issues many those operations in a short
> window. Could we just use CPU throttling to prevent too much resource being
> used? There are a few other similar types of requests such as create/alter
> configs, create/alter topic, etc. Do we plan to add an extra processing
> stage for each of them too in the future?
>
> Thanks,
>
> Jun
>
> On Sun, Sep 8, 2019 at 10:04 AM Rajini Sivaram <ra...@gmail.com>
> wrote:
>
> > Thanks Ismael. That is good point. I have updated the javadoc.
> >
> > On Sun, Sep 8, 2019 at 5:38 PM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Rajini,
> > >
> > > Can you please update the javadoc for every sync method to be explicit
> > > about the fact that they should not block and the reasoning?
> > >
> > > Ismael
> > >
> > > On Sun, Sep 8, 2019, 9:21 AM Rajini Sivaram <ra...@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Thanks everyone for the very useful discussion. I have updated the
> KIP
> > to
> > > > make *createAcls()* and *deleteAcls()* asynchronous. Also added a
> > section
> > > > under `*Rejected Alternatives*` for async *authorize()*, which we can
> > > > revisit in future if requirement arises.
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Fri, Sep 6, 2019 at 9:17 PM Ismael Juma <is...@juma.me.uk>
> wrote:
> > > >
> > > > > It's not uncommon for people to do large numbers of admin
> operations
> > > per
> > > > > second. We've seen it many times in production. It's often a bug,
> but
> > > the
> > > > > system must cope with it without affecting other users of the same
> > > Kafka
> > > > > cluster.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Fri, Sep 6, 2019, 9:32 AM Don Bosco Durai <bo...@apache.org>
> > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > I might be wrong here, but here are few of my assumptions..
> > > > > >
> > > > > > 1. Unlike transactional services, in Kafka most of the users are
> > > > > > application users, so once the connection is established, then
> they
> > > are
> > > > > > going to be long running sessions. So the caching is more
> > predictable
> > > > and
> > > > > > heavy lifting can be done during initial session setup and cache
> > can
> > > be
> > > > > > frequently updated using background thread. KSQL might have a
> > > different
> > > > > > flow, but I don't have enough datapoints for it.
> > > > > > 2. For publish/consume authorization calls, providing async
> option
> > > will
> > > > > be
> > > > > > misleading for Kafka like load
> > > > > > 3. For admin calls, hopefully no one does hundreds of call in
> > > multiple
> > > > > > thread within few seconds. This would be a bad design.
> > > > > >
> > > > > > So I feel, we should be okay with sync calls for now. And
> consider
> > > > async
> > > > > > in the future if needed. At least in Apache Ranger
> implementation,
> > we
> > > > > would
> > > > > > be okay using sync operation. I am not sure if any other plugin
> > > > > > implementation would benefit from async implementation.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Bosco
> > > > > >
> > > > > > On 9/6/19, 8:45 AM, "Ismael Juma" <is...@juma.me.uk> wrote:
> > > > > >
> > > > > > One more thing: if people have strong opinions that authorize
> > has
> > > > to
> > > > > be
> > > > > > sync, we could leave it like that for now. If needed, we can
> > > later
> > > > > add
> > > > > > an
> > > > > > authorizeAsync with another method returning a Boolean
> > indicating
> > > > > that
> > > > > > it's
> > > > > > supported. That is, there is a path to evolve the interface
> > (even
> > > > if
> > > > > a
> > > > > > bit
> > > > > > ugly).
> > > > > >
> > > > > > What about the other methods, is there consensus that they
> > should
> > > > be
> > > > > > updated to return CompletionStage?
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Fri, Sep 6, 2019, 8:32 AM Ismael Juma <is...@juma.me.uk>
> > > > wrote:
> > > > > >
> > > > > > > I'm on the move, but what Rajini said seems reasonable. I
> > don't
> > > > > think
> > > > > > > using SSDs solves the issue. They can still hang for
> seconds
> > > when
> > > > > > they
> > > > > > > fail. Also, many people may not have local SSDs available
> > > (remote
> > > > > > SSDs like
> > > > > > > EBS hang for tens of seconds when there are issues).
> > > > > > >
> > > > > > > We are currently vulnerable to all of these in the normal
> > > > > read/write
> > > > > > path,
> > > > > > > but it seems suboptimal to continue assuming non blocking
> > > > behavior
> > > > > > for
> > > > > > > things that actually do block.
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Fri, Sep 6, 2019, 8:24 AM Rajini Sivaram <
> > > > > rajinisivaram@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Jun,
> > > > > > >>
> > > > > > >> For ACLs, the size is also dependent on the number of
> users.
> > > So
> > > > in
> > > > > > >> deployments with large number of users where some users
> are
> > > not
> > > > > > active, an
> > > > > > >> LRU cache may be useful. Agree that there may be other
> > > solutions
> > > > > > that help
> > > > > > >> avoid the requirement for an async API even for this case.
> > > > > > >>
> > > > > > >> I wasn't expecting our threading model to change very
> much.
> > > > Having
> > > > > > said
> > > > > > >> that, I haven't figured out how the purgatory can be
> adapted
> > > to
> > > > > work
> > > > > > >> efficiently for this. I was thinking:
> > > > > > >> 1) If authorization future is complete when returning from
> > > > > > authorize()
> > > > > > >> call, we invoke the method immediately on the request
> thread
> > > > > > >> 2) If future is not complete, the request goes into a
> > > purgatory
> > > > > > >> 3) When future completes, we mark the request ready to be
> > run.
> > > > > This
> > > > > > can be
> > > > > > >> done on ForkJoinPool.commonPool or an executor. But we
> don't
> > > > > handle
> > > > > > the
> > > > > > >> request on that thread
> > > > > > >> 4) We handle the request on the request thread after 3).
> > > > > > >>
> > > > > > >> I totally agree that this adds complexity to the code.
> Will
> > > wait
> > > > > for
> > > > > > >> Ismael
> > > > > > >> to comment on whether he had a different model in mind.
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >>
> > > > > > >> Rajini
> > > > > > >>
> > > > > > >>
> > > > > > >> On Thu, Sep 5, 2019 at 10:29 PM Jun Rao <jun@confluent.io
> >
> > > > wrote:
> > > > > > >>
> > > > > > >> > Hi, Ismael,
> > > > > > >> >
> > > > > > >> > Yes, I agree that there are 2 separate discussion
> points,
> > > one
> > > > on
> > > > > > the API
> > > > > > >> > and the other on the implementation.
> > > > > > >> >
> > > > > > >> > First on the API, in general, my feeling is that the
> > > > authorize()
> > > > > > call is
> > > > > > >> > expected to be fast to reduce request latency. Adding an
> > > async
> > > > > API
> > > > > > >> could be
> > > > > > >> > mis-leading since it might give users the impression
> that
> > it
> > > > > > would be
> > > > > > >> > arbitrarily long. In the short term, currently, we cache
> > all
> > > > > > partition
> > > > > > >> > level metadata on every broker. The authorization
> metadata
> > > is
> > > > > > >> proportional
> > > > > > >> > to the number of topics, which typically is less in
> size.
> > So
> > > > it
> > > > > > we can
> > > > > > >> > cache all partition level metadata, we should be able to
> > > cache
> > > > > all
> > > > > > >> > authorization metadata. Longer term, if we want to
> support
> > > > more
> > > > > > metadata
> > > > > > >> > beyond memory, another option is to put all metadata in
> an
> > > SSD
> > > > > > backed
> > > > > > >> > key/value store. The lookup time could still be under
> 1ms,
> > > > which
> > > > > > we
> > > > > > >> could
> > > > > > >> > potentially absorb in the request threads. So, I am not
> > sure
> > > > > async
> > > > > > >> > authorize() api is strictly needed even in the long
> term.
> > > > > > >> >
> > > > > > >> > Second on the implementation. I was more concerned about
> > the
> > > > > > impact on
> > > > > > >> our
> > > > > > >> > threading model. If we make authorize() async, some
> > threads
> > > > have
> > > > > > to
> > > > > > >> > complete the future when it's ready. It seems to me that
> > > those
> > > > > > threads
> > > > > > >> have
> > > > > > >> > to be inside the Authorizer implementation since only it
> > > knows
> > > > > > when to
> > > > > > >> > complete a future? If so, it feels weird that all the
> > logic
> > > of
> > > > > > request
> > > > > > >> > handling post authorize() is now handled by external
> > threads
> > > > > > inside a
> > > > > > >> > plugin. It's not clear how we can configure and monitor
> > > them,
> > > > > and
> > > > > > how
> > > > > > >> > throttling would work. Perhaps we can provide a bit more
> > > > detail
> > > > > > on the
> > > > > > >> new
> > > > > > >> > threading model.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> >
> > > > > > >> > Jun
> > > > > > >> >
> > > > > > >> > On Wed, Sep 4, 2019 at 9:56 PM Ismael Juma <
> > > ismael@juma.me.uk
> > > > >
> > > > > > wrote:
> > > > > > >> >
> > > > > > >> > > Hi Jun,
> > > > > > >> > >
> > > > > > >> > > I think it's important to discuss the pluggable API
> > versus
> > > > the
> > > > > > calling
> > > > > > >> > > implementation as two separate points.
> > > > > > >> > >
> > > > > > >> > > From an API perspective, are you suggesting that we
> > should
> > > > > tell
> > > > > > users
> > > > > > >> > that
> > > > > > >> > > they cannot block in authorize? Or are you saying that
> > > it's
> > > > OK
> > > > > > to
> > > > > > >> block
> > > > > > >> > on
> > > > > > >> > > authorize on occasion and the recommendation would be
> > for
> > > > > > people to
> > > > > > >> > > increase the number of threads in the request thread
> > pool?
> > > > > > >> > Architecturally,
> > > > > > >> > > this feels wrong in my opinion.
> > > > > > >> > >
> > > > > > >> > > From the calling implementation perspective, you don't
> > > need
> > > > a
> > > > > > >> callback.
> > > > > > >> > You
> > > > > > >> > > can basically say:
> > > > > > >> > >
> > > > > > >> > > authorize(...).map { result =>
> > > > > > >> > > ...
> > > > > > >> > > }
> > > > > > >> > >
> > > > > > >> > > And then have a common method take that Future and
> > handle
> > > it
> > > > > > >> > synchronously
> > > > > > >> > > if it's already complete or submit it to the purgatory
> > if
> > > > not.
> > > > > > This is
> > > > > > >> > > similar to how many modern async web libraries work.
> > > > > > >> > >
> > > > > > >> > > As Rajini said, this could be done later. The initial
> > > > > > implementation
> > > > > > >> > could
> > > > > > >> > > simply do `toCompletableFuture().get()` to do it
> > > > > synchronously.
> > > > > > But it
> > > > > > >> > > would mean that we could add this functionality
> without
> > > > > > changing the
> > > > > > >> > > pluggable interface.
> > > > > > >> > >
> > > > > > >> > > Ismael
> > > > > > >> > >
> > > > > > >> > > On Wed, Sep 4, 2019 at 5:29 AM Jun Rao <
> > jun@confluent.io>
> > > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Hi, Rajini,
> > > > > > >> > > >
> > > > > > >> > > > Thanks for the KIP. I was concerned about #4 too. If
> > we
> > > > > > change the
> > > > > > >> > > handling
> > > > > > >> > > > of all requests to use an async authorize() api,
> will
> > > that
> > > > > > cause the
> > > > > > >> > code
> > > > > > >> > > > much harder to understand? There are quite a few
> > > callbacks
> > > > > > already.
> > > > > > >> I
> > > > > > >> > am
> > > > > > >> > > > not sure that we want to introduce more of those.
> The
> > > > > benefit
> > > > > > from
> > > > > > >> > async
> > > > > > >> > > > authorize() api seems limited.
> > > > > > >> > > >
> > > > > > >> > > > Jun
> > > > > > >> > > >
> > > > > > >> > > > On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <
> > > > > > >> rajinisivaram@gmail.com
> > > > > > >> > >
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Hi Don,
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks for your note.
> > > > > > >> > > > >
> > > > > > >> > > > > 1) The intention is to avoid blocking in the
> calling
> > > > > > thread. We
> > > > > > >> > already
> > > > > > >> > > > > have several requests that are put into a
> purgatory
> > > when
> > > > > > waiting
> > > > > > >> for
> > > > > > >> > > > remote
> > > > > > >> > > > > communication, for example produce request waiting
> > for
> > > > > > >> replication.
> > > > > > >> > > Since
> > > > > > >> > > > > we have a limited number of request threads in the
> > > > broker,
> > > > > > we
> > > > > > >> want to
> > > > > > >> > > > make
> > > > > > >> > > > > progress with other requests, while one is waiting
> > on
> > > > any
> > > > > > form of
> > > > > > >> > > remote
> > > > > > >> > > > > communication.
> > > > > > >> > > > >
> > > > > > >> > > > > 2) Async management calls will be useful with the
> > > > default
> > > > > > >> authorizer
> > > > > > >> > > when
> > > > > > >> > > > > KIP-500 removes ZK and we rely on Kafka instead.
> Our
> > > > > current
> > > > > > >> ZK-based
> > > > > > >> > > > > implementation as well as any custom
> implementations
> > > > that
> > > > > > don't
> > > > > > >> want
> > > > > > >> > to
> > > > > > >> > > > be
> > > > > > >> > > > > async will just need to return a sync'ed value. So
> > > > instead
> > > > > > of
> > > > > > >> > > returning `
> > > > > > >> > > > > value`, the code would just return
> > > > > > >> > > > > `CompletableFuture.completedFuture(value)
> > > > > > >> > > > > `. So it would be just a single line change in the
> > > > > > implementation
> > > > > > >> > with
> > > > > > >> > > > the
> > > > > > >> > > > > new API. The caller would treat completedFuture
> > > exactly
> > > > as
> > > > > > it does
> > > > > > >> > > today,
> > > > > > >> > > > > processing the request synchronously without
> using a
> > > > > > purgatory.
> > > > > > >> > > > >
> > > > > > >> > > > > 3) For implementations that return a
> completedFuture
> > > as
> > > > > > described
> > > > > > >> in
> > > > > > >> > > 2),
> > > > > > >> > > > > the behaviour would remain exactly the same. No
> > > > additional
> > > > > > >> threads or
> > > > > > >> > > > > purgatory will be used for this case. So there
> would
> > > be
> > > > no
> > > > > > >> > performance
> > > > > > >> > > > > penalty. For implementations that return a future
> > that
> > > > is
> > > > > > not
> > > > > > >> > complete,
> > > > > > >> > > > we
> > > > > > >> > > > > prioritise running more requests concurrently. So
> > in a
> > > > > > deployment
> > > > > > >> > with
> > > > > > >> > > a
> > > > > > >> > > > > large number of clients, we would improve
> > performance
> > > by
> > > > > > allowing
> > > > > > >> > other
> > > > > > >> > > > > requests to be processed on the request threads
> > while
> > > > some
> > > > > > are
> > > > > > >> > waiting
> > > > > > >> > > > for
> > > > > > >> > > > > authorization metadata.
> > > > > > >> > > > >
> > > > > > >> > > > > 4) I was concerned about this too. The goal is to
> > make
> > > > the
> > > > > > API
> > > > > > >> > flexible
> > > > > > >> > > > > enough to handle large scale deployments in future
> > > when
> > > > > > caching
> > > > > > >> all
> > > > > > >> > > > > authorization metadata in each broker is not
> viable.
> > > > Using
> > > > > > an
> > > > > > >> async
> > > > > > >> > API
> > > > > > >> > > > > that returns CompletionStage, the caller has the
> > > option
> > > > to
> > > > > > handle
> > > > > > >> the
> > > > > > >> > > > > result synchronously or asynchronously, so we
> don't
> > > > > > necessarily
> > > > > > >> need
> > > > > > >> > to
> > > > > > >> > > > > update the calling code right away. Custom
> > authorizers
> > > > > > using the
> > > > > > >> > async
> > > > > > >> > > > API
> > > > > > >> > > > > have full control over whether authorization is
> > > > performed
> > > > > > in-line
> > > > > > >> > since
> > > > > > >> > > > > completedFuture will always be handled
> > synchronously.
> > > We
> > > > > do
> > > > > > need
> > > > > > >> to
> > > > > > >> > > > update
> > > > > > >> > > > > KafkaApis to take advantage of the asynchronous
> API
> > to
> > > > > > improve
> > > > > > >> scale.
> > > > > > >> > > > Even
> > > > > > >> > > > > though this is a big change, since we will be
> doing
> > > the
> > > > > > same for
> > > > > > >> all
> > > > > > >> > > > > requests, it shouldn't be too hard to maintain
> since
> > > the
> > > > > > same
> > > > > > >> pattern
> > > > > > >> > > > will
> > > > > > >> > > > > be used for all requests.
> > > > > > >> > > > >
> > > > > > >> > > > > Regards,
> > > > > > >> > > > >
> > > > > > >> > > > > Rajini
> > > > > > >> > > > >
> > > > > > >> > > > > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <
> > > > > > bosco@apache.org
> > > > > > >> >
> > > > > > >> > > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Hi Rajini
> > > > > > >> > > > > >
> > > > > > >> > > > > > Help me understand this a bit more.
> > > > > > >> > > > > >
> > > > > > >> > > > > > 1. For all practical purpose, without
> > authorization
> > > > you
> > > > > > can't
> > > > > > >> go to
> > > > > > >> > > the
> > > > > > >> > > > > > next step. The calling code needs to block
> anyway.
> > > So
> > > > > > should we
> > > > > > >> > just
> > > > > > >> > > > let
> > > > > > >> > > > > > the implementation code do the async part?
> > > > > > >> > > > > > 2. If you feel management calls need to be
> async,
> > > then
> > > > > we
> > > > > > should
> > > > > > >> > > > consider
> > > > > > >> > > > > > another set of async APIs. I don't feel we
> should
> > > > > > complicate it
> > > > > > >> > > > further (
> > > > > > >> > > > > > 3. Another concern I have is wrt performance.
> > Kafka
> > > > has
> > > > > > been
> > > > > > >> built
> > > > > > >> > to
> > > > > > >> > > > > > handle 1000s per second requests. Not sure
> whether
> > > > > making
> > > > > > it
> > > > > > >> async
> > > > > > >> > > will
> > > > > > >> > > > > add
> > > > > > >> > > > > > any unnecessary overhead.
> > > > > > >> > > > > > 4. How much complication would this add on the
> > > calling
> > > > > > side?
> > > > > > >> And is
> > > > > > >> > > it
> > > > > > >> > > > > > worth it?
> > > > > > >> > > > > >
> > > > > > >> > > > > > Thanks
> > > > > > >> > > > > >
> > > > > > >> > > > > > Bosco
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > On 9/3/19, 8:50 AM, "Rajini Sivaram" <
> > > > > > rajinisivaram@gmail.com>
> > > > > > >> > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > Hi all,
> > > > > > >> > > > > >
> > > > > > >> > > > > > Ismael brought up a point that it will be
> good
> > > to
> > > > > > make the
> > > > > > >> > > > Authorizer
> > > > > > >> > > > > > interface asynchronous to avoid blocking
> > request
> > > > > > threads
> > > > > > >> during
> > > > > > >> > > > > remote
> > > > > > >> > > > > > operations.
> > > > > > >> > > > > >
> > > > > > >> > > > > > 1) Since we want to support different
> backends
> > > for
> > > > > > >> > authorization
> > > > > > >> > > > > > metadata,
> > > > > > >> > > > > > making createAcls() and deleteAcls()
> > > asynchronous
> > > > > > makes
> > > > > > >> sense
> > > > > > >> > > since
> > > > > > >> > > > > > these
> > > > > > >> > > > > > always involve remote operations. When
> KIP-500
> > > > > removes
> > > > > > >> > ZooKeeper,
> > > > > > >> > > > we
> > > > > > >> > > > > > would
> > > > > > >> > > > > > want to move ACLs to Kafka and async updates
> > > will
> > > > > > avoid
> > > > > > >> > > unnecessary
> > > > > > >> > > > > > blocking.
> > > > > > >> > > > > > 2) For authorize() method, we currently use
> > > cached
> > > > > > ACLs in
> > > > > > >> the
> > > > > > >> > > > > built-in
> > > > > > >> > > > > > authorizers, so synchronous authorise
> > operations
> > > > > work
> > > > > > well
> > > > > > >> now.
> > > > > > >> > > But
> > > > > > >> > > > > > async
> > > > > > >> > > > > > authorize() would support this scenario as
> > well
> > > as
> > > > > > >> authorizers
> > > > > > >> > in
> > > > > > >> > > > > large
> > > > > > >> > > > > > organisations where an LRU cache would
> enable
> > a
> > > > > > smaller
> > > > > > >> cache
> > > > > > >> > > even
> > > > > > >> > > > > > when the
> > > > > > >> > > > > > backend holds a large amount of ACLs for
> > > > > infrequently
> > > > > > used
> > > > > > >> > > > resources
> > > > > > >> > > > > or
> > > > > > >> > > > > > users who don't use the system frequently.
> > > > > > >> > > > > >
> > > > > > >> > > > > > For both cases, the built-in authorizer will
> > > > > continue
> > > > > > to be
> > > > > > >> > > > > > synchronous,
> > > > > > >> > > > > > using CompletableFuture.completedFuture() to
> > > > return
> > > > > > the
> > > > > > >> actual
> > > > > > >> > > > > result.
> > > > > > >> > > > > > But
> > > > > > >> > > > > > async API will make custom authorizer
> > > > > implementations
> > > > > > more
> > > > > > >> > > > flexible.
> > > > > > >> > > > > I
> > > > > > >> > > > > > would like to know if there are any concerns
> > > with
> > > > > > these
> > > > > > >> changes
> > > > > > >> > > > > before
> > > > > > >> > > > > > updating the KIP.
> > > > > > >> > > > > >
> > > > > > >> > > > > > *Proposed API:*
> > > > > > >> > > > > > public interface Authorizer extends
> > > Configurable,
> > > > > > Closeable
> > > > > > >> {
> > > > > > >> > > > > >
> > > > > > >> > > > > > Map<Endpoint, CompletionStage<Void>>
> > > > > > >> > > start(AuthorizerServerInfo
> > > > > > >> > > > > > serverInfo);
> > > > > > >> > > > > >
> List<CompletionStage<AuthorizationResult>>
> > > > > > >> > > > > > authorize(AuthorizableRequestContext
> > > > requestContext,
> > > > > > >> > List<Action>
> > > > > > >> > > > > > actions);
> > > > > > >> > > > > > List<CompletionStage<AclCreateResult>>
> > > > > > >> > > > > > createAcls(AuthorizableRequestContext
> > > > > requestContext,
> > > > > > >> > > > > List<AclBinding>
> > > > > > >> > > > > > aclBindings);
> > > > > > >> > > > > > List<CompletionStage<AclDeleteResult>>
> > > > > > >> > > > > > deleteAcls(AuthorizableRequestContext
> > > > > requestContext,
> > > > > > >> > > > > > List<AclBindingFilter> aclBindingFilters);
> > > > > > >> > > > > > CompletionStage<Collection<AclBinding>>
> > > > > > >> > acls(AclBindingFilter
> > > > > > >> > > > > > filter);
> > > > > > >> > > > > > }
> > > > > > >> > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > > > Thank you,
> > > > > > >> > > > > >
> > > > > > >> > > > > > Rajini
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco
> > Durai
> > > <
> > > > > > >> > > bosco@apache.org>
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Hi Rajini
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Thanks for clarifying. I am good for now.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Regards
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Bosco
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <
> > > > > > >> > > rajinisivaram@gmail.com
> > > > > > >> > > > >
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Hi Don,
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > That should be fine. I guess Ranger
> > loads
> > > > > > policies
> > > > > > >> from
> > > > > > >> > the
> > > > > > >> > > > > > database
> > > > > > >> > > > > > > synchronously when the authorizer is
> > > > > configured,
> > > > > > >> similar
> > > > > > >> > to
> > > > > > >> > > > > > > SimpleAclAuthorizer loading from
> > > ZooKeeper.
> > > > > > Ranger can
> > > > > > >> > > > continue
> > > > > > >> > > > > > to load
> > > > > > >> > > > > > > synchronously from `configure()` or
> > > > `start()`
> > > > > > and
> > > > > > >> return
> > > > > > >> > an
> > > > > > >> > > > > > empty map
> > > > > > >> > > > > > > from
> > > > > > >> > > > > > > `start()`. That would retain the
> > existing
> > > > > > behaviour..
> > > > > > >> > When
> > > > > > >> > > > the
> > > > > > >> > > > > > same
> > > > > > >> > > > > > > database stores policies for all
> > listeners
> > > > and
> > > > > > the
> > > > > > >> > policies
> > > > > > >> > > > are
> > > > > > >> > > > > > not
> > > > > > >> > > > > > > stored
> > > > > > >> > > > > > > in Kafka, there is no value in making
> > the
> > > > load
> > > > > > >> > > asynchronous.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Regards,
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Rajini
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On Fri, Aug 16, 2019 at 6:43 PM Don
> > Bosco
> > > > > Durai
> > > > > > <
> > > > > > >> > > > > > bosco@apache.org>
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > Hi Rajini
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Assuming this doesn't affect custom
> > > > plugins,
> > > > > > I don't
> > > > > > >> > have
> > > > > > >> > > > any
> > > > > > >> > > > > > > concerns
> > > > > > >> > > > > > > > regarding this.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > I do have one question regarding:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > "For authorizers that don’t store
> > > metadata
> > > > > in
> > > > > > >> > ZooKeeper,
> > > > > > >> > > > > > ensure that
> > > > > > >> > > > > > > > authorizer metadata for each
> listener
> > is
> > > > > > available
> > > > > > >> > before
> > > > > > >> > > > > > starting
> > > > > > >> > > > > > > up the
> > > > > > >> > > > > > > > listener. This enables different
> > > > > authorization
> > > > > > >> metadata
> > > > > > >> > > > > stores
> > > > > > >> > > > > > for
> > > > > > >> > > > > > > > different listeners."
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Since Ranger uses its own database
> for
> > > > > storing
> > > > > > >> > policies,
> > > > > > >> > > do
> > > > > > >> > > > > you
> > > > > > >> > > > > > > anticipate
> > > > > > >> > > > > > > > any issues?
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Thanks
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Bosco
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > On 8/16/19, 6:49 AM, "Rajini
> > Sivaram" <
> > > > > > >> > > > > > rajinisivaram@gmail.com>
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Hi all,
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > I made another change to the
> KIP.
> > > The
> > > > > KIP
> > > > > > was
> > > > > > >> > > > originally
> > > > > > >> > > > > > > proposing to
> > > > > > >> > > > > > > > extend SimpleAclAuthorizer to
> also
> > > > > > implement the
> > > > > > >> > new
> > > > > > >> > > > API
> > > > > > >> > > > > > (in
> > > > > > >> > > > > > > addition
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > the existing API). But since we
> > use
> > > > the
> > > > > > new API
> > > > > > >> > when
> > > > > > >> > > > > > available,
> > > > > > >> > > > > > > this
> > > > > > >> > > > > > > > can
> > > > > > >> > > > > > > > break custom authorizers that
> > extend
> > > > > this
> > > > > > class
> > > > > > >> and
> > > > > > >> > > > > > override
> > > > > > >> > > > > > > specific
> > > > > > >> > > > > > > > methods of the old API. To avoid
> > > > > breaking
> > > > > > any
> > > > > > >> > > existing
> > > > > > >> > > > > > custom
> > > > > > >> > > > > > > > implementations that extend this
> > > > class,
> > > > > > >> > particularly
> > > > > > >> > > > > > because it
> > > > > > >> > > > > > > is in
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > public package
> > kafka.security.auth,
> > > > the
> > > > > > KIP now
> > > > > > >> > > > proposes
> > > > > > >> > > > > to
> > > > > > >> > > > > > > retain the
> > > > > > >> > > > > > > > old
> > > > > > >> > > > > > > > authorizer as-is.
> > > SimpleAclAuthorizer
> > > > > > will
> > > > > > >> > continue
> > > > > > >> > > to
> > > > > > >> > > > > > > implement the
> > > > > > >> > > > > > > > old
> > > > > > >> > > > > > > > API, but will be deprecated. A
> new
> > > > > > authorizer
> > > > > > >> > > > > > implementation
> > > > > > >> > > > > > > >
> > > > kafka.security.authorizer.AclAuthorizer
> > > > > > will be
> > > > > > >> > added
> > > > > > >> > > > for
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > new API
> > > > > > >> > > > > > > > (this
> > > > > > >> > > > > > > > will not be in the public
> > package).
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Please let me know if you have
> any
> > > > > > concerns.
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Regards,
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Rajini
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > On Fri, Aug 16, 2019 at 8:48 AM
> > > Rajini
> > > > > > Sivaram <
> > > > > > >> > > > > > > > rajinisivaram@gmail.com>
> > > > > > >> > > > > > > > wrote:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > Thanks Colin.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > If there are no other
> concerns,
> > I
> > > > will
> > > > > > start
> > > > > > >> vote
> > > > > > >> > > > later
> > > > > > >> > > > > > today.
> > > > > > >> > > > > > > Many
> > > > > > >> > > > > > > > thanks
> > > > > > >> > > > > > > > > to every one for the feedback.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Regards,
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Rajini
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On Thu, Aug 15, 2019 at 11:57
> PM
> > > > Colin
> > > > > > McCabe
> > > > > > >> <
> > > > > > >> > > > > > > cmccabe@apache.org>
> > > > > > >> > > > > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >> Thanks, Rajini. It looks
> good
> > to
> > > > me.
> > > > > > >> > > > > > > > >>
> > > > > > >> > > > > > > > >> best,
> > > > > > >> > > > > > > > >> Colin
> > > > > > >> > > > > > > > >>
> > > > > > >> > > > > > > > >>
> > > > > > >> > > > > > > > >> On Thu, Aug 15, 2019, at
> 11:37,
> > > > > Rajini
> > > > > > >> Sivaram
> > > > > > >> > > > wrote:
> > > > > > >> > > > > > > > >> > Hi Colin,
> > > > > > >> > > > > > > > >> >
> > > > > > >> > > > > > > > >> > Thanks for the review. I
> have
> > > > > > updated the
> > > > > > >> KIP
> > > > > > >> > to
> > > > > > >> > > > > move
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > interfaces for
> > > > > > >> > > > > > > > >> > request context and server
> > info
> > > > to
> > > > > > the
> > > > > > >> > > authorizer
> > > > > > >> > > > > > package.
> > > > > > >> > > > > > > These
> > > > > > >> > > > > > > > are now
> > > > > > >> > > > > > > > >> > called
> > > AuthorizableRequestContext
> > > > > and
> > > > > > >> > > > > > AuthorizerServerInfo.
> > > > > > >> > > > > > > > Endpoint is
> > > > > > >> > > > > > > > >> now
> > > > > > >> > > > > > > > >> > a class in
> > > > org.apache.kafka.common
> > > > > > to make
> > > > > > >> it
> > > > > > >> > > > > reusable
> > > > > > >> > > > > > > since we
> > > > > > >> > > > > > > > already
> > > > > > >> > > > > > > > >> > have multiple
> implementations
> > > of
> > > > > it.
> > > > > > I have
> > > > > > >> > > > removed
> > > > > > >> > > > > > > requestName
> > > > > > >> > > > > > > > from the
> > > > > > >> > > > > > > > >> > request context interface
> > since
> > > > > > authorizers
> > > > > > >> > can
> > > > > > >> > > > > > distinguish
> > > > > > >> > > > > > > > follower
> > > > > > >> > > > > > > > >> fetch
> > > > > > >> > > > > > > > >> > and consumer fetch from the
> > > > > > operation being
> > > > > > >> > > > > > authorized. So
> > > > > > >> > > > > > > 16-bit
> > > > > > >> > > > > > > > >> request
> > > > > > >> > > > > > > > >> > type should be sufficient
> for
> > > > audit
> > > > > > >> logging.
> > > > > > >> > > Also
> > > > > > >> > > > > > replaced
> > > > > > >> > > > > > > > AuditFlag
> > > > > > >> > > > > > > > >> with
> > > > > > >> > > > > > > > >> > two booleans as you
> > suggested.
> > > > > > >> > > > > > > > >> >
> > > > > > >> > > > > > > > >> > Can you take another look
> and
> > > see
> > > > > if
> > > > > > the
> > > > > > >> KIP
> > > > > > >> > is
> > > > > > >> > > > > ready
> > > > > > >> > > > > > for
> > > > > > >> > > > > > > voting?
> > > > > > >> > > > > > > > >> >
> > > > > > >> > > > > > > > >> > Thanks for all your help!
> > > > > > >> > > > > > > > >> >
> > > > > > >> > > > > > > > >> > Regards,
> > > > > > >> > > > > > > > >> >
> > > > > > >> > > > > > > > >> > Rajini
> > > > > > >> > > > > > > > >> >
> > > > > > >> > > > > > > > >> > On Wed, Aug 14, 2019 at
> 8:59
> > PM
> > > > > Colin
> > > > > > >> McCabe <
> > > > > > >> > > > > > > cmccabe@apache.org>
> > > > > > >> > > > > > > > >> wrote:
> > > > > > >> > > > > > > > >> >
> > > > > > >> > > > > > > > >> > > Hi Rajini,
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> > > I think it would be good
> to
> > > > > rename
> > > > > > >> > > > > > KafkaRequestContext to
> > > > > > >> > > > > > > > something
> > > > > > >> > > > > > > > >> like
> > > > > > >> > > > > > > > >> > >
> AuthorizableRequestContext,
> > > and
> > > > > > put it in
> > > > > > >> > the
> > > > > > >> > > > > > > > >> > >
> > > > > org.apache.kafka.server.authorizer
> > > > > > >> > namespace.
> > > > > > >> > > > If
> > > > > > >> > > > > > we put
> > > > > > >> > > > > > > it in
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > >> > > org.apache.kafka.common
> > > > > namespace,
> > > > > > then
> > > > > > >> it's
> > > > > > >> > > not
> > > > > > >> > > > > > really
> > > > > > >> > > > > > > clear
> > > > > > >> > > > > > > > that
> > > > > > >> > > > > > > > >> it's
> > > > > > >> > > > > > > > >> > > part of the Authorizer
> API.
> > > > > Since
> > > > > > this
> > > > > > >> > class
> > > > > > >> > > is
> > > > > > >> > > > > > purely an
> > > > > > >> > > > > > > > interface,
> > > > > > >> > > > > > > > >> with
> > > > > > >> > > > > > > > >> > > no concrete
> implementation
> > of
> > > > > > anything,
> > > > > > >> > > there's
> > > > > > >> > > > > > nothing
> > > > > > >> > > > > > > common
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > >> really
> > > > > > >> > > > > > > > >> > > reuse in any case. We
> > > > definitely
> > > > > > don't
> > > > > > >> want
> > > > > > >> > > > > > someone to
> > > > > > >> > > > > > > > accidentally
> > > > > > >> > > > > > > > >> add or
> > > > > > >> > > > > > > > >> > > remove methods because
> they
> > > > think
> > > > > > this is
> > > > > > >> > just
> > > > > > >> > > > > > another
> > > > > > >> > > > > > > internal
> > > > > > >> > > > > > > > class
> > > > > > >> > > > > > > > >> used
> > > > > > >> > > > > > > > >> > > for requests.
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> > > The BrokerInfo class is a
> > > nice
> > > > > > >> improvement.
> > > > > > >> > > It
> > > > > > >> > > > > > looks
> > > > > > >> > > > > > > like it
> > > > > > >> > > > > > > > will be
> > > > > > >> > > > > > > > >> > > useful for passing in
> > > > information
> > > > > > about
> > > > > > >> the
> > > > > > >> > > > > context
> > > > > > >> > > > > > we're
> > > > > > >> > > > > > > > running
> > > > > > >> > > > > > > > >> in. It
> > > > > > >> > > > > > > > >> > > would be nice to call
> this
> > > > class
> > > > > > >> ServerInfo
> > > > > > >> > > > rather
> > > > > > >> > > > > > than
> > > > > > >> > > > > > > > BrokerInfo,
> > > > > > >> > > > > > > > >> since
> > > > > > >> > > > > > > > >> > > we will want to run the
> > > > > authorizer
> > > > > > on
> > > > > > >> > > > controllers
> > > > > > >> > > > > > as well
> > > > > > >> > > > > > > as on
> > > > > > >> > > > > > > > >> brokers,
> > > > > > >> > > > > > > > >> > > and the controller may
> run
> > > as a
> > > > > > separate
> > > > > > >> > > process
> > > > > > >> > > > > > post
> > > > > > >> > > > > > > KIP-500.
> > > > > > >> > > > > > > > I also
> > > > > > >> > > > > > > > >> > > think that this class
> > should
> > > be
> > > > > in
> > > > > > the
> > > > > > >> > > > > > > > >>
> > > org.apache.kafka.server.authorizer
> > > > > > >> > > > > > > > >> > > namespace. Again, it is
> an
> > > > > > interface,
> > > > > > >> not a
> > > > > > >> > > > > > concrete
> > > > > > >> > > > > > > > implementation,
> > > > > > >> > > > > > > > >> and
> > > > > > >> > > > > > > > >> > > it's an interface that is
> > > very
> > > > > > >> specifically
> > > > > > >> > > for
> > > > > > >> > > > > the
> > > > > > >> > > > > > > authorizer.
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> > > I agree that we probably
> > > don't
> > > > > have
> > > > > > >> enough
> > > > > > >> > > > > > information
> > > > > > >> > > > > > > > preserved for
> > > > > > >> > > > > > > > >> > > requests currently to
> > always
> > > > know
> > > > > > what
> > > > > > >> > entity
> > > > > > >> > > > made
> > > > > > >> > > > > > them.
> > > > > > >> > > > > > > So we
> > > > > > >> > > > > > > > can
> > > > > > >> > > > > > > > >> leave
> > > > > > >> > > > > > > > >> > > that out for now (except
> in
> > > the
> > > > > > special
> > > > > > >> case
> > > > > > >> > > of
> > > > > > >> > > > > > Fetch).
> > > > > > >> > > > > > > > Perhaps we
> > > > > > >> > > > > > > > >> can add
> > > > > > >> > > > > > > > >> > > this later if it's
> needed.
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> > > I understand the
> intention
> > > > behind
> > > > > > >> > > > > AuthorizationMode
> > > > > > >> > > > > > > (which is
> > > > > > >> > > > > > > > now
> > > > > > >> > > > > > > > >> called
> > > > > > >> > > > > > > > >> > > AuditFlag in the latest
> > > > > > revision). But
> > > > > > >> it
> > > > > > >> > > still
> > > > > > >> > > > > > feels
> > > > > > >> > > > > > > > complex. What
> > > > > > >> > > > > > > > >> if we
> > > > > > >> > > > > > > > >> > > just had two booleans in
> > > > Action:
> > > > > > >> > logSuccesses
> > > > > > >> > > > and
> > > > > > >> > > > > > > logFailures?
> > > > > > >> > > > > > > > That
> > > > > > >> > > > > > > > >> seems
> > > > > > >> > > > > > > > >> > > to cover all the cases
> > here.
> > > > > > >> > > > MANDATORY_AUTHORIZE
> > > > > > >> > > > > =
> > > > > > >> > > > > > true,
> > > > > > >> > > > > > > true.
> > > > > > >> > > > > > > > >> > > OPTIONAL_AUTHORIZE =
> true,
> > > > false.
> > > > > > >> FILTER =
> > > > > > >> > > > true,
> > > > > > >> > > > > > false.
> > > > > > >> > > > > > > > >> LIST_AUTHORIZED =
> > > > > > >> > > > > > > > >> > > false, false. Would
> there
> > be
> > > > > > anything
> > > > > > >> lost
> > > > > > >> > > > versus
> > > > > > >> > > > > > having
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > enum?
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> > > best,
> > > > > > >> > > > > > > > >> > > Colin
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> > >
> > > > > > >> > > > > > > > >> > > On Wed, Aug 14, 2019, at
> > > 06:29,
> > > > > > Mickael
> > > > > > >> > Maison
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > > > > >> > > > Hi Rajini,
> > > > > > >> > > > > > > > >> > > >
> > > > > > >> > > > > > > > >> > > > Thanks for the KIP!
> > > > > > >> > > > > > > > >> > > > I really like that
> > > > authorize()
> > > > > > will be
> > > > > > >> > able
> > > > > > >> > > to
> > > > > > >> > > > > > take a
> > > > > > >> > > > > > > batch of
> > > > > > >> > > > > > > > >> > > > requests, this will
> speed
> > > up
> > > > > many
> > > > > > >> > > > > implementations!
> > > > > > >> > > > > > > > >> > > >
> > > > > > >> > > > > > > > >> > > > On Tue, Aug 13, 2019 at
> > > 5:57
> > > > PM
> > > > > > Rajini
> > > > > > >> > > > Sivaram <
> > > > > > >> > > > > > > > >> rajinisivaram@gmail.com>
> > > > > > >> > > > > > > > >> > > wrote:
> > > > > > >> > > > > > > > >> > > > >
> > > > > > >> > > > > > > > >> > > > > Thanks David! I have
> > > fixed
> > > > > the
> > > > > > typo.
> > > > > > >> > > > > > > > >> > > > >
> > > > > > >> > > > > > > > >> > > > > Also made a couple of
> > > > changes
> > > > > > to make
> > > > > > >> > the
> > > > > > >> > > > > > context
> > > > > > >> > > > > > > > interfaces more
> > > > > > >> > > > > > > > >> > > generic.
> > > > > > >> > > > > > > > >> > > > > KafkaRequestContext
> now
> > > > > > returns the
> > > > > > >> > 16-bit
> > > > > > >> > > > API
> > > > > > >> > > > > > key as
> > > > > > >> > > > > > > Colin
> > > > > > >> > > > > > > > >> suggested
> > > > > > >> > > > > > > > >> > > as
> > > > > > >> > > > > > > > >> > > > > well as the friendly
> > name
> > > > > used
> > > > > > in
> > > > > > >> > metrics
> > > > > > >> > > > > which
> > > > > > >> > > > > > are
> > > > > > >> > > > > > > useful
> > > > > > >> > > > > > > > in
> > > > > > >> > > > > > > > >> audit
> > > > > > >> > > > > > > > >> > > logs.
> > > > > > >> > > > > > > > >> > > > > `Authorizer#start` is
> > now
> > > > > > provided a
> > > > > > >> > > generic
> > > > > > >> > > > > > > `BrokerInfo`
> > > > > > >> > > > > > > > >> interface
> > > > > > >> > > > > > > > >> > > that
> > > > > > >> > > > > > > > >> > > > > gives cluster id,
> > broker
> > > id
> > > > > and
> > > > > > >> endpoint
> > > > > > >> > > > > > information.
> > > > > > >> > > > > > > The
> > > > > > >> > > > > > > > generic
> > > > > > >> > > > > > > > >> > > interface
> > > > > > >> > > > > > > > >> > > > > can potentially be
> used
> > > in
> > > > > > other
> > > > > > >> broker
> > > > > > >> > > > > plugins
> > > > > > >> > > > > > in
> > > > > > >> > > > > > > future
> > > > > > >> > > > > > > > and
> > > > > > >> > > > > > > > >> provides
> > > > > > >> > > > > > > > >> > > > > dynamically generated
> > > > configs
> > > > > > like
> > > > > > >> > broker
> > > > > > >> > > id
> > > > > > >> > > > > > and ports
> > > > > > >> > > > > > > > which are
> > > > > > >> > > > > > > > >> > > currently
> > > > > > >> > > > > > > > >> > > > > not available to
> > plugins
> > > > > > unless these
> > > > > > >> > > > configs
> > > > > > >> > > > > > are
> > > > > > >> > > > > > > statically
> > > > > > >> > > > > > > > >> > > configured.
> > > > > > >> > > > > > > > >> > > > > Please let me know if
> > > there
> > > > > > are any
> > > > > > >> > > > concerns.
> > > > > > >> > > > > > > > >> > > > >
> > > > > > >> > > > > > > > >> > > > > Regards,
> > > > > > >> > > > > > > > >> > > > >
> > > > > > >> > > > > > > > >> > > > > Rajini
> > > > > > >> > > > > > > > >> > > > >
> > > > > > >> > > > > > > > >> > > > > On Tue, Aug 13, 2019
> at
> > > > 4:30
> > > > > > PM David
> > > > > > >> > > Jacot
> > > > > > >> > > > <
> > > > > > >> > > > > > > > djacot@confluent.io>
> > > > > > >> > > > > > > > >> > > wrote:
> > > > > > >> > > > > > > > >> > > > >
> > > > > > >> > > > > > > > >> > > > > > Hi Rajini,
> > > > > > >> > > > > > > > >> > > > > >
> > > > > > >> > > > > > > > >> > > > > > Thank you for the
> > > update!
> > > > > It
> > > > > > looks
> > > > > > >> > good
> > > > > > >> > > to
> > > > > > >> > > > > me.
> > > > > > >> > > > > > > There is a
> > > > > > >> > > > > > > > typo
> > > > > > >> > > > > > > > >> in the
> > > > > > >> > > > > > > > >> > > > > > `AuditFlag` enum:
> > > > > > >> > `MANDATORY_AUTHOEIZE`
> > > > > > >> > > ->
> > > > > > >> > > > > > > > >> `MANDATORY_AUTHORIZE`.
> > > > > > >> > > > > > > > >> > > > > >
> > > > > > >> > > > > > > > >> > > > > > Regards,
> > > > > > >> > > > > > > > >> > > > > > David
> > > > > > >> > > > > > > > >> > > > > >
> > > > > > >> > > > > > > > >> > > > > > On Mon, Aug 12,
> 2019
> > at
> > > > > 2:54
> > > > > > PM
> > > > > > >> Rajini
> > > > > > >> > > > > > Sivaram <
> > > > > > >> > > > > > > > >> > > rajinisivaram@gmail.com>
> > > > > > >> > > > > > > > >> > > > > > wrote:
> > > > > > >> > > > > > > > >> > > > > >
> > > > > > >> > > > > > > > >> > > > > > > Hi David,
> > > > > > >> > > > > > > > >> > > > > > >
> > > > > > >> > > > > > > > >> > > > > > > Thanks for
> > reviewing
> > > > the
> > > > > > KIP!
> > > > > > >> Since
> > > > > > >> > > > > > questions
> > > > > > >> > > > > > > about
> > > > > > >> > > > > > > > >> `authorization
> > > > > > >> > > > > > > > >> > > mode`
> > > > > > >> > > > > > > > >> > > > > > > and `count` have
> > come
> > > > up
> > > > > > multiple
> > > > > > >> > > > times, I
> > > > > > >> > > > > > have
> > > > > > >> > > > > > > renamed
> > > > > > >> > > > > > > > both.
> > > > > > >> > > > > > > > >> > > > > > >
> > > > > > >> > > > > > > > >> > > > > > > 1) Renamed
> `count`
> > to
> > > > > > >> > > > > > `resourceReferenceCount`.
> > > > > > >> > > > > > > It is
> > > > > > >> > > > > > > > the
> > > > > > >> > > > > > > > >> number
> > > > > > >> > > > > > > > >> > > of times
> > > > > > >> > > > > > > > >> > > > > > > the resource
> being
> > > > > > authorized is
> > > > > > >> > > > > referenced
> > > > > > >> > > > > > > within the
> > > > > > >> > > > > > > > >> request.
> > > > > > >> > > > > > > > >> > > > > > >
> > > > > > >> > > > > > > > >> > > > > > > 2) Renamed
> > > > > > `AuthorizationMode` to
> > > > > > >> > > > > > `AuditFlag`. It
> > > > > > >> > > > > > > is
> > > > > > >> > > > > > > > provided
> > > > > > >> > > > > > > > >> to
> > > > > > >> > > > > > > > >> > > improve
> > > > > > >> > > > > > > > >> > > > > > > audit logging in
> > the
> > > > > > authorizer.
> > > > > > >> The
> > > > > > >> > > > enum
> > > > > > >> > > > > > values
> > > > > > >> > > > > > > have
> > > > > > >> > > > > > > > javadoc
> > > > > > >> > > > > > > > >> which
> > > > > > >> > > > > > > > >> > > > > > > indicate how the
> > > > > > authorization
> > > > > > >> > result
> > > > > > >> > > is
> > > > > > >> > > > > > used in
> > > > > > >> > > > > > > each
> > > > > > >> > > > > > > > of the
> > > > > > >> > > > > > > > >> modes
> > > > > > >> > > > > > > > >> > > to
> > > > > > >> > > > > > > > >> > > > > > > enable
> authorizers
> > to
> > > > log
> > > > > > audit
> > > > > > >> > > messages
> > > > > > >> > > > > at
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > right
> > > > > > >> > > > > > > > severity
> > > > > > >> > > > > > > > >> > > level.
> > > > > > >> > > > > > > > >> > > > > > >
> > > > > > >> >
> > >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi all,
I have submitted a PR to address a few review comments from Ismael. There
are a couple of interface changes in the PR, I have updated the KIP as well.
1. The `listener()` method in AuthorizableRequestContext and Endpoint
has been renamed to listenerName()
2. Endpoint.listenerName() now returns an Optional<String>. Since this
is in the common package, we may use it in future in clients where we
don't have listener names
Let me know if there are any concerns.
Regards,
Rajini
On Mon, Sep 9, 2019 at 4:53 PM Rajini Sivaram <ra...@gmail.com>
wrote:
> Hi Jun,
>
> Thanks for the response. If we use the existing purgatory implementation,
> we should get additional purgatory metrics for ACL updates with the new
> purgatory name as tag, consistent with what we have for other delayed
> operations. I will add these to the KIP. We also have request metrics which
> indicate local vs remote time with the API key as tag, so that should
> indicate the portion of time waiting for async operations.
>
> Regards,
>
> Rajini
>
> On Mon, Sep 9, 2019 at 4:31 PM Jun Rao <ju...@confluent.io> wrote:
>
>> Hi, Rajini, Ismael,
>>
>> Yes, I can see the argument for making CreateAcls/DeleteAcls async. I am
>> ok
>> with that if you feel the implementation is not too complicated. Should we
>> consider adding some additional metric to reflect the portion of the time
>> spent in waiting for the async operation to complete?
>>
>> Thanks,
>>
>> Jun
>>
>> On Mon, Sep 9, 2019 at 5:20 AM Ismael Juma <is...@gmail.com> wrote:
>>
>> > Hi Jun,
>> >
>> > As you say, even though the average number of operations may be low, the
>> > request rate can be high in bursts. This can overwhelm the request queue
>> > and cause an outage for no good reason. Or the backing storage system
>> may
>> > be slow for a period of time, causing a similar issue.
>> >
>> > I've seen several such issues in production affecting Kafka's
>> > create/alter/delete paths (e.g people creating/deleting thousands/tens
>> of
>> > thousands of topics in a hot loop). As you say, it's not just ACLs that
>> are
>> > vulnerable. In my opinion, we should learn from these experiences when
>> > designing new interfaces and we should adjust the existing ones to be
>> more
>> > resilient. It's also worth mentioning that create/delete topics already
>> > relies on the purgatory if the request timeout is greater than 0.
>> >
>> > Can you elaborate on how CPU throttling would help? It seems to me that
>> it
>> > would not trigger fast enough if there was a slow back-end, for example
>> > (the number of request threads is often lower than 10, so it doesn't
>> take
>> > many blocked requests to cause major problems).
>> >
>> > Ismael
>> >
>> > On Sun, Sep 8, 2019, 9:5k2 PM Jun Rao <ju...@confluent.io> wrote:
>> >
>> > > Hi, Rajini,
>> > >
>> > > Thanks for the reply. The 4-step approach that you outlined seems to
>> > work.
>> > > Overall, I can see that the async authorize() api could lead to an
>> > overall
>> > > more efficient implementation. The tradeoff is that we have to code
>> every
>> > > request with an extra stage. To me, this optimization seems too early.
>> > With
>> > > 100K topics, each with 1KB worth of users, the required ACL space is
>> > about
>> > > 100MB and we should be able to cache everything. Beyond that, we still
>> > have
>> > > the option of dividing up the topic and ACL metadata such that every
>> > broker
>> > > is only required to cache a subset of all metadata.
>> > >
>> > > Regarding making create/delete api async, I still have mixed feelings
>> on
>> > > this. I am wondering if the benefit is worth the added complexity. To
>> me,
>> > > both operations are rare. If one request thread is blocked
>> occasionally
>> > for
>> > > a few millisecs, it's probably ok since it mostly just affects one
>> > client.
>> > > In the case that a particular user issues many those operations in a
>> > short
>> > > window. Could we just use CPU throttling to prevent too much resource
>> > being
>> > > used? There are a few other similar types of requests such as
>> > create/alter
>> > > configs, create/alter topic, etc. Do we plan to add an extra
>> processing
>> > > stage for each of them too in the future?
>> > >
>> > > Thanks,
>> > >
>> > > Jun
>> > >
>> >
>>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Jun,
Thanks for the response. If we use the existing purgatory implementation,
we should get additional purgatory metrics for ACL updates with the new
purgatory name as tag, consistent with what we have for other delayed
operations. I will add these to the KIP. We also have request metrics which
indicate local vs remote time with the API key as tag, so that should
indicate the portion of time waiting for async operations.
Regards,
Rajini
On Mon, Sep 9, 2019 at 4:31 PM Jun Rao <ju...@confluent.io> wrote:
> Hi, Rajini, Ismael,
>
> Yes, I can see the argument for making CreateAcls/DeleteAcls async. I am ok
> with that if you feel the implementation is not too complicated. Should we
> consider adding some additional metric to reflect the portion of the time
> spent in waiting for the async operation to complete?
>
> Thanks,
>
> Jun
>
> On Mon, Sep 9, 2019 at 5:20 AM Ismael Juma <is...@gmail.com> wrote:
>
> > Hi Jun,
> >
> > As you say, even though the average number of operations may be low, the
> > request rate can be high in bursts. This can overwhelm the request queue
> > and cause an outage for no good reason. Or the backing storage system may
> > be slow for a period of time, causing a similar issue.
> >
> > I've seen several such issues in production affecting Kafka's
> > create/alter/delete paths (e.g people creating/deleting thousands/tens of
> > thousands of topics in a hot loop). As you say, it's not just ACLs that
> are
> > vulnerable. In my opinion, we should learn from these experiences when
> > designing new interfaces and we should adjust the existing ones to be
> more
> > resilient. It's also worth mentioning that create/delete topics already
> > relies on the purgatory if the request timeout is greater than 0.
> >
> > Can you elaborate on how CPU throttling would help? It seems to me that
> it
> > would not trigger fast enough if there was a slow back-end, for example
> > (the number of request threads is often lower than 10, so it doesn't take
> > many blocked requests to cause major problems).
> >
> > Ismael
> >
> > On Sun, Sep 8, 2019, 9:5k2 PM Jun Rao <ju...@confluent.io> wrote:
> >
> > > Hi, Rajini,
> > >
> > > Thanks for the reply. The 4-step approach that you outlined seems to
> > work.
> > > Overall, I can see that the async authorize() api could lead to an
> > overall
> > > more efficient implementation. The tradeoff is that we have to code
> every
> > > request with an extra stage. To me, this optimization seems too early.
> > With
> > > 100K topics, each with 1KB worth of users, the required ACL space is
> > about
> > > 100MB and we should be able to cache everything. Beyond that, we still
> > have
> > > the option of dividing up the topic and ACL metadata such that every
> > broker
> > > is only required to cache a subset of all metadata.
> > >
> > > Regarding making create/delete api async, I still have mixed feelings
> on
> > > this. I am wondering if the benefit is worth the added complexity. To
> me,
> > > both operations are rare. If one request thread is blocked occasionally
> > for
> > > a few millisecs, it's probably ok since it mostly just affects one
> > client.
> > > In the case that a particular user issues many those operations in a
> > short
> > > window. Could we just use CPU throttling to prevent too much resource
> > being
> > > used? There are a few other similar types of requests such as
> > create/alter
> > > configs, create/alter topic, etc. Do we plan to add an extra processing
> > > stage for each of them too in the future?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Jun Rao <ju...@confluent.io>.
Hi, Rajini, Ismael,
Yes, I can see the argument for making CreateAcls/DeleteAcls async. I am ok
with that if you feel the implementation is not too complicated. Should we
consider adding some additional metric to reflect the portion of the time
spent in waiting for the async operation to complete?
Thanks,
Jun
On Mon, Sep 9, 2019 at 5:20 AM Ismael Juma <is...@gmail.com> wrote:
> Hi Jun,
>
> As you say, even though the average number of operations may be low, the
> request rate can be high in bursts. This can overwhelm the request queue
> and cause an outage for no good reason. Or the backing storage system may
> be slow for a period of time, causing a similar issue.
>
> I've seen several such issues in production affecting Kafka's
> create/alter/delete paths (e.g people creating/deleting thousands/tens of
> thousands of topics in a hot loop). As you say, it's not just ACLs that are
> vulnerable. In my opinion, we should learn from these experiences when
> designing new interfaces and we should adjust the existing ones to be more
> resilient. It's also worth mentioning that create/delete topics already
> relies on the purgatory if the request timeout is greater than 0.
>
> Can you elaborate on how CPU throttling would help? It seems to me that it
> would not trigger fast enough if there was a slow back-end, for example
> (the number of request threads is often lower than 10, so it doesn't take
> many blocked requests to cause major problems).
>
> Ismael
>
> On Sun, Sep 8, 2019, 9:5k2 PM Jun Rao <ju...@confluent.io> wrote:
>
> > Hi, Rajini,
> >
> > Thanks for the reply. The 4-step approach that you outlined seems to
> work.
> > Overall, I can see that the async authorize() api could lead to an
> overall
> > more efficient implementation. The tradeoff is that we have to code every
> > request with an extra stage. To me, this optimization seems too early.
> With
> > 100K topics, each with 1KB worth of users, the required ACL space is
> about
> > 100MB and we should be able to cache everything. Beyond that, we still
> have
> > the option of dividing up the topic and ACL metadata such that every
> broker
> > is only required to cache a subset of all metadata.
> >
> > Regarding making create/delete api async, I still have mixed feelings on
> > this. I am wondering if the benefit is worth the added complexity. To me,
> > both operations are rare. If one request thread is blocked occasionally
> for
> > a few millisecs, it's probably ok since it mostly just affects one
> client.
> > In the case that a particular user issues many those operations in a
> short
> > window. Could we just use CPU throttling to prevent too much resource
> being
> > used? There are a few other similar types of requests such as
> create/alter
> > configs, create/alter topic, etc. Do we plan to add an extra processing
> > stage for each of them too in the future?
> >
> > Thanks,
> >
> > Jun
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Ismael Juma <is...@gmail.com>.
Hi Jun,
As you say, even though the average number of operations may be low, the
request rate can be high in bursts. This can overwhelm the request queue
and cause an outage for no good reason. Or the backing storage system may
be slow for a period of time, causing a similar issue.
I've seen several such issues in production affecting Kafka's
create/alter/delete paths (e.g people creating/deleting thousands/tens of
thousands of topics in a hot loop). As you say, it's not just ACLs that are
vulnerable. In my opinion, we should learn from these experiences when
designing new interfaces and we should adjust the existing ones to be more
resilient. It's also worth mentioning that create/delete topics already
relies on the purgatory if the request timeout is greater than 0.
Can you elaborate on how CPU throttling would help? It seems to me that it
would not trigger fast enough if there was a slow back-end, for example
(the number of request threads is often lower than 10, so it doesn't take
many blocked requests to cause major problems).
Ismael
On Sun, Sep 8, 2019, 9:5k2 PM Jun Rao <ju...@confluent.io> wrote:
> Hi, Rajini,
>
> Thanks for the reply. The 4-step approach that you outlined seems to work.
> Overall, I can see that the async authorize() api could lead to an overall
> more efficient implementation. The tradeoff is that we have to code every
> request with an extra stage. To me, this optimization seems too early. With
> 100K topics, each with 1KB worth of users, the required ACL space is about
> 100MB and we should be able to cache everything. Beyond that, we still have
> the option of dividing up the topic and ACL metadata such that every broker
> is only required to cache a subset of all metadata.
>
> Regarding making create/delete api async, I still have mixed feelings on
> this. I am wondering if the benefit is worth the added complexity. To me,
> both operations are rare. If one request thread is blocked occasionally for
> a few millisecs, it's probably ok since it mostly just affects one client.
> In the case that a particular user issues many those operations in a short
> window. Could we just use CPU throttling to prevent too much resource being
> used? There are a few other similar types of requests such as create/alter
> configs, create/alter topic, etc. Do we plan to add an extra processing
> stage for each of them too in the future?
>
> Thanks,
>
> Jun
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Jun Rao <ju...@confluent.io>.
Hi, Rajini,
Thanks for the reply. The 4-step approach that you outlined seems to work.
Overall, I can see that the async authorize() api could lead to an overall
more efficient implementation. The tradeoff is that we have to code every
request with an extra stage. To me, this optimization seems too early. With
100K topics, each with 1KB worth of users, the required ACL space is about
100MB and we should be able to cache everything. Beyond that, we still have
the option of dividing up the topic and ACL metadata such that every broker
is only required to cache a subset of all metadata.
Regarding making create/delete api async, I still have mixed feelings on
this. I am wondering if the benefit is worth the added complexity. To me,
both operations are rare. If one request thread is blocked occasionally for
a few millisecs, it's probably ok since it mostly just affects one client.
In the case that a particular user issues many those operations in a short
window. Could we just use CPU throttling to prevent too much resource being
used? There are a few other similar types of requests such as create/alter
configs, create/alter topic, etc. Do we plan to add an extra processing
stage for each of them too in the future?
Thanks,
Jun
On Sun, Sep 8, 2019 at 10:04 AM Rajini Sivaram <ra...@gmail.com>
wrote:
> Thanks Ismael. That is good point. I have updated the javadoc.
>
> On Sun, Sep 8, 2019 at 5:38 PM Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Rajini,
> >
> > Can you please update the javadoc for every sync method to be explicit
> > about the fact that they should not block and the reasoning?
> >
> > Ismael
> >
> > On Sun, Sep 8, 2019, 9:21 AM Rajini Sivaram <ra...@gmail.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > Thanks everyone for the very useful discussion. I have updated the KIP
> to
> > > make *createAcls()* and *deleteAcls()* asynchronous. Also added a
> section
> > > under `*Rejected Alternatives*` for async *authorize()*, which we can
> > > revisit in future if requirement arises.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > > On Fri, Sep 6, 2019 at 9:17 PM Ismael Juma <is...@juma.me.uk> wrote:
> > >
> > > > It's not uncommon for people to do large numbers of admin operations
> > per
> > > > second. We've seen it many times in production. It's often a bug, but
> > the
> > > > system must cope with it without affecting other users of the same
> > Kafka
> > > > cluster.
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Sep 6, 2019, 9:32 AM Don Bosco Durai <bo...@apache.org>
> wrote:
> > > >
> > > > > +1
> > > > >
> > > > > I might be wrong here, but here are few of my assumptions..
> > > > >
> > > > > 1. Unlike transactional services, in Kafka most of the users are
> > > > > application users, so once the connection is established, then they
> > are
> > > > > going to be long running sessions. So the caching is more
> predictable
> > > and
> > > > > heavy lifting can be done during initial session setup and cache
> can
> > be
> > > > > frequently updated using background thread. KSQL might have a
> > different
> > > > > flow, but I don't have enough datapoints for it.
> > > > > 2. For publish/consume authorization calls, providing async option
> > will
> > > > be
> > > > > misleading for Kafka like load
> > > > > 3. For admin calls, hopefully no one does hundreds of call in
> > multiple
> > > > > thread within few seconds. This would be a bad design.
> > > > >
> > > > > So I feel, we should be okay with sync calls for now. And consider
> > > async
> > > > > in the future if needed. At least in Apache Ranger implementation,
> we
> > > > would
> > > > > be okay using sync operation. I am not sure if any other plugin
> > > > > implementation would benefit from async implementation.
> > > > >
> > > > > Thanks
> > > > >
> > > > > Bosco
> > > > >
> > > > > On 9/6/19, 8:45 AM, "Ismael Juma" <is...@juma.me.uk> wrote:
> > > > >
> > > > > One more thing: if people have strong opinions that authorize
> has
> > > to
> > > > be
> > > > > sync, we could leave it like that for now. If needed, we can
> > later
> > > > add
> > > > > an
> > > > > authorizeAsync with another method returning a Boolean
> indicating
> > > > that
> > > > > it's
> > > > > supported. That is, there is a path to evolve the interface
> (even
> > > if
> > > > a
> > > > > bit
> > > > > ugly).
> > > > >
> > > > > What about the other methods, is there consensus that they
> should
> > > be
> > > > > updated to return CompletionStage?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Fri, Sep 6, 2019, 8:32 AM Ismael Juma <is...@juma.me.uk>
> > > wrote:
> > > > >
> > > > > > I'm on the move, but what Rajini said seems reasonable. I
> don't
> > > > think
> > > > > > using SSDs solves the issue. They can still hang for seconds
> > when
> > > > > they
> > > > > > fail. Also, many people may not have local SSDs available
> > (remote
> > > > > SSDs like
> > > > > > EBS hang for tens of seconds when there are issues).
> > > > > >
> > > > > > We are currently vulnerable to all of these in the normal
> > > > read/write
> > > > > path,
> > > > > > but it seems suboptimal to continue assuming non blocking
> > > behavior
> > > > > for
> > > > > > things that actually do block.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Fri, Sep 6, 2019, 8:24 AM Rajini Sivaram <
> > > > rajinisivaram@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Jun,
> > > > > >>
> > > > > >> For ACLs, the size is also dependent on the number of users.
> > So
> > > in
> > > > > >> deployments with large number of users where some users are
> > not
> > > > > active, an
> > > > > >> LRU cache may be useful. Agree that there may be other
> > solutions
> > > > > that help
> > > > > >> avoid the requirement for an async API even for this case.
> > > > > >>
> > > > > >> I wasn't expecting our threading model to change very much.
> > > Having
> > > > > said
> > > > > >> that, I haven't figured out how the purgatory can be adapted
> > to
> > > > work
> > > > > >> efficiently for this. I was thinking:
> > > > > >> 1) If authorization future is complete when returning from
> > > > > authorize()
> > > > > >> call, we invoke the method immediately on the request thread
> > > > > >> 2) If future is not complete, the request goes into a
> > purgatory
> > > > > >> 3) When future completes, we mark the request ready to be
> run.
> > > > This
> > > > > can be
> > > > > >> done on ForkJoinPool.commonPool or an executor. But we don't
> > > > handle
> > > > > the
> > > > > >> request on that thread
> > > > > >> 4) We handle the request on the request thread after 3).
> > > > > >>
> > > > > >> I totally agree that this adds complexity to the code. Will
> > wait
> > > > for
> > > > > >> Ismael
> > > > > >> to comment on whether he had a different model in mind.
> > > > > >>
> > > > > >> Thanks,
> > > > > >>
> > > > > >> Rajini
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Sep 5, 2019 at 10:29 PM Jun Rao <ju...@confluent.io>
> > > wrote:
> > > > > >>
> > > > > >> > Hi, Ismael,
> > > > > >> >
> > > > > >> > Yes, I agree that there are 2 separate discussion points,
> > one
> > > on
> > > > > the API
> > > > > >> > and the other on the implementation.
> > > > > >> >
> > > > > >> > First on the API, in general, my feeling is that the
> > > authorize()
> > > > > call is
> > > > > >> > expected to be fast to reduce request latency. Adding an
> > async
> > > > API
> > > > > >> could be
> > > > > >> > mis-leading since it might give users the impression that
> it
> > > > > would be
> > > > > >> > arbitrarily long. In the short term, currently, we cache
> all
> > > > > partition
> > > > > >> > level metadata on every broker. The authorization metadata
> > is
> > > > > >> proportional
> > > > > >> > to the number of topics, which typically is less in size.
> So
> > > it
> > > > > we can
> > > > > >> > cache all partition level metadata, we should be able to
> > cache
> > > > all
> > > > > >> > authorization metadata. Longer term, if we want to support
> > > more
> > > > > metadata
> > > > > >> > beyond memory, another option is to put all metadata in an
> > SSD
> > > > > backed
> > > > > >> > key/value store. The lookup time could still be under 1ms,
> > > which
> > > > > we
> > > > > >> could
> > > > > >> > potentially absorb in the request threads. So, I am not
> sure
> > > > async
> > > > > >> > authorize() api is strictly needed even in the long term.
> > > > > >> >
> > > > > >> > Second on the implementation. I was more concerned about
> the
> > > > > impact on
> > > > > >> our
> > > > > >> > threading model. If we make authorize() async, some
> threads
> > > have
> > > > > to
> > > > > >> > complete the future when it's ready. It seems to me that
> > those
> > > > > threads
> > > > > >> have
> > > > > >> > to be inside the Authorizer implementation since only it
> > knows
> > > > > when to
> > > > > >> > complete a future? If so, it feels weird that all the
> logic
> > of
> > > > > request
> > > > > >> > handling post authorize() is now handled by external
> threads
> > > > > inside a
> > > > > >> > plugin. It's not clear how we can configure and monitor
> > them,
> > > > and
> > > > > how
> > > > > >> > throttling would work. Perhaps we can provide a bit more
> > > detail
> > > > > on the
> > > > > >> new
> > > > > >> > threading model.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> >
> > > > > >> > Jun
> > > > > >> >
> > > > > >> > On Wed, Sep 4, 2019 at 9:56 PM Ismael Juma <
> > ismael@juma.me.uk
> > > >
> > > > > wrote:
> > > > > >> >
> > > > > >> > > Hi Jun,
> > > > > >> > >
> > > > > >> > > I think it's important to discuss the pluggable API
> versus
> > > the
> > > > > calling
> > > > > >> > > implementation as two separate points.
> > > > > >> > >
> > > > > >> > > From an API perspective, are you suggesting that we
> should
> > > > tell
> > > > > users
> > > > > >> > that
> > > > > >> > > they cannot block in authorize? Or are you saying that
> > it's
> > > OK
> > > > > to
> > > > > >> block
> > > > > >> > on
> > > > > >> > > authorize on occasion and the recommendation would be
> for
> > > > > people to
> > > > > >> > > increase the number of threads in the request thread
> pool?
> > > > > >> > Architecturally,
> > > > > >> > > this feels wrong in my opinion.
> > > > > >> > >
> > > > > >> > > From the calling implementation perspective, you don't
> > need
> > > a
> > > > > >> callback.
> > > > > >> > You
> > > > > >> > > can basically say:
> > > > > >> > >
> > > > > >> > > authorize(...).map { result =>
> > > > > >> > > ...
> > > > > >> > > }
> > > > > >> > >
> > > > > >> > > And then have a common method take that Future and
> handle
> > it
> > > > > >> > synchronously
> > > > > >> > > if it's already complete or submit it to the purgatory
> if
> > > not.
> > > > > This is
> > > > > >> > > similar to how many modern async web libraries work.
> > > > > >> > >
> > > > > >> > > As Rajini said, this could be done later. The initial
> > > > > implementation
> > > > > >> > could
> > > > > >> > > simply do `toCompletableFuture().get()` to do it
> > > > synchronously.
> > > > > But it
> > > > > >> > > would mean that we could add this functionality without
> > > > > changing the
> > > > > >> > > pluggable interface.
> > > > > >> > >
> > > > > >> > > Ismael
> > > > > >> > >
> > > > > >> > > On Wed, Sep 4, 2019 at 5:29 AM Jun Rao <
> jun@confluent.io>
> > > > > wrote:
> > > > > >> > >
> > > > > >> > > > Hi, Rajini,
> > > > > >> > > >
> > > > > >> > > > Thanks for the KIP. I was concerned about #4 too. If
> we
> > > > > change the
> > > > > >> > > handling
> > > > > >> > > > of all requests to use an async authorize() api, will
> > that
> > > > > cause the
> > > > > >> > code
> > > > > >> > > > much harder to understand? There are quite a few
> > callbacks
> > > > > already.
> > > > > >> I
> > > > > >> > am
> > > > > >> > > > not sure that we want to introduce more of those. The
> > > > benefit
> > > > > from
> > > > > >> > async
> > > > > >> > > > authorize() api seems limited.
> > > > > >> > > >
> > > > > >> > > > Jun
> > > > > >> > > >
> > > > > >> > > > On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <
> > > > > >> rajinisivaram@gmail.com
> > > > > >> > >
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Hi Don,
> > > > > >> > > > >
> > > > > >> > > > > Thanks for your note.
> > > > > >> > > > >
> > > > > >> > > > > 1) The intention is to avoid blocking in the calling
> > > > > thread. We
> > > > > >> > already
> > > > > >> > > > > have several requests that are put into a purgatory
> > when
> > > > > waiting
> > > > > >> for
> > > > > >> > > > remote
> > > > > >> > > > > communication, for example produce request waiting
> for
> > > > > >> replication.
> > > > > >> > > Since
> > > > > >> > > > > we have a limited number of request threads in the
> > > broker,
> > > > > we
> > > > > >> want to
> > > > > >> > > > make
> > > > > >> > > > > progress with other requests, while one is waiting
> on
> > > any
> > > > > form of
> > > > > >> > > remote
> > > > > >> > > > > communication.
> > > > > >> > > > >
> > > > > >> > > > > 2) Async management calls will be useful with the
> > > default
> > > > > >> authorizer
> > > > > >> > > when
> > > > > >> > > > > KIP-500 removes ZK and we rely on Kafka instead. Our
> > > > current
> > > > > >> ZK-based
> > > > > >> > > > > implementation as well as any custom implementations
> > > that
> > > > > don't
> > > > > >> want
> > > > > >> > to
> > > > > >> > > > be
> > > > > >> > > > > async will just need to return a sync'ed value. So
> > > instead
> > > > > of
> > > > > >> > > returning `
> > > > > >> > > > > value`, the code would just return
> > > > > >> > > > > `CompletableFuture.completedFuture(value)
> > > > > >> > > > > `. So it would be just a single line change in the
> > > > > implementation
> > > > > >> > with
> > > > > >> > > > the
> > > > > >> > > > > new API. The caller would treat completedFuture
> > exactly
> > > as
> > > > > it does
> > > > > >> > > today,
> > > > > >> > > > > processing the request synchronously without using a
> > > > > purgatory.
> > > > > >> > > > >
> > > > > >> > > > > 3) For implementations that return a completedFuture
> > as
> > > > > described
> > > > > >> in
> > > > > >> > > 2),
> > > > > >> > > > > the behaviour would remain exactly the same. No
> > > additional
> > > > > >> threads or
> > > > > >> > > > > purgatory will be used for this case. So there would
> > be
> > > no
> > > > > >> > performance
> > > > > >> > > > > penalty. For implementations that return a future
> that
> > > is
> > > > > not
> > > > > >> > complete,
> > > > > >> > > > we
> > > > > >> > > > > prioritise running more requests concurrently. So
> in a
> > > > > deployment
> > > > > >> > with
> > > > > >> > > a
> > > > > >> > > > > large number of clients, we would improve
> performance
> > by
> > > > > allowing
> > > > > >> > other
> > > > > >> > > > > requests to be processed on the request threads
> while
> > > some
> > > > > are
> > > > > >> > waiting
> > > > > >> > > > for
> > > > > >> > > > > authorization metadata.
> > > > > >> > > > >
> > > > > >> > > > > 4) I was concerned about this too. The goal is to
> make
> > > the
> > > > > API
> > > > > >> > flexible
> > > > > >> > > > > enough to handle large scale deployments in future
> > when
> > > > > caching
> > > > > >> all
> > > > > >> > > > > authorization metadata in each broker is not viable.
> > > Using
> > > > > an
> > > > > >> async
> > > > > >> > API
> > > > > >> > > > > that returns CompletionStage, the caller has the
> > option
> > > to
> > > > > handle
> > > > > >> the
> > > > > >> > > > > result synchronously or asynchronously, so we don't
> > > > > necessarily
> > > > > >> need
> > > > > >> > to
> > > > > >> > > > > update the calling code right away. Custom
> authorizers
> > > > > using the
> > > > > >> > async
> > > > > >> > > > API
> > > > > >> > > > > have full control over whether authorization is
> > > performed
> > > > > in-line
> > > > > >> > since
> > > > > >> > > > > completedFuture will always be handled
> synchronously.
> > We
> > > > do
> > > > > need
> > > > > >> to
> > > > > >> > > > update
> > > > > >> > > > > KafkaApis to take advantage of the asynchronous API
> to
> > > > > improve
> > > > > >> scale.
> > > > > >> > > > Even
> > > > > >> > > > > though this is a big change, since we will be doing
> > the
> > > > > same for
> > > > > >> all
> > > > > >> > > > > requests, it shouldn't be too hard to maintain since
> > the
> > > > > same
> > > > > >> pattern
> > > > > >> > > > will
> > > > > >> > > > > be used for all requests.
> > > > > >> > > > >
> > > > > >> > > > > Regards,
> > > > > >> > > > >
> > > > > >> > > > > Rajini
> > > > > >> > > > >
> > > > > >> > > > > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <
> > > > > bosco@apache.org
> > > > > >> >
> > > > > >> > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Hi Rajini
> > > > > >> > > > > >
> > > > > >> > > > > > Help me understand this a bit more.
> > > > > >> > > > > >
> > > > > >> > > > > > 1. For all practical purpose, without
> authorization
> > > you
> > > > > can't
> > > > > >> go to
> > > > > >> > > the
> > > > > >> > > > > > next step. The calling code needs to block anyway.
> > So
> > > > > should we
> > > > > >> > just
> > > > > >> > > > let
> > > > > >> > > > > > the implementation code do the async part?
> > > > > >> > > > > > 2. If you feel management calls need to be async,
> > then
> > > > we
> > > > > should
> > > > > >> > > > consider
> > > > > >> > > > > > another set of async APIs. I don't feel we should
> > > > > complicate it
> > > > > >> > > > further (
> > > > > >> > > > > > 3. Another concern I have is wrt performance.
> Kafka
> > > has
> > > > > been
> > > > > >> built
> > > > > >> > to
> > > > > >> > > > > > handle 1000s per second requests. Not sure whether
> > > > making
> > > > > it
> > > > > >> async
> > > > > >> > > will
> > > > > >> > > > > add
> > > > > >> > > > > > any unnecessary overhead.
> > > > > >> > > > > > 4. How much complication would this add on the
> > calling
> > > > > side?
> > > > > >> And is
> > > > > >> > > it
> > > > > >> > > > > > worth it?
> > > > > >> > > > > >
> > > > > >> > > > > > Thanks
> > > > > >> > > > > >
> > > > > >> > > > > > Bosco
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > On 9/3/19, 8:50 AM, "Rajini Sivaram" <
> > > > > rajinisivaram@gmail.com>
> > > > > >> > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > Hi all,
> > > > > >> > > > > >
> > > > > >> > > > > > Ismael brought up a point that it will be good
> > to
> > > > > make the
> > > > > >> > > > Authorizer
> > > > > >> > > > > > interface asynchronous to avoid blocking
> request
> > > > > threads
> > > > > >> during
> > > > > >> > > > > remote
> > > > > >> > > > > > operations.
> > > > > >> > > > > >
> > > > > >> > > > > > 1) Since we want to support different backends
> > for
> > > > > >> > authorization
> > > > > >> > > > > > metadata,
> > > > > >> > > > > > making createAcls() and deleteAcls()
> > asynchronous
> > > > > makes
> > > > > >> sense
> > > > > >> > > since
> > > > > >> > > > > > these
> > > > > >> > > > > > always involve remote operations. When KIP-500
> > > > removes
> > > > > >> > ZooKeeper,
> > > > > >> > > > we
> > > > > >> > > > > > would
> > > > > >> > > > > > want to move ACLs to Kafka and async updates
> > will
> > > > > avoid
> > > > > >> > > unnecessary
> > > > > >> > > > > > blocking.
> > > > > >> > > > > > 2) For authorize() method, we currently use
> > cached
> > > > > ACLs in
> > > > > >> the
> > > > > >> > > > > built-in
> > > > > >> > > > > > authorizers, so synchronous authorise
> operations
> > > > work
> > > > > well
> > > > > >> now.
> > > > > >> > > But
> > > > > >> > > > > > async
> > > > > >> > > > > > authorize() would support this scenario as
> well
> > as
> > > > > >> authorizers
> > > > > >> > in
> > > > > >> > > > > large
> > > > > >> > > > > > organisations where an LRU cache would enable
> a
> > > > > smaller
> > > > > >> cache
> > > > > >> > > even
> > > > > >> > > > > > when the
> > > > > >> > > > > > backend holds a large amount of ACLs for
> > > > infrequently
> > > > > used
> > > > > >> > > > resources
> > > > > >> > > > > or
> > > > > >> > > > > > users who don't use the system frequently.
> > > > > >> > > > > >
> > > > > >> > > > > > For both cases, the built-in authorizer will
> > > > continue
> > > > > to be
> > > > > >> > > > > > synchronous,
> > > > > >> > > > > > using CompletableFuture.completedFuture() to
> > > return
> > > > > the
> > > > > >> actual
> > > > > >> > > > > result.
> > > > > >> > > > > > But
> > > > > >> > > > > > async API will make custom authorizer
> > > > implementations
> > > > > more
> > > > > >> > > > flexible.
> > > > > >> > > > > I
> > > > > >> > > > > > would like to know if there are any concerns
> > with
> > > > > these
> > > > > >> changes
> > > > > >> > > > > before
> > > > > >> > > > > > updating the KIP.
> > > > > >> > > > > >
> > > > > >> > > > > > *Proposed API:*
> > > > > >> > > > > > public interface Authorizer extends
> > Configurable,
> > > > > Closeable
> > > > > >> {
> > > > > >> > > > > >
> > > > > >> > > > > > Map<Endpoint, CompletionStage<Void>>
> > > > > >> > > start(AuthorizerServerInfo
> > > > > >> > > > > > serverInfo);
> > > > > >> > > > > > List<CompletionStage<AuthorizationResult>>
> > > > > >> > > > > > authorize(AuthorizableRequestContext
> > > requestContext,
> > > > > >> > List<Action>
> > > > > >> > > > > > actions);
> > > > > >> > > > > > List<CompletionStage<AclCreateResult>>
> > > > > >> > > > > > createAcls(AuthorizableRequestContext
> > > > requestContext,
> > > > > >> > > > > List<AclBinding>
> > > > > >> > > > > > aclBindings);
> > > > > >> > > > > > List<CompletionStage<AclDeleteResult>>
> > > > > >> > > > > > deleteAcls(AuthorizableRequestContext
> > > > requestContext,
> > > > > >> > > > > > List<AclBindingFilter> aclBindingFilters);
> > > > > >> > > > > > CompletionStage<Collection<AclBinding>>
> > > > > >> > acls(AclBindingFilter
> > > > > >> > > > > > filter);
> > > > > >> > > > > > }
> > > > > >> > > > > >
> > > > > >> > > > > >
> > > > > >> > > > > > Thank you,
> > > > > >> > > > > >
> > > > > >> > > > > > Rajini
> > > > > >> > > > > >
> > > > > >> > > > > > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco
> Durai
> > <
> > > > > >> > > bosco@apache.org>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Hi Rajini
> > > > > >> > > > > > >
> > > > > >> > > > > > > Thanks for clarifying. I am good for now.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Regards
> > > > > >> > > > > > >
> > > > > >> > > > > > > Bosco
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <
> > > > > >> > > rajinisivaram@gmail.com
> > > > > >> > > > >
> > > > > >> > > > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > Hi Don,
> > > > > >> > > > > > >
> > > > > >> > > > > > > That should be fine. I guess Ranger
> loads
> > > > > policies
> > > > > >> from
> > > > > >> > the
> > > > > >> > > > > > database
> > > > > >> > > > > > > synchronously when the authorizer is
> > > > configured,
> > > > > >> similar
> > > > > >> > to
> > > > > >> > > > > > > SimpleAclAuthorizer loading from
> > ZooKeeper.
> > > > > Ranger can
> > > > > >> > > > continue
> > > > > >> > > > > > to load
> > > > > >> > > > > > > synchronously from `configure()` or
> > > `start()`
> > > > > and
> > > > > >> return
> > > > > >> > an
> > > > > >> > > > > > empty map
> > > > > >> > > > > > > from
> > > > > >> > > > > > > `start()`. That would retain the
> existing
> > > > > behaviour..
> > > > > >> > When
> > > > > >> > > > the
> > > > > >> > > > > > same
> > > > > >> > > > > > > database stores policies for all
> listeners
> > > and
> > > > > the
> > > > > >> > policies
> > > > > >> > > > are
> > > > > >> > > > > > not
> > > > > >> > > > > > > stored
> > > > > >> > > > > > > in Kafka, there is no value in making
> the
> > > load
> > > > > >> > > asynchronous.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Regards,
> > > > > >> > > > > > >
> > > > > >> > > > > > > Rajini
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Fri, Aug 16, 2019 at 6:43 PM Don
> Bosco
> > > > Durai
> > > > > <
> > > > > >> > > > > > bosco@apache.org>
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Hi Rajini
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Assuming this doesn't affect custom
> > > plugins,
> > > > > I don't
> > > > > >> > have
> > > > > >> > > > any
> > > > > >> > > > > > > concerns
> > > > > >> > > > > > > > regarding this.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > I do have one question regarding:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > "For authorizers that don’t store
> > metadata
> > > > in
> > > > > >> > ZooKeeper,
> > > > > >> > > > > > ensure that
> > > > > >> > > > > > > > authorizer metadata for each listener
> is
> > > > > available
> > > > > >> > before
> > > > > >> > > > > > starting
> > > > > >> > > > > > > up the
> > > > > >> > > > > > > > listener. This enables different
> > > > authorization
> > > > > >> metadata
> > > > > >> > > > > stores
> > > > > >> > > > > > for
> > > > > >> > > > > > > > different listeners."
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Since Ranger uses its own database for
> > > > storing
> > > > > >> > policies,
> > > > > >> > > do
> > > > > >> > > > > you
> > > > > >> > > > > > > anticipate
> > > > > >> > > > > > > > any issues?
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Thanks
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Bosco
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On 8/16/19, 6:49 AM, "Rajini
> Sivaram" <
> > > > > >> > > > > > rajinisivaram@gmail.com>
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Hi all,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > I made another change to the KIP.
> > The
> > > > KIP
> > > > > was
> > > > > >> > > > originally
> > > > > >> > > > > > > proposing to
> > > > > >> > > > > > > > extend SimpleAclAuthorizer to also
> > > > > implement the
> > > > > >> > new
> > > > > >> > > > API
> > > > > >> > > > > > (in
> > > > > >> > > > > > > addition
> > > > > >> > > > > > > > to
> > > > > >> > > > > > > > the existing API). But since we
> use
> > > the
> > > > > new API
> > > > > >> > when
> > > > > >> > > > > > available,
> > > > > >> > > > > > > this
> > > > > >> > > > > > > > can
> > > > > >> > > > > > > > break custom authorizers that
> extend
> > > > this
> > > > > class
> > > > > >> and
> > > > > >> > > > > > override
> > > > > >> > > > > > > specific
> > > > > >> > > > > > > > methods of the old API. To avoid
> > > > breaking
> > > > > any
> > > > > >> > > existing
> > > > > >> > > > > > custom
> > > > > >> > > > > > > > implementations that extend this
> > > class,
> > > > > >> > particularly
> > > > > >> > > > > > because it
> > > > > >> > > > > > > is in
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > public package
> kafka.security.auth,
> > > the
> > > > > KIP now
> > > > > >> > > > proposes
> > > > > >> > > > > to
> > > > > >> > > > > > > retain the
> > > > > >> > > > > > > > old
> > > > > >> > > > > > > > authorizer as-is.
> > SimpleAclAuthorizer
> > > > > will
> > > > > >> > continue
> > > > > >> > > to
> > > > > >> > > > > > > implement the
> > > > > >> > > > > > > > old
> > > > > >> > > > > > > > API, but will be deprecated. A new
> > > > > authorizer
> > > > > >> > > > > > implementation
> > > > > >> > > > > > > >
> > > kafka.security.authorizer.AclAuthorizer
> > > > > will be
> > > > > >> > added
> > > > > >> > > > for
> > > > > >> > > > > > the
> > > > > >> > > > > > > new API
> > > > > >> > > > > > > > (this
> > > > > >> > > > > > > > will not be in the public
> package).
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Please let me know if you have any
> > > > > concerns.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Regards,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Rajini
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Fri, Aug 16, 2019 at 8:48 AM
> > Rajini
> > > > > Sivaram <
> > > > > >> > > > > > > > rajinisivaram@gmail.com>
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > Thanks Colin.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > If there are no other concerns,
> I
> > > will
> > > > > start
> > > > > >> vote
> > > > > >> > > > later
> > > > > >> > > > > > today.
> > > > > >> > > > > > > Many
> > > > > >> > > > > > > > thanks
> > > > > >> > > > > > > > > to every one for the feedback.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Regards,
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Rajini
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Thu, Aug 15, 2019 at 11:57 PM
> > > Colin
> > > > > McCabe
> > > > > >> <
> > > > > >> > > > > > > cmccabe@apache.org>
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >> Thanks, Rajini. It looks good
> to
> > > me.
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >> best,
> > > > > >> > > > > > > > >> Colin
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >>
> > > > > >> > > > > > > > >> On Thu, Aug 15, 2019, at 11:37,
> > > > Rajini
> > > > > >> Sivaram
> > > > > >> > > > wrote:
> > > > > >> > > > > > > > >> > Hi Colin,
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >> > Thanks for the review. I have
> > > > > updated the
> > > > > >> KIP
> > > > > >> > to
> > > > > >> > > > > move
> > > > > >> > > > > > the
> > > > > >> > > > > > > > interfaces for
> > > > > >> > > > > > > > >> > request context and server
> info
> > > to
> > > > > the
> > > > > >> > > authorizer
> > > > > >> > > > > > package.
> > > > > >> > > > > > > These
> > > > > >> > > > > > > > are now
> > > > > >> > > > > > > > >> > called
> > AuthorizableRequestContext
> > > > and
> > > > > >> > > > > > AuthorizerServerInfo.
> > > > > >> > > > > > > > Endpoint is
> > > > > >> > > > > > > > >> now
> > > > > >> > > > > > > > >> > a class in
> > > org.apache.kafka.common
> > > > > to make
> > > > > >> it
> > > > > >> > > > > reusable
> > > > > >> > > > > > > since we
> > > > > >> > > > > > > > already
> > > > > >> > > > > > > > >> > have multiple implementations
> > of
> > > > it.
> > > > > I have
> > > > > >> > > > removed
> > > > > >> > > > > > > requestName
> > > > > >> > > > > > > > from the
> > > > > >> > > > > > > > >> > request context interface
> since
> > > > > authorizers
> > > > > >> > can
> > > > > >> > > > > > distinguish
> > > > > >> > > > > > > > follower
> > > > > >> > > > > > > > >> fetch
> > > > > >> > > > > > > > >> > and consumer fetch from the
> > > > > operation being
> > > > > >> > > > > > authorized. So
> > > > > >> > > > > > > 16-bit
> > > > > >> > > > > > > > >> request
> > > > > >> > > > > > > > >> > type should be sufficient for
> > > audit
> > > > > >> logging.
> > > > > >> > > Also
> > > > > >> > > > > > replaced
> > > > > >> > > > > > > > AuditFlag
> > > > > >> > > > > > > > >> with
> > > > > >> > > > > > > > >> > two booleans as you
> suggested.
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >> > Can you take another look and
> > see
> > > > if
> > > > > the
> > > > > >> KIP
> > > > > >> > is
> > > > > >> > > > > ready
> > > > > >> > > > > > for
> > > > > >> > > > > > > voting?
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >> > Thanks for all your help!
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >> > Regards,
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >> > Rajini
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >> > On Wed, Aug 14, 2019 at 8:59
> PM
> > > > Colin
> > > > > >> McCabe <
> > > > > >> > > > > > > cmccabe@apache.org>
> > > > > >> > > > > > > > >> wrote:
> > > > > >> > > > > > > > >> >
> > > > > >> > > > > > > > >> > > Hi Rajini,
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > > I think it would be good to
> > > > rename
> > > > > >> > > > > > KafkaRequestContext to
> > > > > >> > > > > > > > something
> > > > > >> > > > > > > > >> like
> > > > > >> > > > > > > > >> > > AuthorizableRequestContext,
> > and
> > > > > put it in
> > > > > >> > the
> > > > > >> > > > > > > > >> > >
> > > > org.apache.kafka.server.authorizer
> > > > > >> > namespace.
> > > > > >> > > > If
> > > > > >> > > > > > we put
> > > > > >> > > > > > > it in
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > >> > > org.apache.kafka.common
> > > > namespace,
> > > > > then
> > > > > >> it's
> > > > > >> > > not
> > > > > >> > > > > > really
> > > > > >> > > > > > > clear
> > > > > >> > > > > > > > that
> > > > > >> > > > > > > > >> it's
> > > > > >> > > > > > > > >> > > part of the Authorizer API.
> > > > Since
> > > > > this
> > > > > >> > class
> > > > > >> > > is
> > > > > >> > > > > > purely an
> > > > > >> > > > > > > > interface,
> > > > > >> > > > > > > > >> with
> > > > > >> > > > > > > > >> > > no concrete implementation
> of
> > > > > anything,
> > > > > >> > > there's
> > > > > >> > > > > > nothing
> > > > > >> > > > > > > common
> > > > > >> > > > > > > > to
> > > > > >> > > > > > > > >> really
> > > > > >> > > > > > > > >> > > reuse in any case. We
> > > definitely
> > > > > don't
> > > > > >> want
> > > > > >> > > > > > someone to
> > > > > >> > > > > > > > accidentally
> > > > > >> > > > > > > > >> add or
> > > > > >> > > > > > > > >> > > remove methods because they
> > > think
> > > > > this is
> > > > > >> > just
> > > > > >> > > > > > another
> > > > > >> > > > > > > internal
> > > > > >> > > > > > > > class
> > > > > >> > > > > > > > >> used
> > > > > >> > > > > > > > >> > > for requests.
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > > The BrokerInfo class is a
> > nice
> > > > > >> improvement.
> > > > > >> > > It
> > > > > >> > > > > > looks
> > > > > >> > > > > > > like it
> > > > > >> > > > > > > > will be
> > > > > >> > > > > > > > >> > > useful for passing in
> > > information
> > > > > about
> > > > > >> the
> > > > > >> > > > > context
> > > > > >> > > > > > we're
> > > > > >> > > > > > > > running
> > > > > >> > > > > > > > >> in. It
> > > > > >> > > > > > > > >> > > would be nice to call this
> > > class
> > > > > >> ServerInfo
> > > > > >> > > > rather
> > > > > >> > > > > > than
> > > > > >> > > > > > > > BrokerInfo,
> > > > > >> > > > > > > > >> since
> > > > > >> > > > > > > > >> > > we will want to run the
> > > > authorizer
> > > > > on
> > > > > >> > > > controllers
> > > > > >> > > > > > as well
> > > > > >> > > > > > > as on
> > > > > >> > > > > > > > >> brokers,
> > > > > >> > > > > > > > >> > > and the controller may run
> > as a
> > > > > separate
> > > > > >> > > process
> > > > > >> > > > > > post
> > > > > >> > > > > > > KIP-500.
> > > > > >> > > > > > > > I also
> > > > > >> > > > > > > > >> > > think that this class
> should
> > be
> > > > in
> > > > > the
> > > > > >> > > > > > > > >>
> > org.apache.kafka.server.authorizer
> > > > > >> > > > > > > > >> > > namespace. Again, it is an
> > > > > interface,
> > > > > >> not a
> > > > > >> > > > > > concrete
> > > > > >> > > > > > > > implementation,
> > > > > >> > > > > > > > >> and
> > > > > >> > > > > > > > >> > > it's an interface that is
> > very
> > > > > >> specifically
> > > > > >> > > for
> > > > > >> > > > > the
> > > > > >> > > > > > > authorizer.
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > > I agree that we probably
> > don't
> > > > have
> > > > > >> enough
> > > > > >> > > > > > information
> > > > > >> > > > > > > > preserved for
> > > > > >> > > > > > > > >> > > requests currently to
> always
> > > know
> > > > > what
> > > > > >> > entity
> > > > > >> > > > made
> > > > > >> > > > > > them.
> > > > > >> > > > > > > So we
> > > > > >> > > > > > > > can
> > > > > >> > > > > > > > >> leave
> > > > > >> > > > > > > > >> > > that out for now (except in
> > the
> > > > > special
> > > > > >> case
> > > > > >> > > of
> > > > > >> > > > > > Fetch).
> > > > > >> > > > > > > > Perhaps we
> > > > > >> > > > > > > > >> can add
> > > > > >> > > > > > > > >> > > this later if it's needed.
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > > I understand the intention
> > > behind
> > > > > >> > > > > AuthorizationMode
> > > > > >> > > > > > > (which is
> > > > > >> > > > > > > > now
> > > > > >> > > > > > > > >> called
> > > > > >> > > > > > > > >> > > AuditFlag in the latest
> > > > > revision). But
> > > > > >> it
> > > > > >> > > still
> > > > > >> > > > > > feels
> > > > > >> > > > > > > > complex. What
> > > > > >> > > > > > > > >> if we
> > > > > >> > > > > > > > >> > > just had two booleans in
> > > Action:
> > > > > >> > logSuccesses
> > > > > >> > > > and
> > > > > >> > > > > > > logFailures?
> > > > > >> > > > > > > > That
> > > > > >> > > > > > > > >> seems
> > > > > >> > > > > > > > >> > > to cover all the cases
> here.
> > > > > >> > > > MANDATORY_AUTHORIZE
> > > > > >> > > > > =
> > > > > >> > > > > > true,
> > > > > >> > > > > > > true.
> > > > > >> > > > > > > > >> > > OPTIONAL_AUTHORIZE = true,
> > > false.
> > > > > >> FILTER =
> > > > > >> > > > true,
> > > > > >> > > > > > false.
> > > > > >> > > > > > > > >> LIST_AUTHORIZED =
> > > > > >> > > > > > > > >> > > false, false. Would there
> be
> > > > > anything
> > > > > >> lost
> > > > > >> > > > versus
> > > > > >> > > > > > having
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > enum?
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > > best,
> > > > > >> > > > > > > > >> > > Colin
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > >
> > > > > >> > > > > > > > >> > > On Wed, Aug 14, 2019, at
> > 06:29,
> > > > > Mickael
> > > > > >> > Maison
> > > > > >> > > > > > wrote:
> > > > > >> > > > > > > > >> > > > Hi Rajini,
> > > > > >> > > > > > > > >> > > >
> > > > > >> > > > > > > > >> > > > Thanks for the KIP!
> > > > > >> > > > > > > > >> > > > I really like that
> > > authorize()
> > > > > will be
> > > > > >> > able
> > > > > >> > > to
> > > > > >> > > > > > take a
> > > > > >> > > > > > > batch of
> > > > > >> > > > > > > > >> > > > requests, this will speed
> > up
> > > > many
> > > > > >> > > > > implementations!
> > > > > >> > > > > > > > >> > > >
> > > > > >> > > > > > > > >> > > > On Tue, Aug 13, 2019 at
> > 5:57
> > > PM
> > > > > Rajini
> > > > > >> > > > Sivaram <
> > > > > >> > > > > > > > >> rajinisivaram@gmail.com>
> > > > > >> > > > > > > > >> > > wrote:
> > > > > >> > > > > > > > >> > > > >
> > > > > >> > > > > > > > >> > > > > Thanks David! I have
> > fixed
> > > > the
> > > > > typo.
> > > > > >> > > > > > > > >> > > > >
> > > > > >> > > > > > > > >> > > > > Also made a couple of
> > > changes
> > > > > to make
> > > > > >> > the
> > > > > >> > > > > > context
> > > > > >> > > > > > > > interfaces more
> > > > > >> > > > > > > > >> > > generic.
> > > > > >> > > > > > > > >> > > > > KafkaRequestContext now
> > > > > returns the
> > > > > >> > 16-bit
> > > > > >> > > > API
> > > > > >> > > > > > key as
> > > > > >> > > > > > > Colin
> > > > > >> > > > > > > > >> suggested
> > > > > >> > > > > > > > >> > > as
> > > > > >> > > > > > > > >> > > > > well as the friendly
> name
> > > > used
> > > > > in
> > > > > >> > metrics
> > > > > >> > > > > which
> > > > > >> > > > > > are
> > > > > >> > > > > > > useful
> > > > > >> > > > > > > > in
> > > > > >> > > > > > > > >> audit
> > > > > >> > > > > > > > >> > > logs.
> > > > > >> > > > > > > > >> > > > > `Authorizer#start` is
> now
> > > > > provided a
> > > > > >> > > generic
> > > > > >> > > > > > > `BrokerInfo`
> > > > > >> > > > > > > > >> interface
> > > > > >> > > > > > > > >> > > that
> > > > > >> > > > > > > > >> > > > > gives cluster id,
> broker
> > id
> > > > and
> > > > > >> endpoint
> > > > > >> > > > > > information.
> > > > > >> > > > > > > The
> > > > > >> > > > > > > > generic
> > > > > >> > > > > > > > >> > > interface
> > > > > >> > > > > > > > >> > > > > can potentially be used
> > in
> > > > > other
> > > > > >> broker
> > > > > >> > > > > plugins
> > > > > >> > > > > > in
> > > > > >> > > > > > > future
> > > > > >> > > > > > > > and
> > > > > >> > > > > > > > >> provides
> > > > > >> > > > > > > > >> > > > > dynamically generated
> > > configs
> > > > > like
> > > > > >> > broker
> > > > > >> > > id
> > > > > >> > > > > > and ports
> > > > > >> > > > > > > > which are
> > > > > >> > > > > > > > >> > > currently
> > > > > >> > > > > > > > >> > > > > not available to
> plugins
> > > > > unless these
> > > > > >> > > > configs
> > > > > >> > > > > > are
> > > > > >> > > > > > > statically
> > > > > >> > > > > > > > >> > > configured.
> > > > > >> > > > > > > > >> > > > > Please let me know if
> > there
> > > > > are any
> > > > > >> > > > concerns.
> > > > > >> > > > > > > > >> > > > >
> > > > > >> > > > > > > > >> > > > > Regards,
> > > > > >> > > > > > > > >> > > > >
> > > > > >> > > > > > > > >> > > > > Rajini
> > > > > >> > > > > > > > >> > > > >
> > > > > >> > > > > > > > >> > > > > On Tue, Aug 13, 2019 at
> > > 4:30
> > > > > PM David
> > > > > >> > > Jacot
> > > > > >> > > > <
> > > > > >> > > > > > > > djacot@confluent.io>
> > > > > >> > > > > > > > >> > > wrote:
> > > > > >> > > > > > > > >> > > > >
> > > > > >> > > > > > > > >> > > > > > Hi Rajini,
> > > > > >> > > > > > > > >> > > > > >
> > > > > >> > > > > > > > >> > > > > > Thank you for the
> > update!
> > > > It
> > > > > looks
> > > > > >> > good
> > > > > >> > > to
> > > > > >> > > > > me.
> > > > > >> > > > > > > There is a
> > > > > >> > > > > > > > typo
> > > > > >> > > > > > > > >> in the
> > > > > >> > > > > > > > >> > > > > > `AuditFlag` enum:
> > > > > >> > `MANDATORY_AUTHOEIZE`
> > > > > >> > > ->
> > > > > >> > > > > > > > >> `MANDATORY_AUTHORIZE`.
> > > > > >> > > > > > > > >> > > > > >
> > > > > >> > > > > > > > >> > > > > > Regards,
> > > > > >> > > > > > > > >> > > > > > David
> > > > > >> > > > > > > > >> > > > > >
> > > > > >> > > > > > > > >> > > > > > On Mon, Aug 12, 2019
> at
> > > > 2:54
> > > > > PM
> > > > > >> Rajini
> > > > > >> > > > > > Sivaram <
> > > > > >> > > > > > > > >> > > rajinisivaram@gmail.com>
> > > > > >> > > > > > > > >> > > > > > wrote:
> > > > > >> > > > > > > > >> > > > > >
> > > > > >> > > > > > > > >> > > > > > > Hi David,
> > > > > >> > > > > > > > >> > > > > > >
> > > > > >> > > > > > > > >> > > > > > > Thanks for
> reviewing
> > > the
> > > > > KIP!
> > > > > >> Since
> > > > > >> > > > > > questions
> > > > > >> > > > > > > about
> > > > > >> > > > > > > > >> `authorization
> > > > > >> > > > > > > > >> > > mode`
> > > > > >> > > > > > > > >> > > > > > > and `count` have
> come
> > > up
> > > > > multiple
> > > > > >> > > > times, I
> > > > > >> > > > > > have
> > > > > >> > > > > > > renamed
> > > > > >> > > > > > > > both.
> > > > > >> > > > > > > > >> > > > > > >
> > > > > >> > > > > > > > >> > > > > > > 1) Renamed `count`
> to
> > > > > >> > > > > > `resourceReferenceCount`.
> > > > > >> > > > > > > It is
> > > > > >> > > > > > > > the
> > > > > >> > > > > > > > >> number
> > > > > >> > > > > > > > >> > > of times
> > > > > >> > > > > > > > >> > > > > > > the resource being
> > > > > authorized is
> > > > > >> > > > > referenced
> > > > > >> > > > > > > within the
> > > > > >> > > > > > > > >> request.
> > > > > >> > > > > > > > >> > > > > > >
> > > > > >> > > > > > > > >> > > > > > > 2) Renamed
> > > > > `AuthorizationMode` to
> > > > > >> > > > > > `AuditFlag`. It
> > > > > >> > > > > > > is
> > > > > >> > > > > > > > provided
> > > > > >> > > > > > > > >> to
> > > > > >> > > > > > > > >> > > improve
> > > > > >> > > > > > > > >> > > > > > > audit logging in
> the
> > > > > authorizer.
> > > > > >> The
> > > > > >> > > > enum
> > > > > >> > > > > > values
> > > > > >> > > > > > > have
> > > > > >> > > > > > > > javadoc
> > > > > >> > > > > > > > >> which
> > > > > >> > > > > > > > >> > > > > > > indicate how the
> > > > > authorization
> > > > > >> > result
> > > > > >> > > is
> > > > > >> > > > > > used in
> > > > > >> > > > > > > each
> > > > > >> > > > > > > > of the
> > > > > >> > > > > > > > >> modes
> > > > > >> > > > > > > > >> > > to
> > > > > >> > > > > > > > >> > > > > > > enable authorizers
> to
> > > log
> > > > > audit
> > > > > >> > > messages
> > > > > >> > > > > at
> > > > > >> > > > > > the
> > > > > >> > > > > > > right
> > > > > >> > > > > > > > severity
> > > > > >> > > > > > > > >> > > level.
> > > > > >> > > > > > > > >> > > > > > >
> > > > > >> >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Thanks Ismael. That is good point. I have updated the javadoc.
On Sun, Sep 8, 2019 at 5:38 PM Ismael Juma <is...@juma.me.uk> wrote:
> Hi Rajini,
>
> Can you please update the javadoc for every sync method to be explicit
> about the fact that they should not block and the reasoning?
>
> Ismael
>
> On Sun, Sep 8, 2019, 9:21 AM Rajini Sivaram <ra...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > Thanks everyone for the very useful discussion. I have updated the KIP to
> > make *createAcls()* and *deleteAcls()* asynchronous. Also added a section
> > under `*Rejected Alternatives*` for async *authorize()*, which we can
> > revisit in future if requirement arises.
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Fri, Sep 6, 2019 at 9:17 PM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > It's not uncommon for people to do large numbers of admin operations
> per
> > > second. We've seen it many times in production. It's often a bug, but
> the
> > > system must cope with it without affecting other users of the same
> Kafka
> > > cluster.
> > >
> > > Ismael
> > >
> > > On Fri, Sep 6, 2019, 9:32 AM Don Bosco Durai <bo...@apache.org> wrote:
> > >
> > > > +1
> > > >
> > > > I might be wrong here, but here are few of my assumptions..
> > > >
> > > > 1. Unlike transactional services, in Kafka most of the users are
> > > > application users, so once the connection is established, then they
> are
> > > > going to be long running sessions. So the caching is more predictable
> > and
> > > > heavy lifting can be done during initial session setup and cache can
> be
> > > > frequently updated using background thread. KSQL might have a
> different
> > > > flow, but I don't have enough datapoints for it.
> > > > 2. For publish/consume authorization calls, providing async option
> will
> > > be
> > > > misleading for Kafka like load
> > > > 3. For admin calls, hopefully no one does hundreds of call in
> multiple
> > > > thread within few seconds. This would be a bad design.
> > > >
> > > > So I feel, we should be okay with sync calls for now. And consider
> > async
> > > > in the future if needed. At least in Apache Ranger implementation, we
> > > would
> > > > be okay using sync operation. I am not sure if any other plugin
> > > > implementation would benefit from async implementation.
> > > >
> > > > Thanks
> > > >
> > > > Bosco
> > > >
> > > > On 9/6/19, 8:45 AM, "Ismael Juma" <is...@juma.me.uk> wrote:
> > > >
> > > > One more thing: if people have strong opinions that authorize has
> > to
> > > be
> > > > sync, we could leave it like that for now. If needed, we can
> later
> > > add
> > > > an
> > > > authorizeAsync with another method returning a Boolean indicating
> > > that
> > > > it's
> > > > supported. That is, there is a path to evolve the interface (even
> > if
> > > a
> > > > bit
> > > > ugly).
> > > >
> > > > What about the other methods, is there consensus that they should
> > be
> > > > updated to return CompletionStage?
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Sep 6, 2019, 8:32 AM Ismael Juma <is...@juma.me.uk>
> > wrote:
> > > >
> > > > > I'm on the move, but what Rajini said seems reasonable. I don't
> > > think
> > > > > using SSDs solves the issue. They can still hang for seconds
> when
> > > > they
> > > > > fail. Also, many people may not have local SSDs available
> (remote
> > > > SSDs like
> > > > > EBS hang for tens of seconds when there are issues).
> > > > >
> > > > > We are currently vulnerable to all of these in the normal
> > > read/write
> > > > path,
> > > > > but it seems suboptimal to continue assuming non blocking
> > behavior
> > > > for
> > > > > things that actually do block.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Fri, Sep 6, 2019, 8:24 AM Rajini Sivaram <
> > > rajinisivaram@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hi Jun,
> > > > >>
> > > > >> For ACLs, the size is also dependent on the number of users.
> So
> > in
> > > > >> deployments with large number of users where some users are
> not
> > > > active, an
> > > > >> LRU cache may be useful. Agree that there may be other
> solutions
> > > > that help
> > > > >> avoid the requirement for an async API even for this case.
> > > > >>
> > > > >> I wasn't expecting our threading model to change very much.
> > Having
> > > > said
> > > > >> that, I haven't figured out how the purgatory can be adapted
> to
> > > work
> > > > >> efficiently for this. I was thinking:
> > > > >> 1) If authorization future is complete when returning from
> > > > authorize()
> > > > >> call, we invoke the method immediately on the request thread
> > > > >> 2) If future is not complete, the request goes into a
> purgatory
> > > > >> 3) When future completes, we mark the request ready to be run.
> > > This
> > > > can be
> > > > >> done on ForkJoinPool.commonPool or an executor. But we don't
> > > handle
> > > > the
> > > > >> request on that thread
> > > > >> 4) We handle the request on the request thread after 3).
> > > > >>
> > > > >> I totally agree that this adds complexity to the code. Will
> wait
> > > for
> > > > >> Ismael
> > > > >> to comment on whether he had a different model in mind.
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Rajini
> > > > >>
> > > > >>
> > > > >> On Thu, Sep 5, 2019 at 10:29 PM Jun Rao <ju...@confluent.io>
> > wrote:
> > > > >>
> > > > >> > Hi, Ismael,
> > > > >> >
> > > > >> > Yes, I agree that there are 2 separate discussion points,
> one
> > on
> > > > the API
> > > > >> > and the other on the implementation.
> > > > >> >
> > > > >> > First on the API, in general, my feeling is that the
> > authorize()
> > > > call is
> > > > >> > expected to be fast to reduce request latency. Adding an
> async
> > > API
> > > > >> could be
> > > > >> > mis-leading since it might give users the impression that it
> > > > would be
> > > > >> > arbitrarily long. In the short term, currently, we cache all
> > > > partition
> > > > >> > level metadata on every broker. The authorization metadata
> is
> > > > >> proportional
> > > > >> > to the number of topics, which typically is less in size. So
> > it
> > > > we can
> > > > >> > cache all partition level metadata, we should be able to
> cache
> > > all
> > > > >> > authorization metadata. Longer term, if we want to support
> > more
> > > > metadata
> > > > >> > beyond memory, another option is to put all metadata in an
> SSD
> > > > backed
> > > > >> > key/value store. The lookup time could still be under 1ms,
> > which
> > > > we
> > > > >> could
> > > > >> > potentially absorb in the request threads. So, I am not sure
> > > async
> > > > >> > authorize() api is strictly needed even in the long term.
> > > > >> >
> > > > >> > Second on the implementation. I was more concerned about the
> > > > impact on
> > > > >> our
> > > > >> > threading model. If we make authorize() async, some threads
> > have
> > > > to
> > > > >> > complete the future when it's ready. It seems to me that
> those
> > > > threads
> > > > >> have
> > > > >> > to be inside the Authorizer implementation since only it
> knows
> > > > when to
> > > > >> > complete a future? If so, it feels weird that all the logic
> of
> > > > request
> > > > >> > handling post authorize() is now handled by external threads
> > > > inside a
> > > > >> > plugin. It's not clear how we can configure and monitor
> them,
> > > and
> > > > how
> > > > >> > throttling would work. Perhaps we can provide a bit more
> > detail
> > > > on the
> > > > >> new
> > > > >> > threading model.
> > > > >> >
> > > > >> > Thanks,
> > > > >> >
> > > > >> > Jun
> > > > >> >
> > > > >> > On Wed, Sep 4, 2019 at 9:56 PM Ismael Juma <
> ismael@juma.me.uk
> > >
> > > > wrote:
> > > > >> >
> > > > >> > > Hi Jun,
> > > > >> > >
> > > > >> > > I think it's important to discuss the pluggable API versus
> > the
> > > > calling
> > > > >> > > implementation as two separate points.
> > > > >> > >
> > > > >> > > From an API perspective, are you suggesting that we should
> > > tell
> > > > users
> > > > >> > that
> > > > >> > > they cannot block in authorize? Or are you saying that
> it's
> > OK
> > > > to
> > > > >> block
> > > > >> > on
> > > > >> > > authorize on occasion and the recommendation would be for
> > > > people to
> > > > >> > > increase the number of threads in the request thread pool?
> > > > >> > Architecturally,
> > > > >> > > this feels wrong in my opinion.
> > > > >> > >
> > > > >> > > From the calling implementation perspective, you don't
> need
> > a
> > > > >> callback.
> > > > >> > You
> > > > >> > > can basically say:
> > > > >> > >
> > > > >> > > authorize(...).map { result =>
> > > > >> > > ...
> > > > >> > > }
> > > > >> > >
> > > > >> > > And then have a common method take that Future and handle
> it
> > > > >> > synchronously
> > > > >> > > if it's already complete or submit it to the purgatory if
> > not.
> > > > This is
> > > > >> > > similar to how many modern async web libraries work.
> > > > >> > >
> > > > >> > > As Rajini said, this could be done later. The initial
> > > > implementation
> > > > >> > could
> > > > >> > > simply do `toCompletableFuture().get()` to do it
> > > synchronously.
> > > > But it
> > > > >> > > would mean that we could add this functionality without
> > > > changing the
> > > > >> > > pluggable interface.
> > > > >> > >
> > > > >> > > Ismael
> > > > >> > >
> > > > >> > > On Wed, Sep 4, 2019 at 5:29 AM Jun Rao <ju...@confluent.io>
> > > > wrote:
> > > > >> > >
> > > > >> > > > Hi, Rajini,
> > > > >> > > >
> > > > >> > > > Thanks for the KIP. I was concerned about #4 too. If we
> > > > change the
> > > > >> > > handling
> > > > >> > > > of all requests to use an async authorize() api, will
> that
> > > > cause the
> > > > >> > code
> > > > >> > > > much harder to understand? There are quite a few
> callbacks
> > > > already.
> > > > >> I
> > > > >> > am
> > > > >> > > > not sure that we want to introduce more of those. The
> > > benefit
> > > > from
> > > > >> > async
> > > > >> > > > authorize() api seems limited.
> > > > >> > > >
> > > > >> > > > Jun
> > > > >> > > >
> > > > >> > > > On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <
> > > > >> rajinisivaram@gmail.com
> > > > >> > >
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Hi Don,
> > > > >> > > > >
> > > > >> > > > > Thanks for your note.
> > > > >> > > > >
> > > > >> > > > > 1) The intention is to avoid blocking in the calling
> > > > thread. We
> > > > >> > already
> > > > >> > > > > have several requests that are put into a purgatory
> when
> > > > waiting
> > > > >> for
> > > > >> > > > remote
> > > > >> > > > > communication, for example produce request waiting for
> > > > >> replication.
> > > > >> > > Since
> > > > >> > > > > we have a limited number of request threads in the
> > broker,
> > > > we
> > > > >> want to
> > > > >> > > > make
> > > > >> > > > > progress with other requests, while one is waiting on
> > any
> > > > form of
> > > > >> > > remote
> > > > >> > > > > communication.
> > > > >> > > > >
> > > > >> > > > > 2) Async management calls will be useful with the
> > default
> > > > >> authorizer
> > > > >> > > when
> > > > >> > > > > KIP-500 removes ZK and we rely on Kafka instead. Our
> > > current
> > > > >> ZK-based
> > > > >> > > > > implementation as well as any custom implementations
> > that
> > > > don't
> > > > >> want
> > > > >> > to
> > > > >> > > > be
> > > > >> > > > > async will just need to return a sync'ed value. So
> > instead
> > > > of
> > > > >> > > returning `
> > > > >> > > > > value`, the code would just return
> > > > >> > > > > `CompletableFuture.completedFuture(value)
> > > > >> > > > > `. So it would be just a single line change in the
> > > > implementation
> > > > >> > with
> > > > >> > > > the
> > > > >> > > > > new API. The caller would treat completedFuture
> exactly
> > as
> > > > it does
> > > > >> > > today,
> > > > >> > > > > processing the request synchronously without using a
> > > > purgatory.
> > > > >> > > > >
> > > > >> > > > > 3) For implementations that return a completedFuture
> as
> > > > described
> > > > >> in
> > > > >> > > 2),
> > > > >> > > > > the behaviour would remain exactly the same. No
> > additional
> > > > >> threads or
> > > > >> > > > > purgatory will be used for this case. So there would
> be
> > no
> > > > >> > performance
> > > > >> > > > > penalty. For implementations that return a future that
> > is
> > > > not
> > > > >> > complete,
> > > > >> > > > we
> > > > >> > > > > prioritise running more requests concurrently. So in a
> > > > deployment
> > > > >> > with
> > > > >> > > a
> > > > >> > > > > large number of clients, we would improve performance
> by
> > > > allowing
> > > > >> > other
> > > > >> > > > > requests to be processed on the request threads while
> > some
> > > > are
> > > > >> > waiting
> > > > >> > > > for
> > > > >> > > > > authorization metadata.
> > > > >> > > > >
> > > > >> > > > > 4) I was concerned about this too. The goal is to make
> > the
> > > > API
> > > > >> > flexible
> > > > >> > > > > enough to handle large scale deployments in future
> when
> > > > caching
> > > > >> all
> > > > >> > > > > authorization metadata in each broker is not viable.
> > Using
> > > > an
> > > > >> async
> > > > >> > API
> > > > >> > > > > that returns CompletionStage, the caller has the
> option
> > to
> > > > handle
> > > > >> the
> > > > >> > > > > result synchronously or asynchronously, so we don't
> > > > necessarily
> > > > >> need
> > > > >> > to
> > > > >> > > > > update the calling code right away. Custom authorizers
> > > > using the
> > > > >> > async
> > > > >> > > > API
> > > > >> > > > > have full control over whether authorization is
> > performed
> > > > in-line
> > > > >> > since
> > > > >> > > > > completedFuture will always be handled synchronously.
> We
> > > do
> > > > need
> > > > >> to
> > > > >> > > > update
> > > > >> > > > > KafkaApis to take advantage of the asynchronous API to
> > > > improve
> > > > >> scale.
> > > > >> > > > Even
> > > > >> > > > > though this is a big change, since we will be doing
> the
> > > > same for
> > > > >> all
> > > > >> > > > > requests, it shouldn't be too hard to maintain since
> the
> > > > same
> > > > >> pattern
> > > > >> > > > will
> > > > >> > > > > be used for all requests.
> > > > >> > > > >
> > > > >> > > > > Regards,
> > > > >> > > > >
> > > > >> > > > > Rajini
> > > > >> > > > >
> > > > >> > > > > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <
> > > > bosco@apache.org
> > > > >> >
> > > > >> > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Hi Rajini
> > > > >> > > > > >
> > > > >> > > > > > Help me understand this a bit more.
> > > > >> > > > > >
> > > > >> > > > > > 1. For all practical purpose, without authorization
> > you
> > > > can't
> > > > >> go to
> > > > >> > > the
> > > > >> > > > > > next step. The calling code needs to block anyway.
> So
> > > > should we
> > > > >> > just
> > > > >> > > > let
> > > > >> > > > > > the implementation code do the async part?
> > > > >> > > > > > 2. If you feel management calls need to be async,
> then
> > > we
> > > > should
> > > > >> > > > consider
> > > > >> > > > > > another set of async APIs. I don't feel we should
> > > > complicate it
> > > > >> > > > further (
> > > > >> > > > > > 3. Another concern I have is wrt performance. Kafka
> > has
> > > > been
> > > > >> built
> > > > >> > to
> > > > >> > > > > > handle 1000s per second requests. Not sure whether
> > > making
> > > > it
> > > > >> async
> > > > >> > > will
> > > > >> > > > > add
> > > > >> > > > > > any unnecessary overhead.
> > > > >> > > > > > 4. How much complication would this add on the
> calling
> > > > side?
> > > > >> And is
> > > > >> > > it
> > > > >> > > > > > worth it?
> > > > >> > > > > >
> > > > >> > > > > > Thanks
> > > > >> > > > > >
> > > > >> > > > > > Bosco
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > On 9/3/19, 8:50 AM, "Rajini Sivaram" <
> > > > rajinisivaram@gmail.com>
> > > > >> > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > Hi all,
> > > > >> > > > > >
> > > > >> > > > > > Ismael brought up a point that it will be good
> to
> > > > make the
> > > > >> > > > Authorizer
> > > > >> > > > > > interface asynchronous to avoid blocking request
> > > > threads
> > > > >> during
> > > > >> > > > > remote
> > > > >> > > > > > operations.
> > > > >> > > > > >
> > > > >> > > > > > 1) Since we want to support different backends
> for
> > > > >> > authorization
> > > > >> > > > > > metadata,
> > > > >> > > > > > making createAcls() and deleteAcls()
> asynchronous
> > > > makes
> > > > >> sense
> > > > >> > > since
> > > > >> > > > > > these
> > > > >> > > > > > always involve remote operations. When KIP-500
> > > removes
> > > > >> > ZooKeeper,
> > > > >> > > > we
> > > > >> > > > > > would
> > > > >> > > > > > want to move ACLs to Kafka and async updates
> will
> > > > avoid
> > > > >> > > unnecessary
> > > > >> > > > > > blocking.
> > > > >> > > > > > 2) For authorize() method, we currently use
> cached
> > > > ACLs in
> > > > >> the
> > > > >> > > > > built-in
> > > > >> > > > > > authorizers, so synchronous authorise operations
> > > work
> > > > well
> > > > >> now.
> > > > >> > > But
> > > > >> > > > > > async
> > > > >> > > > > > authorize() would support this scenario as well
> as
> > > > >> authorizers
> > > > >> > in
> > > > >> > > > > large
> > > > >> > > > > > organisations where an LRU cache would enable a
> > > > smaller
> > > > >> cache
> > > > >> > > even
> > > > >> > > > > > when the
> > > > >> > > > > > backend holds a large amount of ACLs for
> > > infrequently
> > > > used
> > > > >> > > > resources
> > > > >> > > > > or
> > > > >> > > > > > users who don't use the system frequently.
> > > > >> > > > > >
> > > > >> > > > > > For both cases, the built-in authorizer will
> > > continue
> > > > to be
> > > > >> > > > > > synchronous,
> > > > >> > > > > > using CompletableFuture.completedFuture() to
> > return
> > > > the
> > > > >> actual
> > > > >> > > > > result.
> > > > >> > > > > > But
> > > > >> > > > > > async API will make custom authorizer
> > > implementations
> > > > more
> > > > >> > > > flexible.
> > > > >> > > > > I
> > > > >> > > > > > would like to know if there are any concerns
> with
> > > > these
> > > > >> changes
> > > > >> > > > > before
> > > > >> > > > > > updating the KIP.
> > > > >> > > > > >
> > > > >> > > > > > *Proposed API:*
> > > > >> > > > > > public interface Authorizer extends
> Configurable,
> > > > Closeable
> > > > >> {
> > > > >> > > > > >
> > > > >> > > > > > Map<Endpoint, CompletionStage<Void>>
> > > > >> > > start(AuthorizerServerInfo
> > > > >> > > > > > serverInfo);
> > > > >> > > > > > List<CompletionStage<AuthorizationResult>>
> > > > >> > > > > > authorize(AuthorizableRequestContext
> > requestContext,
> > > > >> > List<Action>
> > > > >> > > > > > actions);
> > > > >> > > > > > List<CompletionStage<AclCreateResult>>
> > > > >> > > > > > createAcls(AuthorizableRequestContext
> > > requestContext,
> > > > >> > > > > List<AclBinding>
> > > > >> > > > > > aclBindings);
> > > > >> > > > > > List<CompletionStage<AclDeleteResult>>
> > > > >> > > > > > deleteAcls(AuthorizableRequestContext
> > > requestContext,
> > > > >> > > > > > List<AclBindingFilter> aclBindingFilters);
> > > > >> > > > > > CompletionStage<Collection<AclBinding>>
> > > > >> > acls(AclBindingFilter
> > > > >> > > > > > filter);
> > > > >> > > > > > }
> > > > >> > > > > >
> > > > >> > > > > >
> > > > >> > > > > > Thank you,
> > > > >> > > > > >
> > > > >> > > > > > Rajini
> > > > >> > > > > >
> > > > >> > > > > > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai
> <
> > > > >> > > bosco@apache.org>
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > Hi Rajini
> > > > >> > > > > > >
> > > > >> > > > > > > Thanks for clarifying. I am good for now.
> > > > >> > > > > > >
> > > > >> > > > > > > Regards
> > > > >> > > > > > >
> > > > >> > > > > > > Bosco
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <
> > > > >> > > rajinisivaram@gmail.com
> > > > >> > > > >
> > > > >> > > > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > Hi Don,
> > > > >> > > > > > >
> > > > >> > > > > > > That should be fine. I guess Ranger loads
> > > > policies
> > > > >> from
> > > > >> > the
> > > > >> > > > > > database
> > > > >> > > > > > > synchronously when the authorizer is
> > > configured,
> > > > >> similar
> > > > >> > to
> > > > >> > > > > > > SimpleAclAuthorizer loading from
> ZooKeeper.
> > > > Ranger can
> > > > >> > > > continue
> > > > >> > > > > > to load
> > > > >> > > > > > > synchronously from `configure()` or
> > `start()`
> > > > and
> > > > >> return
> > > > >> > an
> > > > >> > > > > > empty map
> > > > >> > > > > > > from
> > > > >> > > > > > > `start()`. That would retain the existing
> > > > behaviour..
> > > > >> > When
> > > > >> > > > the
> > > > >> > > > > > same
> > > > >> > > > > > > database stores policies for all listeners
> > and
> > > > the
> > > > >> > policies
> > > > >> > > > are
> > > > >> > > > > > not
> > > > >> > > > > > > stored
> > > > >> > > > > > > in Kafka, there is no value in making the
> > load
> > > > >> > > asynchronous.
> > > > >> > > > > > >
> > > > >> > > > > > > Regards,
> > > > >> > > > > > >
> > > > >> > > > > > > Rajini
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco
> > > Durai
> > > > <
> > > > >> > > > > > bosco@apache.org>
> > > > >> > > > > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > Hi Rajini
> > > > >> > > > > > > >
> > > > >> > > > > > > > Assuming this doesn't affect custom
> > plugins,
> > > > I don't
> > > > >> > have
> > > > >> > > > any
> > > > >> > > > > > > concerns
> > > > >> > > > > > > > regarding this.
> > > > >> > > > > > > >
> > > > >> > > > > > > > I do have one question regarding:
> > > > >> > > > > > > >
> > > > >> > > > > > > > "For authorizers that don’t store
> metadata
> > > in
> > > > >> > ZooKeeper,
> > > > >> > > > > > ensure that
> > > > >> > > > > > > > authorizer metadata for each listener is
> > > > available
> > > > >> > before
> > > > >> > > > > > starting
> > > > >> > > > > > > up the
> > > > >> > > > > > > > listener. This enables different
> > > authorization
> > > > >> metadata
> > > > >> > > > > stores
> > > > >> > > > > > for
> > > > >> > > > > > > > different listeners."
> > > > >> > > > > > > >
> > > > >> > > > > > > > Since Ranger uses its own database for
> > > storing
> > > > >> > policies,
> > > > >> > > do
> > > > >> > > > > you
> > > > >> > > > > > > anticipate
> > > > >> > > > > > > > any issues?
> > > > >> > > > > > > >
> > > > >> > > > > > > > Thanks
> > > > >> > > > > > > >
> > > > >> > > > > > > > Bosco
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
> > > > >> > > > > > rajinisivaram@gmail.com>
> > > > >> > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > Hi all,
> > > > >> > > > > > > >
> > > > >> > > > > > > > I made another change to the KIP.
> The
> > > KIP
> > > > was
> > > > >> > > > originally
> > > > >> > > > > > > proposing to
> > > > >> > > > > > > > extend SimpleAclAuthorizer to also
> > > > implement the
> > > > >> > new
> > > > >> > > > API
> > > > >> > > > > > (in
> > > > >> > > > > > > addition
> > > > >> > > > > > > > to
> > > > >> > > > > > > > the existing API). But since we use
> > the
> > > > new API
> > > > >> > when
> > > > >> > > > > > available,
> > > > >> > > > > > > this
> > > > >> > > > > > > > can
> > > > >> > > > > > > > break custom authorizers that extend
> > > this
> > > > class
> > > > >> and
> > > > >> > > > > > override
> > > > >> > > > > > > specific
> > > > >> > > > > > > > methods of the old API. To avoid
> > > breaking
> > > > any
> > > > >> > > existing
> > > > >> > > > > > custom
> > > > >> > > > > > > > implementations that extend this
> > class,
> > > > >> > particularly
> > > > >> > > > > > because it
> > > > >> > > > > > > is in
> > > > >> > > > > > > > the
> > > > >> > > > > > > > public package kafka.security.auth,
> > the
> > > > KIP now
> > > > >> > > > proposes
> > > > >> > > > > to
> > > > >> > > > > > > retain the
> > > > >> > > > > > > > old
> > > > >> > > > > > > > authorizer as-is.
> SimpleAclAuthorizer
> > > > will
> > > > >> > continue
> > > > >> > > to
> > > > >> > > > > > > implement the
> > > > >> > > > > > > > old
> > > > >> > > > > > > > API, but will be deprecated. A new
> > > > authorizer
> > > > >> > > > > > implementation
> > > > >> > > > > > > >
> > kafka.security.authorizer.AclAuthorizer
> > > > will be
> > > > >> > added
> > > > >> > > > for
> > > > >> > > > > > the
> > > > >> > > > > > > new API
> > > > >> > > > > > > > (this
> > > > >> > > > > > > > will not be in the public package).
> > > > >> > > > > > > >
> > > > >> > > > > > > > Please let me know if you have any
> > > > concerns.
> > > > >> > > > > > > >
> > > > >> > > > > > > > Regards,
> > > > >> > > > > > > >
> > > > >> > > > > > > > Rajini
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Fri, Aug 16, 2019 at 8:48 AM
> Rajini
> > > > Sivaram <
> > > > >> > > > > > > > rajinisivaram@gmail.com>
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Thanks Colin.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > If there are no other concerns, I
> > will
> > > > start
> > > > >> vote
> > > > >> > > > later
> > > > >> > > > > > today.
> > > > >> > > > > > > Many
> > > > >> > > > > > > > thanks
> > > > >> > > > > > > > > to every one for the feedback.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Regards,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Rajini
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Thu, Aug 15, 2019 at 11:57 PM
> > Colin
> > > > McCabe
> > > > >> <
> > > > >> > > > > > > cmccabe@apache.org>
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >> Thanks, Rajini. It looks good to
> > me.
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> best,
> > > > >> > > > > > > > >> Colin
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >>
> > > > >> > > > > > > > >> On Thu, Aug 15, 2019, at 11:37,
> > > Rajini
> > > > >> Sivaram
> > > > >> > > > wrote:
> > > > >> > > > > > > > >> > Hi Colin,
> > > > >> > > > > > > > >> >
> > > > >> > > > > > > > >> > Thanks for the review. I have
> > > > updated the
> > > > >> KIP
> > > > >> > to
> > > > >> > > > > move
> > > > >> > > > > > the
> > > > >> > > > > > > > interfaces for
> > > > >> > > > > > > > >> > request context and server info
> > to
> > > > the
> > > > >> > > authorizer
> > > > >> > > > > > package.
> > > > >> > > > > > > These
> > > > >> > > > > > > > are now
> > > > >> > > > > > > > >> > called
> AuthorizableRequestContext
> > > and
> > > > >> > > > > > AuthorizerServerInfo.
> > > > >> > > > > > > > Endpoint is
> > > > >> > > > > > > > >> now
> > > > >> > > > > > > > >> > a class in
> > org.apache.kafka.common
> > > > to make
> > > > >> it
> > > > >> > > > > reusable
> > > > >> > > > > > > since we
> > > > >> > > > > > > > already
> > > > >> > > > > > > > >> > have multiple implementations
> of
> > > it.
> > > > I have
> > > > >> > > > removed
> > > > >> > > > > > > requestName
> > > > >> > > > > > > > from the
> > > > >> > > > > > > > >> > request context interface since
> > > > authorizers
> > > > >> > can
> > > > >> > > > > > distinguish
> > > > >> > > > > > > > follower
> > > > >> > > > > > > > >> fetch
> > > > >> > > > > > > > >> > and consumer fetch from the
> > > > operation being
> > > > >> > > > > > authorized. So
> > > > >> > > > > > > 16-bit
> > > > >> > > > > > > > >> request
> > > > >> > > > > > > > >> > type should be sufficient for
> > audit
> > > > >> logging.
> > > > >> > > Also
> > > > >> > > > > > replaced
> > > > >> > > > > > > > AuditFlag
> > > > >> > > > > > > > >> with
> > > > >> > > > > > > > >> > two booleans as you suggested.
> > > > >> > > > > > > > >> >
> > > > >> > > > > > > > >> > Can you take another look and
> see
> > > if
> > > > the
> > > > >> KIP
> > > > >> > is
> > > > >> > > > > ready
> > > > >> > > > > > for
> > > > >> > > > > > > voting?
> > > > >> > > > > > > > >> >
> > > > >> > > > > > > > >> > Thanks for all your help!
> > > > >> > > > > > > > >> >
> > > > >> > > > > > > > >> > Regards,
> > > > >> > > > > > > > >> >
> > > > >> > > > > > > > >> > Rajini
> > > > >> > > > > > > > >> >
> > > > >> > > > > > > > >> > On Wed, Aug 14, 2019 at 8:59 PM
> > > Colin
> > > > >> McCabe <
> > > > >> > > > > > > cmccabe@apache.org>
> > > > >> > > > > > > > >> wrote:
> > > > >> > > > > > > > >> >
> > > > >> > > > > > > > >> > > Hi Rajini,
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> > > I think it would be good to
> > > rename
> > > > >> > > > > > KafkaRequestContext to
> > > > >> > > > > > > > something
> > > > >> > > > > > > > >> like
> > > > >> > > > > > > > >> > > AuthorizableRequestContext,
> and
> > > > put it in
> > > > >> > the
> > > > >> > > > > > > > >> > >
> > > org.apache.kafka.server.authorizer
> > > > >> > namespace.
> > > > >> > > > If
> > > > >> > > > > > we put
> > > > >> > > > > > > it in
> > > > >> > > > > > > > the
> > > > >> > > > > > > > >> > > org.apache.kafka.common
> > > namespace,
> > > > then
> > > > >> it's
> > > > >> > > not
> > > > >> > > > > > really
> > > > >> > > > > > > clear
> > > > >> > > > > > > > that
> > > > >> > > > > > > > >> it's
> > > > >> > > > > > > > >> > > part of the Authorizer API.
> > > Since
> > > > this
> > > > >> > class
> > > > >> > > is
> > > > >> > > > > > purely an
> > > > >> > > > > > > > interface,
> > > > >> > > > > > > > >> with
> > > > >> > > > > > > > >> > > no concrete implementation of
> > > > anything,
> > > > >> > > there's
> > > > >> > > > > > nothing
> > > > >> > > > > > > common
> > > > >> > > > > > > > to
> > > > >> > > > > > > > >> really
> > > > >> > > > > > > > >> > > reuse in any case. We
> > definitely
> > > > don't
> > > > >> want
> > > > >> > > > > > someone to
> > > > >> > > > > > > > accidentally
> > > > >> > > > > > > > >> add or
> > > > >> > > > > > > > >> > > remove methods because they
> > think
> > > > this is
> > > > >> > just
> > > > >> > > > > > another
> > > > >> > > > > > > internal
> > > > >> > > > > > > > class
> > > > >> > > > > > > > >> used
> > > > >> > > > > > > > >> > > for requests.
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> > > The BrokerInfo class is a
> nice
> > > > >> improvement.
> > > > >> > > It
> > > > >> > > > > > looks
> > > > >> > > > > > > like it
> > > > >> > > > > > > > will be
> > > > >> > > > > > > > >> > > useful for passing in
> > information
> > > > about
> > > > >> the
> > > > >> > > > > context
> > > > >> > > > > > we're
> > > > >> > > > > > > > running
> > > > >> > > > > > > > >> in. It
> > > > >> > > > > > > > >> > > would be nice to call this
> > class
> > > > >> ServerInfo
> > > > >> > > > rather
> > > > >> > > > > > than
> > > > >> > > > > > > > BrokerInfo,
> > > > >> > > > > > > > >> since
> > > > >> > > > > > > > >> > > we will want to run the
> > > authorizer
> > > > on
> > > > >> > > > controllers
> > > > >> > > > > > as well
> > > > >> > > > > > > as on
> > > > >> > > > > > > > >> brokers,
> > > > >> > > > > > > > >> > > and the controller may run
> as a
> > > > separate
> > > > >> > > process
> > > > >> > > > > > post
> > > > >> > > > > > > KIP-500.
> > > > >> > > > > > > > I also
> > > > >> > > > > > > > >> > > think that this class should
> be
> > > in
> > > > the
> > > > >> > > > > > > > >>
> org.apache.kafka.server.authorizer
> > > > >> > > > > > > > >> > > namespace. Again, it is an
> > > > interface,
> > > > >> not a
> > > > >> > > > > > concrete
> > > > >> > > > > > > > implementation,
> > > > >> > > > > > > > >> and
> > > > >> > > > > > > > >> > > it's an interface that is
> very
> > > > >> specifically
> > > > >> > > for
> > > > >> > > > > the
> > > > >> > > > > > > authorizer.
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> > > I agree that we probably
> don't
> > > have
> > > > >> enough
> > > > >> > > > > > information
> > > > >> > > > > > > > preserved for
> > > > >> > > > > > > > >> > > requests currently to always
> > know
> > > > what
> > > > >> > entity
> > > > >> > > > made
> > > > >> > > > > > them.
> > > > >> > > > > > > So we
> > > > >> > > > > > > > can
> > > > >> > > > > > > > >> leave
> > > > >> > > > > > > > >> > > that out for now (except in
> the
> > > > special
> > > > >> case
> > > > >> > > of
> > > > >> > > > > > Fetch).
> > > > >> > > > > > > > Perhaps we
> > > > >> > > > > > > > >> can add
> > > > >> > > > > > > > >> > > this later if it's needed.
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> > > I understand the intention
> > behind
> > > > >> > > > > AuthorizationMode
> > > > >> > > > > > > (which is
> > > > >> > > > > > > > now
> > > > >> > > > > > > > >> called
> > > > >> > > > > > > > >> > > AuditFlag in the latest
> > > > revision). But
> > > > >> it
> > > > >> > > still
> > > > >> > > > > > feels
> > > > >> > > > > > > > complex. What
> > > > >> > > > > > > > >> if we
> > > > >> > > > > > > > >> > > just had two booleans in
> > Action:
> > > > >> > logSuccesses
> > > > >> > > > and
> > > > >> > > > > > > logFailures?
> > > > >> > > > > > > > That
> > > > >> > > > > > > > >> seems
> > > > >> > > > > > > > >> > > to cover all the cases here.
> > > > >> > > > MANDATORY_AUTHORIZE
> > > > >> > > > > =
> > > > >> > > > > > true,
> > > > >> > > > > > > true.
> > > > >> > > > > > > > >> > > OPTIONAL_AUTHORIZE = true,
> > false.
> > > > >> FILTER =
> > > > >> > > > true,
> > > > >> > > > > > false.
> > > > >> > > > > > > > >> LIST_AUTHORIZED =
> > > > >> > > > > > > > >> > > false, false. Would there be
> > > > anything
> > > > >> lost
> > > > >> > > > versus
> > > > >> > > > > > having
> > > > >> > > > > > > the
> > > > >> > > > > > > > enum?
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> > > best,
> > > > >> > > > > > > > >> > > Colin
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> > >
> > > > >> > > > > > > > >> > > On Wed, Aug 14, 2019, at
> 06:29,
> > > > Mickael
> > > > >> > Maison
> > > > >> > > > > > wrote:
> > > > >> > > > > > > > >> > > > Hi Rajini,
> > > > >> > > > > > > > >> > > >
> > > > >> > > > > > > > >> > > > Thanks for the KIP!
> > > > >> > > > > > > > >> > > > I really like that
> > authorize()
> > > > will be
> > > > >> > able
> > > > >> > > to
> > > > >> > > > > > take a
> > > > >> > > > > > > batch of
> > > > >> > > > > > > > >> > > > requests, this will speed
> up
> > > many
> > > > >> > > > > implementations!
> > > > >> > > > > > > > >> > > >
> > > > >> > > > > > > > >> > > > On Tue, Aug 13, 2019 at
> 5:57
> > PM
> > > > Rajini
> > > > >> > > > Sivaram <
> > > > >> > > > > > > > >> rajinisivaram@gmail.com>
> > > > >> > > > > > > > >> > > wrote:
> > > > >> > > > > > > > >> > > > >
> > > > >> > > > > > > > >> > > > > Thanks David! I have
> fixed
> > > the
> > > > typo.
> > > > >> > > > > > > > >> > > > >
> > > > >> > > > > > > > >> > > > > Also made a couple of
> > changes
> > > > to make
> > > > >> > the
> > > > >> > > > > > context
> > > > >> > > > > > > > interfaces more
> > > > >> > > > > > > > >> > > generic.
> > > > >> > > > > > > > >> > > > > KafkaRequestContext now
> > > > returns the
> > > > >> > 16-bit
> > > > >> > > > API
> > > > >> > > > > > key as
> > > > >> > > > > > > Colin
> > > > >> > > > > > > > >> suggested
> > > > >> > > > > > > > >> > > as
> > > > >> > > > > > > > >> > > > > well as the friendly name
> > > used
> > > > in
> > > > >> > metrics
> > > > >> > > > > which
> > > > >> > > > > > are
> > > > >> > > > > > > useful
> > > > >> > > > > > > > in
> > > > >> > > > > > > > >> audit
> > > > >> > > > > > > > >> > > logs.
> > > > >> > > > > > > > >> > > > > `Authorizer#start` is now
> > > > provided a
> > > > >> > > generic
> > > > >> > > > > > > `BrokerInfo`
> > > > >> > > > > > > > >> interface
> > > > >> > > > > > > > >> > > that
> > > > >> > > > > > > > >> > > > > gives cluster id, broker
> id
> > > and
> > > > >> endpoint
> > > > >> > > > > > information.
> > > > >> > > > > > > The
> > > > >> > > > > > > > generic
> > > > >> > > > > > > > >> > > interface
> > > > >> > > > > > > > >> > > > > can potentially be used
> in
> > > > other
> > > > >> broker
> > > > >> > > > > plugins
> > > > >> > > > > > in
> > > > >> > > > > > > future
> > > > >> > > > > > > > and
> > > > >> > > > > > > > >> provides
> > > > >> > > > > > > > >> > > > > dynamically generated
> > configs
> > > > like
> > > > >> > broker
> > > > >> > > id
> > > > >> > > > > > and ports
> > > > >> > > > > > > > which are
> > > > >> > > > > > > > >> > > currently
> > > > >> > > > > > > > >> > > > > not available to plugins
> > > > unless these
> > > > >> > > > configs
> > > > >> > > > > > are
> > > > >> > > > > > > statically
> > > > >> > > > > > > > >> > > configured.
> > > > >> > > > > > > > >> > > > > Please let me know if
> there
> > > > are any
> > > > >> > > > concerns.
> > > > >> > > > > > > > >> > > > >
> > > > >> > > > > > > > >> > > > > Regards,
> > > > >> > > > > > > > >> > > > >
> > > > >> > > > > > > > >> > > > > Rajini
> > > > >> > > > > > > > >> > > > >
> > > > >> > > > > > > > >> > > > > On Tue, Aug 13, 2019 at
> > 4:30
> > > > PM David
> > > > >> > > Jacot
> > > > >> > > > <
> > > > >> > > > > > > > djacot@confluent.io>
> > > > >> > > > > > > > >> > > wrote:
> > > > >> > > > > > > > >> > > > >
> > > > >> > > > > > > > >> > > > > > Hi Rajini,
> > > > >> > > > > > > > >> > > > > >
> > > > >> > > > > > > > >> > > > > > Thank you for the
> update!
> > > It
> > > > looks
> > > > >> > good
> > > > >> > > to
> > > > >> > > > > me.
> > > > >> > > > > > > There is a
> > > > >> > > > > > > > typo
> > > > >> > > > > > > > >> in the
> > > > >> > > > > > > > >> > > > > > `AuditFlag` enum:
> > > > >> > `MANDATORY_AUTHOEIZE`
> > > > >> > > ->
> > > > >> > > > > > > > >> `MANDATORY_AUTHORIZE`.
> > > > >> > > > > > > > >> > > > > >
> > > > >> > > > > > > > >> > > > > > Regards,
> > > > >> > > > > > > > >> > > > > > David
> > > > >> > > > > > > > >> > > > > >
> > > > >> > > > > > > > >> > > > > > On Mon, Aug 12, 2019 at
> > > 2:54
> > > > PM
> > > > >> Rajini
> > > > >> > > > > > Sivaram <
> > > > >> > > > > > > > >> > > rajinisivaram@gmail.com>
> > > > >> > > > > > > > >> > > > > > wrote:
> > > > >> > > > > > > > >> > > > > >
> > > > >> > > > > > > > >> > > > > > > Hi David,
> > > > >> > > > > > > > >> > > > > > >
> > > > >> > > > > > > > >> > > > > > > Thanks for reviewing
> > the
> > > > KIP!
> > > > >> Since
> > > > >> > > > > > questions
> > > > >> > > > > > > about
> > > > >> > > > > > > > >> `authorization
> > > > >> > > > > > > > >> > > mode`
> > > > >> > > > > > > > >> > > > > > > and `count` have come
> > up
> > > > multiple
> > > > >> > > > times, I
> > > > >> > > > > > have
> > > > >> > > > > > > renamed
> > > > >> > > > > > > > both.
> > > > >> > > > > > > > >> > > > > > >
> > > > >> > > > > > > > >> > > > > > > 1) Renamed `count` to
> > > > >> > > > > > `resourceReferenceCount`.
> > > > >> > > > > > > It is
> > > > >> > > > > > > > the
> > > > >> > > > > > > > >> number
> > > > >> > > > > > > > >> > > of times
> > > > >> > > > > > > > >> > > > > > > the resource being
> > > > authorized is
> > > > >> > > > > referenced
> > > > >> > > > > > > within the
> > > > >> > > > > > > > >> request.
> > > > >> > > > > > > > >> > > > > > >
> > > > >> > > > > > > > >> > > > > > > 2) Renamed
> > > > `AuthorizationMode` to
> > > > >> > > > > > `AuditFlag`. It
> > > > >> > > > > > > is
> > > > >> > > > > > > > provided
> > > > >> > > > > > > > >> to
> > > > >> > > > > > > > >> > > improve
> > > > >> > > > > > > > >> > > > > > > audit logging in the
> > > > authorizer.
> > > > >> The
> > > > >> > > > enum
> > > > >> > > > > > values
> > > > >> > > > > > > have
> > > > >> > > > > > > > javadoc
> > > > >> > > > > > > > >> which
> > > > >> > > > > > > > >> > > > > > > indicate how the
> > > > authorization
> > > > >> > result
> > > > >> > > is
> > > > >> > > > > > used in
> > > > >> > > > > > > each
> > > > >> > > > > > > > of the
> > > > >> > > > > > > > >> modes
> > > > >> > > > > > > > >> > > to
> > > > >> > > > > > > > >> > > > > > > enable authorizers to
> > log
> > > > audit
> > > > >> > > messages
> > > > >> > > > > at
> > > > >> > > > > > the
> > > > >> > > > > > > right
> > > > >> > > > > > > > severity
> > > > >> > > > > > > > >> > > level.
> > > > >> > > > > > > > >> > > > > > >
> > > > >> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Rajini,
Can you please update the javadoc for every sync method to be explicit
about the fact that they should not block and the reasoning?
Ismael
On Sun, Sep 8, 2019, 9:21 AM Rajini Sivaram <ra...@gmail.com> wrote:
> Hi all,
>
> Thanks everyone for the very useful discussion. I have updated the KIP to
> make *createAcls()* and *deleteAcls()* asynchronous. Also added a section
> under `*Rejected Alternatives*` for async *authorize()*, which we can
> revisit in future if requirement arises.
>
> Regards,
>
> Rajini
>
>
> On Fri, Sep 6, 2019 at 9:17 PM Ismael Juma <is...@juma.me.uk> wrote:
>
> > It's not uncommon for people to do large numbers of admin operations per
> > second. We've seen it many times in production. It's often a bug, but the
> > system must cope with it without affecting other users of the same Kafka
> > cluster.
> >
> > Ismael
> >
> > On Fri, Sep 6, 2019, 9:32 AM Don Bosco Durai <bo...@apache.org> wrote:
> >
> > > +1
> > >
> > > I might be wrong here, but here are few of my assumptions..
> > >
> > > 1. Unlike transactional services, in Kafka most of the users are
> > > application users, so once the connection is established, then they are
> > > going to be long running sessions. So the caching is more predictable
> and
> > > heavy lifting can be done during initial session setup and cache can be
> > > frequently updated using background thread. KSQL might have a different
> > > flow, but I don't have enough datapoints for it.
> > > 2. For publish/consume authorization calls, providing async option will
> > be
> > > misleading for Kafka like load
> > > 3. For admin calls, hopefully no one does hundreds of call in multiple
> > > thread within few seconds. This would be a bad design.
> > >
> > > So I feel, we should be okay with sync calls for now. And consider
> async
> > > in the future if needed. At least in Apache Ranger implementation, we
> > would
> > > be okay using sync operation. I am not sure if any other plugin
> > > implementation would benefit from async implementation.
> > >
> > > Thanks
> > >
> > > Bosco
> > >
> > > On 9/6/19, 8:45 AM, "Ismael Juma" <is...@juma.me.uk> wrote:
> > >
> > > One more thing: if people have strong opinions that authorize has
> to
> > be
> > > sync, we could leave it like that for now. If needed, we can later
> > add
> > > an
> > > authorizeAsync with another method returning a Boolean indicating
> > that
> > > it's
> > > supported. That is, there is a path to evolve the interface (even
> if
> > a
> > > bit
> > > ugly).
> > >
> > > What about the other methods, is there consensus that they should
> be
> > > updated to return CompletionStage?
> > >
> > > Ismael
> > >
> > > On Fri, Sep 6, 2019, 8:32 AM Ismael Juma <is...@juma.me.uk>
> wrote:
> > >
> > > > I'm on the move, but what Rajini said seems reasonable. I don't
> > think
> > > > using SSDs solves the issue. They can still hang for seconds when
> > > they
> > > > fail. Also, many people may not have local SSDs available (remote
> > > SSDs like
> > > > EBS hang for tens of seconds when there are issues).
> > > >
> > > > We are currently vulnerable to all of these in the normal
> > read/write
> > > path,
> > > > but it seems suboptimal to continue assuming non blocking
> behavior
> > > for
> > > > things that actually do block.
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Sep 6, 2019, 8:24 AM Rajini Sivaram <
> > rajinisivaram@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Hi Jun,
> > > >>
> > > >> For ACLs, the size is also dependent on the number of users. So
> in
> > > >> deployments with large number of users where some users are not
> > > active, an
> > > >> LRU cache may be useful. Agree that there may be other solutions
> > > that help
> > > >> avoid the requirement for an async API even for this case.
> > > >>
> > > >> I wasn't expecting our threading model to change very much.
> Having
> > > said
> > > >> that, I haven't figured out how the purgatory can be adapted to
> > work
> > > >> efficiently for this. I was thinking:
> > > >> 1) If authorization future is complete when returning from
> > > authorize()
> > > >> call, we invoke the method immediately on the request thread
> > > >> 2) If future is not complete, the request goes into a purgatory
> > > >> 3) When future completes, we mark the request ready to be run.
> > This
> > > can be
> > > >> done on ForkJoinPool.commonPool or an executor. But we don't
> > handle
> > > the
> > > >> request on that thread
> > > >> 4) We handle the request on the request thread after 3).
> > > >>
> > > >> I totally agree that this adds complexity to the code. Will wait
> > for
> > > >> Ismael
> > > >> to comment on whether he had a different model in mind.
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Rajini
> > > >>
> > > >>
> > > >> On Thu, Sep 5, 2019 at 10:29 PM Jun Rao <ju...@confluent.io>
> wrote:
> > > >>
> > > >> > Hi, Ismael,
> > > >> >
> > > >> > Yes, I agree that there are 2 separate discussion points, one
> on
> > > the API
> > > >> > and the other on the implementation.
> > > >> >
> > > >> > First on the API, in general, my feeling is that the
> authorize()
> > > call is
> > > >> > expected to be fast to reduce request latency. Adding an async
> > API
> > > >> could be
> > > >> > mis-leading since it might give users the impression that it
> > > would be
> > > >> > arbitrarily long. In the short term, currently, we cache all
> > > partition
> > > >> > level metadata on every broker. The authorization metadata is
> > > >> proportional
> > > >> > to the number of topics, which typically is less in size. So
> it
> > > we can
> > > >> > cache all partition level metadata, we should be able to cache
> > all
> > > >> > authorization metadata. Longer term, if we want to support
> more
> > > metadata
> > > >> > beyond memory, another option is to put all metadata in an SSD
> > > backed
> > > >> > key/value store. The lookup time could still be under 1ms,
> which
> > > we
> > > >> could
> > > >> > potentially absorb in the request threads. So, I am not sure
> > async
> > > >> > authorize() api is strictly needed even in the long term.
> > > >> >
> > > >> > Second on the implementation. I was more concerned about the
> > > impact on
> > > >> our
> > > >> > threading model. If we make authorize() async, some threads
> have
> > > to
> > > >> > complete the future when it's ready. It seems to me that those
> > > threads
> > > >> have
> > > >> > to be inside the Authorizer implementation since only it knows
> > > when to
> > > >> > complete a future? If so, it feels weird that all the logic of
> > > request
> > > >> > handling post authorize() is now handled by external threads
> > > inside a
> > > >> > plugin. It's not clear how we can configure and monitor them,
> > and
> > > how
> > > >> > throttling would work. Perhaps we can provide a bit more
> detail
> > > on the
> > > >> new
> > > >> > threading model.
> > > >> >
> > > >> > Thanks,
> > > >> >
> > > >> > Jun
> > > >> >
> > > >> > On Wed, Sep 4, 2019 at 9:56 PM Ismael Juma <ismael@juma.me.uk
> >
> > > wrote:
> > > >> >
> > > >> > > Hi Jun,
> > > >> > >
> > > >> > > I think it's important to discuss the pluggable API versus
> the
> > > calling
> > > >> > > implementation as two separate points.
> > > >> > >
> > > >> > > From an API perspective, are you suggesting that we should
> > tell
> > > users
> > > >> > that
> > > >> > > they cannot block in authorize? Or are you saying that it's
> OK
> > > to
> > > >> block
> > > >> > on
> > > >> > > authorize on occasion and the recommendation would be for
> > > people to
> > > >> > > increase the number of threads in the request thread pool?
> > > >> > Architecturally,
> > > >> > > this feels wrong in my opinion.
> > > >> > >
> > > >> > > From the calling implementation perspective, you don't need
> a
> > > >> callback.
> > > >> > You
> > > >> > > can basically say:
> > > >> > >
> > > >> > > authorize(...).map { result =>
> > > >> > > ...
> > > >> > > }
> > > >> > >
> > > >> > > And then have a common method take that Future and handle it
> > > >> > synchronously
> > > >> > > if it's already complete or submit it to the purgatory if
> not.
> > > This is
> > > >> > > similar to how many modern async web libraries work.
> > > >> > >
> > > >> > > As Rajini said, this could be done later. The initial
> > > implementation
> > > >> > could
> > > >> > > simply do `toCompletableFuture().get()` to do it
> > synchronously.
> > > But it
> > > >> > > would mean that we could add this functionality without
> > > changing the
> > > >> > > pluggable interface.
> > > >> > >
> > > >> > > Ismael
> > > >> > >
> > > >> > > On Wed, Sep 4, 2019 at 5:29 AM Jun Rao <ju...@confluent.io>
> > > wrote:
> > > >> > >
> > > >> > > > Hi, Rajini,
> > > >> > > >
> > > >> > > > Thanks for the KIP. I was concerned about #4 too. If we
> > > change the
> > > >> > > handling
> > > >> > > > of all requests to use an async authorize() api, will that
> > > cause the
> > > >> > code
> > > >> > > > much harder to understand? There are quite a few callbacks
> > > already.
> > > >> I
> > > >> > am
> > > >> > > > not sure that we want to introduce more of those. The
> > benefit
> > > from
> > > >> > async
> > > >> > > > authorize() api seems limited.
> > > >> > > >
> > > >> > > > Jun
> > > >> > > >
> > > >> > > > On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <
> > > >> rajinisivaram@gmail.com
> > > >> > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Hi Don,
> > > >> > > > >
> > > >> > > > > Thanks for your note.
> > > >> > > > >
> > > >> > > > > 1) The intention is to avoid blocking in the calling
> > > thread. We
> > > >> > already
> > > >> > > > > have several requests that are put into a purgatory when
> > > waiting
> > > >> for
> > > >> > > > remote
> > > >> > > > > communication, for example produce request waiting for
> > > >> replication.
> > > >> > > Since
> > > >> > > > > we have a limited number of request threads in the
> broker,
> > > we
> > > >> want to
> > > >> > > > make
> > > >> > > > > progress with other requests, while one is waiting on
> any
> > > form of
> > > >> > > remote
> > > >> > > > > communication.
> > > >> > > > >
> > > >> > > > > 2) Async management calls will be useful with the
> default
> > > >> authorizer
> > > >> > > when
> > > >> > > > > KIP-500 removes ZK and we rely on Kafka instead. Our
> > current
> > > >> ZK-based
> > > >> > > > > implementation as well as any custom implementations
> that
> > > don't
> > > >> want
> > > >> > to
> > > >> > > > be
> > > >> > > > > async will just need to return a sync'ed value. So
> instead
> > > of
> > > >> > > returning `
> > > >> > > > > value`, the code would just return
> > > >> > > > > `CompletableFuture.completedFuture(value)
> > > >> > > > > `. So it would be just a single line change in the
> > > implementation
> > > >> > with
> > > >> > > > the
> > > >> > > > > new API. The caller would treat completedFuture exactly
> as
> > > it does
> > > >> > > today,
> > > >> > > > > processing the request synchronously without using a
> > > purgatory.
> > > >> > > > >
> > > >> > > > > 3) For implementations that return a completedFuture as
> > > described
> > > >> in
> > > >> > > 2),
> > > >> > > > > the behaviour would remain exactly the same. No
> additional
> > > >> threads or
> > > >> > > > > purgatory will be used for this case. So there would be
> no
> > > >> > performance
> > > >> > > > > penalty. For implementations that return a future that
> is
> > > not
> > > >> > complete,
> > > >> > > > we
> > > >> > > > > prioritise running more requests concurrently. So in a
> > > deployment
> > > >> > with
> > > >> > > a
> > > >> > > > > large number of clients, we would improve performance by
> > > allowing
> > > >> > other
> > > >> > > > > requests to be processed on the request threads while
> some
> > > are
> > > >> > waiting
> > > >> > > > for
> > > >> > > > > authorization metadata.
> > > >> > > > >
> > > >> > > > > 4) I was concerned about this too. The goal is to make
> the
> > > API
> > > >> > flexible
> > > >> > > > > enough to handle large scale deployments in future when
> > > caching
> > > >> all
> > > >> > > > > authorization metadata in each broker is not viable.
> Using
> > > an
> > > >> async
> > > >> > API
> > > >> > > > > that returns CompletionStage, the caller has the option
> to
> > > handle
> > > >> the
> > > >> > > > > result synchronously or asynchronously, so we don't
> > > necessarily
> > > >> need
> > > >> > to
> > > >> > > > > update the calling code right away. Custom authorizers
> > > using the
> > > >> > async
> > > >> > > > API
> > > >> > > > > have full control over whether authorization is
> performed
> > > in-line
> > > >> > since
> > > >> > > > > completedFuture will always be handled synchronously. We
> > do
> > > need
> > > >> to
> > > >> > > > update
> > > >> > > > > KafkaApis to take advantage of the asynchronous API to
> > > improve
> > > >> scale.
> > > >> > > > Even
> > > >> > > > > though this is a big change, since we will be doing the
> > > same for
> > > >> all
> > > >> > > > > requests, it shouldn't be too hard to maintain since the
> > > same
> > > >> pattern
> > > >> > > > will
> > > >> > > > > be used for all requests.
> > > >> > > > >
> > > >> > > > > Regards,
> > > >> > > > >
> > > >> > > > > Rajini
> > > >> > > > >
> > > >> > > > > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <
> > > bosco@apache.org
> > > >> >
> > > >> > > > wrote:
> > > >> > > > >
> > > >> > > > > > Hi Rajini
> > > >> > > > > >
> > > >> > > > > > Help me understand this a bit more.
> > > >> > > > > >
> > > >> > > > > > 1. For all practical purpose, without authorization
> you
> > > can't
> > > >> go to
> > > >> > > the
> > > >> > > > > > next step. The calling code needs to block anyway. So
> > > should we
> > > >> > just
> > > >> > > > let
> > > >> > > > > > the implementation code do the async part?
> > > >> > > > > > 2. If you feel management calls need to be async, then
> > we
> > > should
> > > >> > > > consider
> > > >> > > > > > another set of async APIs. I don't feel we should
> > > complicate it
> > > >> > > > further (
> > > >> > > > > > 3. Another concern I have is wrt performance. Kafka
> has
> > > been
> > > >> built
> > > >> > to
> > > >> > > > > > handle 1000s per second requests. Not sure whether
> > making
> > > it
> > > >> async
> > > >> > > will
> > > >> > > > > add
> > > >> > > > > > any unnecessary overhead.
> > > >> > > > > > 4. How much complication would this add on the calling
> > > side?
> > > >> And is
> > > >> > > it
> > > >> > > > > > worth it?
> > > >> > > > > >
> > > >> > > > > > Thanks
> > > >> > > > > >
> > > >> > > > > > Bosco
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > On 9/3/19, 8:50 AM, "Rajini Sivaram" <
> > > rajinisivaram@gmail.com>
> > > >> > > wrote:
> > > >> > > > > >
> > > >> > > > > > Hi all,
> > > >> > > > > >
> > > >> > > > > > Ismael brought up a point that it will be good to
> > > make the
> > > >> > > > Authorizer
> > > >> > > > > > interface asynchronous to avoid blocking request
> > > threads
> > > >> during
> > > >> > > > > remote
> > > >> > > > > > operations.
> > > >> > > > > >
> > > >> > > > > > 1) Since we want to support different backends for
> > > >> > authorization
> > > >> > > > > > metadata,
> > > >> > > > > > making createAcls() and deleteAcls() asynchronous
> > > makes
> > > >> sense
> > > >> > > since
> > > >> > > > > > these
> > > >> > > > > > always involve remote operations. When KIP-500
> > removes
> > > >> > ZooKeeper,
> > > >> > > > we
> > > >> > > > > > would
> > > >> > > > > > want to move ACLs to Kafka and async updates will
> > > avoid
> > > >> > > unnecessary
> > > >> > > > > > blocking.
> > > >> > > > > > 2) For authorize() method, we currently use cached
> > > ACLs in
> > > >> the
> > > >> > > > > built-in
> > > >> > > > > > authorizers, so synchronous authorise operations
> > work
> > > well
> > > >> now.
> > > >> > > But
> > > >> > > > > > async
> > > >> > > > > > authorize() would support this scenario as well as
> > > >> authorizers
> > > >> > in
> > > >> > > > > large
> > > >> > > > > > organisations where an LRU cache would enable a
> > > smaller
> > > >> cache
> > > >> > > even
> > > >> > > > > > when the
> > > >> > > > > > backend holds a large amount of ACLs for
> > infrequently
> > > used
> > > >> > > > resources
> > > >> > > > > or
> > > >> > > > > > users who don't use the system frequently.
> > > >> > > > > >
> > > >> > > > > > For both cases, the built-in authorizer will
> > continue
> > > to be
> > > >> > > > > > synchronous,
> > > >> > > > > > using CompletableFuture.completedFuture() to
> return
> > > the
> > > >> actual
> > > >> > > > > result.
> > > >> > > > > > But
> > > >> > > > > > async API will make custom authorizer
> > implementations
> > > more
> > > >> > > > flexible.
> > > >> > > > > I
> > > >> > > > > > would like to know if there are any concerns with
> > > these
> > > >> changes
> > > >> > > > > before
> > > >> > > > > > updating the KIP.
> > > >> > > > > >
> > > >> > > > > > *Proposed API:*
> > > >> > > > > > public interface Authorizer extends Configurable,
> > > Closeable
> > > >> {
> > > >> > > > > >
> > > >> > > > > > Map<Endpoint, CompletionStage<Void>>
> > > >> > > start(AuthorizerServerInfo
> > > >> > > > > > serverInfo);
> > > >> > > > > > List<CompletionStage<AuthorizationResult>>
> > > >> > > > > > authorize(AuthorizableRequestContext
> requestContext,
> > > >> > List<Action>
> > > >> > > > > > actions);
> > > >> > > > > > List<CompletionStage<AclCreateResult>>
> > > >> > > > > > createAcls(AuthorizableRequestContext
> > requestContext,
> > > >> > > > > List<AclBinding>
> > > >> > > > > > aclBindings);
> > > >> > > > > > List<CompletionStage<AclDeleteResult>>
> > > >> > > > > > deleteAcls(AuthorizableRequestContext
> > requestContext,
> > > >> > > > > > List<AclBindingFilter> aclBindingFilters);
> > > >> > > > > > CompletionStage<Collection<AclBinding>>
> > > >> > acls(AclBindingFilter
> > > >> > > > > > filter);
> > > >> > > > > > }
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > > > Thank you,
> > > >> > > > > >
> > > >> > > > > > Rajini
> > > >> > > > > >
> > > >> > > > > > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <
> > > >> > > bosco@apache.org>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi Rajini
> > > >> > > > > > >
> > > >> > > > > > > Thanks for clarifying. I am good for now.
> > > >> > > > > > >
> > > >> > > > > > > Regards
> > > >> > > > > > >
> > > >> > > > > > > Bosco
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <
> > > >> > > rajinisivaram@gmail.com
> > > >> > > > >
> > > >> > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > Hi Don,
> > > >> > > > > > >
> > > >> > > > > > > That should be fine. I guess Ranger loads
> > > policies
> > > >> from
> > > >> > the
> > > >> > > > > > database
> > > >> > > > > > > synchronously when the authorizer is
> > configured,
> > > >> similar
> > > >> > to
> > > >> > > > > > > SimpleAclAuthorizer loading from ZooKeeper.
> > > Ranger can
> > > >> > > > continue
> > > >> > > > > > to load
> > > >> > > > > > > synchronously from `configure()` or
> `start()`
> > > and
> > > >> return
> > > >> > an
> > > >> > > > > > empty map
> > > >> > > > > > > from
> > > >> > > > > > > `start()`. That would retain the existing
> > > behaviour..
> > > >> > When
> > > >> > > > the
> > > >> > > > > > same
> > > >> > > > > > > database stores policies for all listeners
> and
> > > the
> > > >> > policies
> > > >> > > > are
> > > >> > > > > > not
> > > >> > > > > > > stored
> > > >> > > > > > > in Kafka, there is no value in making the
> load
> > > >> > > asynchronous.
> > > >> > > > > > >
> > > >> > > > > > > Regards,
> > > >> > > > > > >
> > > >> > > > > > > Rajini
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco
> > Durai
> > > <
> > > >> > > > > > bosco@apache.org>
> > > >> > > > > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Hi Rajini
> > > >> > > > > > > >
> > > >> > > > > > > > Assuming this doesn't affect custom
> plugins,
> > > I don't
> > > >> > have
> > > >> > > > any
> > > >> > > > > > > concerns
> > > >> > > > > > > > regarding this.
> > > >> > > > > > > >
> > > >> > > > > > > > I do have one question regarding:
> > > >> > > > > > > >
> > > >> > > > > > > > "For authorizers that don’t store metadata
> > in
> > > >> > ZooKeeper,
> > > >> > > > > > ensure that
> > > >> > > > > > > > authorizer metadata for each listener is
> > > available
> > > >> > before
> > > >> > > > > > starting
> > > >> > > > > > > up the
> > > >> > > > > > > > listener. This enables different
> > authorization
> > > >> metadata
> > > >> > > > > stores
> > > >> > > > > > for
> > > >> > > > > > > > different listeners."
> > > >> > > > > > > >
> > > >> > > > > > > > Since Ranger uses its own database for
> > storing
> > > >> > policies,
> > > >> > > do
> > > >> > > > > you
> > > >> > > > > > > anticipate
> > > >> > > > > > > > any issues?
> > > >> > > > > > > >
> > > >> > > > > > > > Thanks
> > > >> > > > > > > >
> > > >> > > > > > > > Bosco
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
> > > >> > > > > > rajinisivaram@gmail.com>
> > > >> > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > Hi all,
> > > >> > > > > > > >
> > > >> > > > > > > > I made another change to the KIP. The
> > KIP
> > > was
> > > >> > > > originally
> > > >> > > > > > > proposing to
> > > >> > > > > > > > extend SimpleAclAuthorizer to also
> > > implement the
> > > >> > new
> > > >> > > > API
> > > >> > > > > > (in
> > > >> > > > > > > addition
> > > >> > > > > > > > to
> > > >> > > > > > > > the existing API). But since we use
> the
> > > new API
> > > >> > when
> > > >> > > > > > available,
> > > >> > > > > > > this
> > > >> > > > > > > > can
> > > >> > > > > > > > break custom authorizers that extend
> > this
> > > class
> > > >> and
> > > >> > > > > > override
> > > >> > > > > > > specific
> > > >> > > > > > > > methods of the old API. To avoid
> > breaking
> > > any
> > > >> > > existing
> > > >> > > > > > custom
> > > >> > > > > > > > implementations that extend this
> class,
> > > >> > particularly
> > > >> > > > > > because it
> > > >> > > > > > > is in
> > > >> > > > > > > > the
> > > >> > > > > > > > public package kafka.security.auth,
> the
> > > KIP now
> > > >> > > > proposes
> > > >> > > > > to
> > > >> > > > > > > retain the
> > > >> > > > > > > > old
> > > >> > > > > > > > authorizer as-is. SimpleAclAuthorizer
> > > will
> > > >> > continue
> > > >> > > to
> > > >> > > > > > > implement the
> > > >> > > > > > > > old
> > > >> > > > > > > > API, but will be deprecated. A new
> > > authorizer
> > > >> > > > > > implementation
> > > >> > > > > > > >
> kafka.security.authorizer.AclAuthorizer
> > > will be
> > > >> > added
> > > >> > > > for
> > > >> > > > > > the
> > > >> > > > > > > new API
> > > >> > > > > > > > (this
> > > >> > > > > > > > will not be in the public package).
> > > >> > > > > > > >
> > > >> > > > > > > > Please let me know if you have any
> > > concerns.
> > > >> > > > > > > >
> > > >> > > > > > > > Regards,
> > > >> > > > > > > >
> > > >> > > > > > > > Rajini
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On Fri, Aug 16, 2019 at 8:48 AM Rajini
> > > Sivaram <
> > > >> > > > > > > > rajinisivaram@gmail.com>
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Thanks Colin.
> > > >> > > > > > > > >
> > > >> > > > > > > > > If there are no other concerns, I
> will
> > > start
> > > >> vote
> > > >> > > > later
> > > >> > > > > > today.
> > > >> > > > > > > Many
> > > >> > > > > > > > thanks
> > > >> > > > > > > > > to every one for the feedback.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Regards,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Rajini
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Thu, Aug 15, 2019 at 11:57 PM
> Colin
> > > McCabe
> > > >> <
> > > >> > > > > > > cmccabe@apache.org>
> > > >> > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > >> Thanks, Rajini. It looks good to
> me.
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> best,
> > > >> > > > > > > > >> Colin
> > > >> > > > > > > > >>
> > > >> > > > > > > > >>
> > > >> > > > > > > > >> On Thu, Aug 15, 2019, at 11:37,
> > Rajini
> > > >> Sivaram
> > > >> > > > wrote:
> > > >> > > > > > > > >> > Hi Colin,
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Thanks for the review. I have
> > > updated the
> > > >> KIP
> > > >> > to
> > > >> > > > > move
> > > >> > > > > > the
> > > >> > > > > > > > interfaces for
> > > >> > > > > > > > >> > request context and server info
> to
> > > the
> > > >> > > authorizer
> > > >> > > > > > package.
> > > >> > > > > > > These
> > > >> > > > > > > > are now
> > > >> > > > > > > > >> > called AuthorizableRequestContext
> > and
> > > >> > > > > > AuthorizerServerInfo.
> > > >> > > > > > > > Endpoint is
> > > >> > > > > > > > >> now
> > > >> > > > > > > > >> > a class in
> org.apache.kafka.common
> > > to make
> > > >> it
> > > >> > > > > reusable
> > > >> > > > > > > since we
> > > >> > > > > > > > already
> > > >> > > > > > > > >> > have multiple implementations of
> > it.
> > > I have
> > > >> > > > removed
> > > >> > > > > > > requestName
> > > >> > > > > > > > from the
> > > >> > > > > > > > >> > request context interface since
> > > authorizers
> > > >> > can
> > > >> > > > > > distinguish
> > > >> > > > > > > > follower
> > > >> > > > > > > > >> fetch
> > > >> > > > > > > > >> > and consumer fetch from the
> > > operation being
> > > >> > > > > > authorized. So
> > > >> > > > > > > 16-bit
> > > >> > > > > > > > >> request
> > > >> > > > > > > > >> > type should be sufficient for
> audit
> > > >> logging.
> > > >> > > Also
> > > >> > > > > > replaced
> > > >> > > > > > > > AuditFlag
> > > >> > > > > > > > >> with
> > > >> > > > > > > > >> > two booleans as you suggested.
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Can you take another look and see
> > if
> > > the
> > > >> KIP
> > > >> > is
> > > >> > > > > ready
> > > >> > > > > > for
> > > >> > > > > > > voting?
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Thanks for all your help!
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Regards,
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > Rajini
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > On Wed, Aug 14, 2019 at 8:59 PM
> > Colin
> > > >> McCabe <
> > > >> > > > > > > cmccabe@apache.org>
> > > >> > > > > > > > >> wrote:
> > > >> > > > > > > > >> >
> > > >> > > > > > > > >> > > Hi Rajini,
> > > >> > > > > > > > >> > >
> > > >> > > > > > > > >> > > I think it would be good to
> > rename
> > > >> > > > > > KafkaRequestContext to
> > > >> > > > > > > > something
> > > >> > > > > > > > >> like
> > > >> > > > > > > > >> > > AuthorizableRequestContext, and
> > > put it in
> > > >> > the
> > > >> > > > > > > > >> > >
> > org.apache.kafka.server.authorizer
> > > >> > namespace.
> > > >> > > > If
> > > >> > > > > > we put
> > > >> > > > > > > it in
> > > >> > > > > > > > the
> > > >> > > > > > > > >> > > org.apache.kafka.common
> > namespace,
> > > then
> > > >> it's
> > > >> > > not
> > > >> > > > > > really
> > > >> > > > > > > clear
> > > >> > > > > > > > that
> > > >> > > > > > > > >> it's
> > > >> > > > > > > > >> > > part of the Authorizer API.
> > Since
> > > this
> > > >> > class
> > > >> > > is
> > > >> > > > > > purely an
> > > >> > > > > > > > interface,
> > > >> > > > > > > > >> with
> > > >> > > > > > > > >> > > no concrete implementation of
> > > anything,
> > > >> > > there's
> > > >> > > > > > nothing
> > > >> > > > > > > common
> > > >> > > > > > > > to
> > > >> > > > > > > > >> really
> > > >> > > > > > > > >> > > reuse in any case. We
> definitely
> > > don't
> > > >> want
> > > >> > > > > > someone to
> > > >> > > > > > > > accidentally
> > > >> > > > > > > > >> add or
> > > >> > > > > > > > >> > > remove methods because they
> think
> > > this is
> > > >> > just
> > > >> > > > > > another
> > > >> > > > > > > internal
> > > >> > > > > > > > class
> > > >> > > > > > > > >> used
> > > >> > > > > > > > >> > > for requests.
> > > >> > > > > > > > >> > >
> > > >> > > > > > > > >> > > The BrokerInfo class is a nice
> > > >> improvement.
> > > >> > > It
> > > >> > > > > > looks
> > > >> > > > > > > like it
> > > >> > > > > > > > will be
> > > >> > > > > > > > >> > > useful for passing in
> information
> > > about
> > > >> the
> > > >> > > > > context
> > > >> > > > > > we're
> > > >> > > > > > > > running
> > > >> > > > > > > > >> in. It
> > > >> > > > > > > > >> > > would be nice to call this
> class
> > > >> ServerInfo
> > > >> > > > rather
> > > >> > > > > > than
> > > >> > > > > > > > BrokerInfo,
> > > >> > > > > > > > >> since
> > > >> > > > > > > > >> > > we will want to run the
> > authorizer
> > > on
> > > >> > > > controllers
> > > >> > > > > > as well
> > > >> > > > > > > as on
> > > >> > > > > > > > >> brokers,
> > > >> > > > > > > > >> > > and the controller may run as a
> > > separate
> > > >> > > process
> > > >> > > > > > post
> > > >> > > > > > > KIP-500.
> > > >> > > > > > > > I also
> > > >> > > > > > > > >> > > think that this class should be
> > in
> > > the
> > > >> > > > > > > > >> org.apache.kafka.server.authorizer
> > > >> > > > > > > > >> > > namespace. Again, it is an
> > > interface,
> > > >> not a
> > > >> > > > > > concrete
> > > >> > > > > > > > implementation,
> > > >> > > > > > > > >> and
> > > >> > > > > > > > >> > > it's an interface that is very
> > > >> specifically
> > > >> > > for
> > > >> > > > > the
> > > >> > > > > > > authorizer.
> > > >> > > > > > > > >> > >
> > > >> > > > > > > > >> > > I agree that we probably don't
> > have
> > > >> enough
> > > >> > > > > > information
> > > >> > > > > > > > preserved for
> > > >> > > > > > > > >> > > requests currently to always
> know
> > > what
> > > >> > entity
> > > >> > > > made
> > > >> > > > > > them.
> > > >> > > > > > > So we
> > > >> > > > > > > > can
> > > >> > > > > > > > >> leave
> > > >> > > > > > > > >> > > that out for now (except in the
> > > special
> > > >> case
> > > >> > > of
> > > >> > > > > > Fetch).
> > > >> > > > > > > > Perhaps we
> > > >> > > > > > > > >> can add
> > > >> > > > > > > > >> > > this later if it's needed.
> > > >> > > > > > > > >> > >
> > > >> > > > > > > > >> > > I understand the intention
> behind
> > > >> > > > > AuthorizationMode
> > > >> > > > > > > (which is
> > > >> > > > > > > > now
> > > >> > > > > > > > >> called
> > > >> > > > > > > > >> > > AuditFlag in the latest
> > > revision). But
> > > >> it
> > > >> > > still
> > > >> > > > > > feels
> > > >> > > > > > > > complex. What
> > > >> > > > > > > > >> if we
> > > >> > > > > > > > >> > > just had two booleans in
> Action:
> > > >> > logSuccesses
> > > >> > > > and
> > > >> > > > > > > logFailures?
> > > >> > > > > > > > That
> > > >> > > > > > > > >> seems
> > > >> > > > > > > > >> > > to cover all the cases here.
> > > >> > > > MANDATORY_AUTHORIZE
> > > >> > > > > =
> > > >> > > > > > true,
> > > >> > > > > > > true.
> > > >> > > > > > > > >> > > OPTIONAL_AUTHORIZE = true,
> false.
> > > >> FILTER =
> > > >> > > > true,
> > > >> > > > > > false.
> > > >> > > > > > > > >> LIST_AUTHORIZED =
> > > >> > > > > > > > >> > > false, false. Would there be
> > > anything
> > > >> lost
> > > >> > > > versus
> > > >> > > > > > having
> > > >> > > > > > > the
> > > >> > > > > > > > enum?
> > > >> > > > > > > > >> > >
> > > >> > > > > > > > >> > > best,
> > > >> > > > > > > > >> > > Colin
> > > >> > > > > > > > >> > >
> > > >> > > > > > > > >> > >
> > > >> > > > > > > > >> > > On Wed, Aug 14, 2019, at 06:29,
> > > Mickael
> > > >> > Maison
> > > >> > > > > > wrote:
> > > >> > > > > > > > >> > > > Hi Rajini,
> > > >> > > > > > > > >> > > >
> > > >> > > > > > > > >> > > > Thanks for the KIP!
> > > >> > > > > > > > >> > > > I really like that
> authorize()
> > > will be
> > > >> > able
> > > >> > > to
> > > >> > > > > > take a
> > > >> > > > > > > batch of
> > > >> > > > > > > > >> > > > requests, this will speed up
> > many
> > > >> > > > > implementations!
> > > >> > > > > > > > >> > > >
> > > >> > > > > > > > >> > > > On Tue, Aug 13, 2019 at 5:57
> PM
> > > Rajini
> > > >> > > > Sivaram <
> > > >> > > > > > > > >> rajinisivaram@gmail.com>
> > > >> > > > > > > > >> > > wrote:
> > > >> > > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > Thanks David! I have fixed
> > the
> > > typo.
> > > >> > > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > Also made a couple of
> changes
> > > to make
> > > >> > the
> > > >> > > > > > context
> > > >> > > > > > > > interfaces more
> > > >> > > > > > > > >> > > generic.
> > > >> > > > > > > > >> > > > > KafkaRequestContext now
> > > returns the
> > > >> > 16-bit
> > > >> > > > API
> > > >> > > > > > key as
> > > >> > > > > > > Colin
> > > >> > > > > > > > >> suggested
> > > >> > > > > > > > >> > > as
> > > >> > > > > > > > >> > > > > well as the friendly name
> > used
> > > in
> > > >> > metrics
> > > >> > > > > which
> > > >> > > > > > are
> > > >> > > > > > > useful
> > > >> > > > > > > > in
> > > >> > > > > > > > >> audit
> > > >> > > > > > > > >> > > logs.
> > > >> > > > > > > > >> > > > > `Authorizer#start` is now
> > > provided a
> > > >> > > generic
> > > >> > > > > > > `BrokerInfo`
> > > >> > > > > > > > >> interface
> > > >> > > > > > > > >> > > that
> > > >> > > > > > > > >> > > > > gives cluster id, broker id
> > and
> > > >> endpoint
> > > >> > > > > > information.
> > > >> > > > > > > The
> > > >> > > > > > > > generic
> > > >> > > > > > > > >> > > interface
> > > >> > > > > > > > >> > > > > can potentially be used in
> > > other
> > > >> broker
> > > >> > > > > plugins
> > > >> > > > > > in
> > > >> > > > > > > future
> > > >> > > > > > > > and
> > > >> > > > > > > > >> provides
> > > >> > > > > > > > >> > > > > dynamically generated
> configs
> > > like
> > > >> > broker
> > > >> > > id
> > > >> > > > > > and ports
> > > >> > > > > > > > which are
> > > >> > > > > > > > >> > > currently
> > > >> > > > > > > > >> > > > > not available to plugins
> > > unless these
> > > >> > > > configs
> > > >> > > > > > are
> > > >> > > > > > > statically
> > > >> > > > > > > > >> > > configured.
> > > >> > > > > > > > >> > > > > Please let me know if there
> > > are any
> > > >> > > > concerns.
> > > >> > > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > Regards,
> > > >> > > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > Rajini
> > > >> > > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > On Tue, Aug 13, 2019 at
> 4:30
> > > PM David
> > > >> > > Jacot
> > > >> > > > <
> > > >> > > > > > > > djacot@confluent.io>
> > > >> > > > > > > > >> > > wrote:
> > > >> > > > > > > > >> > > > >
> > > >> > > > > > > > >> > > > > > Hi Rajini,
> > > >> > > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > Thank you for the update!
> > It
> > > looks
> > > >> > good
> > > >> > > to
> > > >> > > > > me.
> > > >> > > > > > > There is a
> > > >> > > > > > > > typo
> > > >> > > > > > > > >> in the
> > > >> > > > > > > > >> > > > > > `AuditFlag` enum:
> > > >> > `MANDATORY_AUTHOEIZE`
> > > >> > > ->
> > > >> > > > > > > > >> `MANDATORY_AUTHORIZE`.
> > > >> > > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > Regards,
> > > >> > > > > > > > >> > > > > > David
> > > >> > > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > On Mon, Aug 12, 2019 at
> > 2:54
> > > PM
> > > >> Rajini
> > > >> > > > > > Sivaram <
> > > >> > > > > > > > >> > > rajinisivaram@gmail.com>
> > > >> > > > > > > > >> > > > > > wrote:
> > > >> > > > > > > > >> > > > > >
> > > >> > > > > > > > >> > > > > > > Hi David,
> > > >> > > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > > > Thanks for reviewing
> the
> > > KIP!
> > > >> Since
> > > >> > > > > > questions
> > > >> > > > > > > about
> > > >> > > > > > > > >> `authorization
> > > >> > > > > > > > >> > > mode`
> > > >> > > > > > > > >> > > > > > > and `count` have come
> up
> > > multiple
> > > >> > > > times, I
> > > >> > > > > > have
> > > >> > > > > > > renamed
> > > >> > > > > > > > both.
> > > >> > > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > > > 1) Renamed `count` to
> > > >> > > > > > `resourceReferenceCount`.
> > > >> > > > > > > It is
> > > >> > > > > > > > the
> > > >> > > > > > > > >> number
> > > >> > > > > > > > >> > > of times
> > > >> > > > > > > > >> > > > > > > the resource being
> > > authorized is
> > > >> > > > > referenced
> > > >> > > > > > > within the
> > > >> > > > > > > > >> request.
> > > >> > > > > > > > >> > > > > > >
> > > >> > > > > > > > >> > > > > > > 2) Renamed
> > > `AuthorizationMode` to
> > > >> > > > > > `AuditFlag`. It
> > > >> > > > > > > is
> > > >> > > > > > > > provided
> > > >> > > > > > > > >> to
> > > >> > > > > > > > >> > > improve
> > > >> > > > > > > > >> > > > > > > audit logging in the
> > > authorizer.
> > > >> The
> > > >> > > > enum
> > > >> > > > > > values
> > > >> > > > > > > have
> > > >> > > > > > > > javadoc
> > > >> > > > > > > > >> which
> > > >> > > > > > > > >> > > > > > > indicate how the
> > > authorization
> > > >> > result
> > > >> > > is
> > > >> > > > > > used in
> > > >> > > > > > > each
> > > >> > > > > > > > of the
> > > >> > > > > > > > >> modes
> > > >> > > > > > > > >> > > to
> > > >> > > > > > > > >> > > > > > > enable authorizers to
> log
> > > audit
> > > >> > > messages
> > > >> > > > > at
> > > >> > > > > > the
> > > >> > > > > > > right
> > > >> > > > > > > > severity
> > > >> > > > > > > > >> > > level.
> > > >> > > > > > > > >> > > > > > >
> > > >> >
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi all,
Thanks everyone for the very useful discussion. I have updated the KIP to
make *createAcls()* and *deleteAcls()* asynchronous. Also added a section
under `*Rejected Alternatives*` for async *authorize()*, which we can
revisit in future if requirement arises.
Regards,
Rajini
On Fri, Sep 6, 2019 at 9:17 PM Ismael Juma <is...@juma.me.uk> wrote:
> It's not uncommon for people to do large numbers of admin operations per
> second. We've seen it many times in production. It's often a bug, but the
> system must cope with it without affecting other users of the same Kafka
> cluster.
>
> Ismael
>
> On Fri, Sep 6, 2019, 9:32 AM Don Bosco Durai <bo...@apache.org> wrote:
>
> > +1
> >
> > I might be wrong here, but here are few of my assumptions..
> >
> > 1. Unlike transactional services, in Kafka most of the users are
> > application users, so once the connection is established, then they are
> > going to be long running sessions. So the caching is more predictable and
> > heavy lifting can be done during initial session setup and cache can be
> > frequently updated using background thread. KSQL might have a different
> > flow, but I don't have enough datapoints for it.
> > 2. For publish/consume authorization calls, providing async option will
> be
> > misleading for Kafka like load
> > 3. For admin calls, hopefully no one does hundreds of call in multiple
> > thread within few seconds. This would be a bad design.
> >
> > So I feel, we should be okay with sync calls for now. And consider async
> > in the future if needed. At least in Apache Ranger implementation, we
> would
> > be okay using sync operation. I am not sure if any other plugin
> > implementation would benefit from async implementation.
> >
> > Thanks
> >
> > Bosco
> >
> > On 9/6/19, 8:45 AM, "Ismael Juma" <is...@juma.me.uk> wrote:
> >
> > One more thing: if people have strong opinions that authorize has to
> be
> > sync, we could leave it like that for now. If needed, we can later
> add
> > an
> > authorizeAsync with another method returning a Boolean indicating
> that
> > it's
> > supported. That is, there is a path to evolve the interface (even if
> a
> > bit
> > ugly).
> >
> > What about the other methods, is there consensus that they should be
> > updated to return CompletionStage?
> >
> > Ismael
> >
> > On Fri, Sep 6, 2019, 8:32 AM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > I'm on the move, but what Rajini said seems reasonable. I don't
> think
> > > using SSDs solves the issue. They can still hang for seconds when
> > they
> > > fail. Also, many people may not have local SSDs available (remote
> > SSDs like
> > > EBS hang for tens of seconds when there are issues).
> > >
> > > We are currently vulnerable to all of these in the normal
> read/write
> > path,
> > > but it seems suboptimal to continue assuming non blocking behavior
> > for
> > > things that actually do block.
> > >
> > > Ismael
> > >
> > > On Fri, Sep 6, 2019, 8:24 AM Rajini Sivaram <
> rajinisivaram@gmail.com
> > >
> > > wrote:
> > >
> > >> Hi Jun,
> > >>
> > >> For ACLs, the size is also dependent on the number of users. So in
> > >> deployments with large number of users where some users are not
> > active, an
> > >> LRU cache may be useful. Agree that there may be other solutions
> > that help
> > >> avoid the requirement for an async API even for this case.
> > >>
> > >> I wasn't expecting our threading model to change very much. Having
> > said
> > >> that, I haven't figured out how the purgatory can be adapted to
> work
> > >> efficiently for this. I was thinking:
> > >> 1) If authorization future is complete when returning from
> > authorize()
> > >> call, we invoke the method immediately on the request thread
> > >> 2) If future is not complete, the request goes into a purgatory
> > >> 3) When future completes, we mark the request ready to be run.
> This
> > can be
> > >> done on ForkJoinPool.commonPool or an executor. But we don't
> handle
> > the
> > >> request on that thread
> > >> 4) We handle the request on the request thread after 3).
> > >>
> > >> I totally agree that this adds complexity to the code. Will wait
> for
> > >> Ismael
> > >> to comment on whether he had a different model in mind.
> > >>
> > >> Thanks,
> > >>
> > >> Rajini
> > >>
> > >>
> > >> On Thu, Sep 5, 2019 at 10:29 PM Jun Rao <ju...@confluent.io> wrote:
> > >>
> > >> > Hi, Ismael,
> > >> >
> > >> > Yes, I agree that there are 2 separate discussion points, one on
> > the API
> > >> > and the other on the implementation.
> > >> >
> > >> > First on the API, in general, my feeling is that the authorize()
> > call is
> > >> > expected to be fast to reduce request latency. Adding an async
> API
> > >> could be
> > >> > mis-leading since it might give users the impression that it
> > would be
> > >> > arbitrarily long. In the short term, currently, we cache all
> > partition
> > >> > level metadata on every broker. The authorization metadata is
> > >> proportional
> > >> > to the number of topics, which typically is less in size. So it
> > we can
> > >> > cache all partition level metadata, we should be able to cache
> all
> > >> > authorization metadata. Longer term, if we want to support more
> > metadata
> > >> > beyond memory, another option is to put all metadata in an SSD
> > backed
> > >> > key/value store. The lookup time could still be under 1ms, which
> > we
> > >> could
> > >> > potentially absorb in the request threads. So, I am not sure
> async
> > >> > authorize() api is strictly needed even in the long term.
> > >> >
> > >> > Second on the implementation. I was more concerned about the
> > impact on
> > >> our
> > >> > threading model. If we make authorize() async, some threads have
> > to
> > >> > complete the future when it's ready. It seems to me that those
> > threads
> > >> have
> > >> > to be inside the Authorizer implementation since only it knows
> > when to
> > >> > complete a future? If so, it feels weird that all the logic of
> > request
> > >> > handling post authorize() is now handled by external threads
> > inside a
> > >> > plugin. It's not clear how we can configure and monitor them,
> and
> > how
> > >> > throttling would work. Perhaps we can provide a bit more detail
> > on the
> > >> new
> > >> > threading model.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Jun
> > >> >
> > >> > On Wed, Sep 4, 2019 at 9:56 PM Ismael Juma <is...@juma.me.uk>
> > wrote:
> > >> >
> > >> > > Hi Jun,
> > >> > >
> > >> > > I think it's important to discuss the pluggable API versus the
> > calling
> > >> > > implementation as two separate points.
> > >> > >
> > >> > > From an API perspective, are you suggesting that we should
> tell
> > users
> > >> > that
> > >> > > they cannot block in authorize? Or are you saying that it's OK
> > to
> > >> block
> > >> > on
> > >> > > authorize on occasion and the recommendation would be for
> > people to
> > >> > > increase the number of threads in the request thread pool?
> > >> > Architecturally,
> > >> > > this feels wrong in my opinion.
> > >> > >
> > >> > > From the calling implementation perspective, you don't need a
> > >> callback.
> > >> > You
> > >> > > can basically say:
> > >> > >
> > >> > > authorize(...).map { result =>
> > >> > > ...
> > >> > > }
> > >> > >
> > >> > > And then have a common method take that Future and handle it
> > >> > synchronously
> > >> > > if it's already complete or submit it to the purgatory if not.
> > This is
> > >> > > similar to how many modern async web libraries work.
> > >> > >
> > >> > > As Rajini said, this could be done later. The initial
> > implementation
> > >> > could
> > >> > > simply do `toCompletableFuture().get()` to do it
> synchronously.
> > But it
> > >> > > would mean that we could add this functionality without
> > changing the
> > >> > > pluggable interface.
> > >> > >
> > >> > > Ismael
> > >> > >
> > >> > > On Wed, Sep 4, 2019 at 5:29 AM Jun Rao <ju...@confluent.io>
> > wrote:
> > >> > >
> > >> > > > Hi, Rajini,
> > >> > > >
> > >> > > > Thanks for the KIP. I was concerned about #4 too. If we
> > change the
> > >> > > handling
> > >> > > > of all requests to use an async authorize() api, will that
> > cause the
> > >> > code
> > >> > > > much harder to understand? There are quite a few callbacks
> > already.
> > >> I
> > >> > am
> > >> > > > not sure that we want to introduce more of those. The
> benefit
> > from
> > >> > async
> > >> > > > authorize() api seems limited.
> > >> > > >
> > >> > > > Jun
> > >> > > >
> > >> > > > On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <
> > >> rajinisivaram@gmail.com
> > >> > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi Don,
> > >> > > > >
> > >> > > > > Thanks for your note.
> > >> > > > >
> > >> > > > > 1) The intention is to avoid blocking in the calling
> > thread. We
> > >> > already
> > >> > > > > have several requests that are put into a purgatory when
> > waiting
> > >> for
> > >> > > > remote
> > >> > > > > communication, for example produce request waiting for
> > >> replication.
> > >> > > Since
> > >> > > > > we have a limited number of request threads in the broker,
> > we
> > >> want to
> > >> > > > make
> > >> > > > > progress with other requests, while one is waiting on any
> > form of
> > >> > > remote
> > >> > > > > communication.
> > >> > > > >
> > >> > > > > 2) Async management calls will be useful with the default
> > >> authorizer
> > >> > > when
> > >> > > > > KIP-500 removes ZK and we rely on Kafka instead. Our
> current
> > >> ZK-based
> > >> > > > > implementation as well as any custom implementations that
> > don't
> > >> want
> > >> > to
> > >> > > > be
> > >> > > > > async will just need to return a sync'ed value. So instead
> > of
> > >> > > returning `
> > >> > > > > value`, the code would just return
> > >> > > > > `CompletableFuture.completedFuture(value)
> > >> > > > > `. So it would be just a single line change in the
> > implementation
> > >> > with
> > >> > > > the
> > >> > > > > new API. The caller would treat completedFuture exactly as
> > it does
> > >> > > today,
> > >> > > > > processing the request synchronously without using a
> > purgatory.
> > >> > > > >
> > >> > > > > 3) For implementations that return a completedFuture as
> > described
> > >> in
> > >> > > 2),
> > >> > > > > the behaviour would remain exactly the same. No additional
> > >> threads or
> > >> > > > > purgatory will be used for this case. So there would be no
> > >> > performance
> > >> > > > > penalty. For implementations that return a future that is
> > not
> > >> > complete,
> > >> > > > we
> > >> > > > > prioritise running more requests concurrently. So in a
> > deployment
> > >> > with
> > >> > > a
> > >> > > > > large number of clients, we would improve performance by
> > allowing
> > >> > other
> > >> > > > > requests to be processed on the request threads while some
> > are
> > >> > waiting
> > >> > > > for
> > >> > > > > authorization metadata.
> > >> > > > >
> > >> > > > > 4) I was concerned about this too. The goal is to make the
> > API
> > >> > flexible
> > >> > > > > enough to handle large scale deployments in future when
> > caching
> > >> all
> > >> > > > > authorization metadata in each broker is not viable. Using
> > an
> > >> async
> > >> > API
> > >> > > > > that returns CompletionStage, the caller has the option to
> > handle
> > >> the
> > >> > > > > result synchronously or asynchronously, so we don't
> > necessarily
> > >> need
> > >> > to
> > >> > > > > update the calling code right away. Custom authorizers
> > using the
> > >> > async
> > >> > > > API
> > >> > > > > have full control over whether authorization is performed
> > in-line
> > >> > since
> > >> > > > > completedFuture will always be handled synchronously. We
> do
> > need
> > >> to
> > >> > > > update
> > >> > > > > KafkaApis to take advantage of the asynchronous API to
> > improve
> > >> scale.
> > >> > > > Even
> > >> > > > > though this is a big change, since we will be doing the
> > same for
> > >> all
> > >> > > > > requests, it shouldn't be too hard to maintain since the
> > same
> > >> pattern
> > >> > > > will
> > >> > > > > be used for all requests.
> > >> > > > >
> > >> > > > > Regards,
> > >> > > > >
> > >> > > > > Rajini
> > >> > > > >
> > >> > > > > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <
> > bosco@apache.org
> > >> >
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > > Hi Rajini
> > >> > > > > >
> > >> > > > > > Help me understand this a bit more.
> > >> > > > > >
> > >> > > > > > 1. For all practical purpose, without authorization you
> > can't
> > >> go to
> > >> > > the
> > >> > > > > > next step. The calling code needs to block anyway. So
> > should we
> > >> > just
> > >> > > > let
> > >> > > > > > the implementation code do the async part?
> > >> > > > > > 2. If you feel management calls need to be async, then
> we
> > should
> > >> > > > consider
> > >> > > > > > another set of async APIs. I don't feel we should
> > complicate it
> > >> > > > further (
> > >> > > > > > 3. Another concern I have is wrt performance. Kafka has
> > been
> > >> built
> > >> > to
> > >> > > > > > handle 1000s per second requests. Not sure whether
> making
> > it
> > >> async
> > >> > > will
> > >> > > > > add
> > >> > > > > > any unnecessary overhead.
> > >> > > > > > 4. How much complication would this add on the calling
> > side?
> > >> And is
> > >> > > it
> > >> > > > > > worth it?
> > >> > > > > >
> > >> > > > > > Thanks
> > >> > > > > >
> > >> > > > > > Bosco
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On 9/3/19, 8:50 AM, "Rajini Sivaram" <
> > rajinisivaram@gmail.com>
> > >> > > wrote:
> > >> > > > > >
> > >> > > > > > Hi all,
> > >> > > > > >
> > >> > > > > > Ismael brought up a point that it will be good to
> > make the
> > >> > > > Authorizer
> > >> > > > > > interface asynchronous to avoid blocking request
> > threads
> > >> during
> > >> > > > > remote
> > >> > > > > > operations.
> > >> > > > > >
> > >> > > > > > 1) Since we want to support different backends for
> > >> > authorization
> > >> > > > > > metadata,
> > >> > > > > > making createAcls() and deleteAcls() asynchronous
> > makes
> > >> sense
> > >> > > since
> > >> > > > > > these
> > >> > > > > > always involve remote operations. When KIP-500
> removes
> > >> > ZooKeeper,
> > >> > > > we
> > >> > > > > > would
> > >> > > > > > want to move ACLs to Kafka and async updates will
> > avoid
> > >> > > unnecessary
> > >> > > > > > blocking.
> > >> > > > > > 2) For authorize() method, we currently use cached
> > ACLs in
> > >> the
> > >> > > > > built-in
> > >> > > > > > authorizers, so synchronous authorise operations
> work
> > well
> > >> now.
> > >> > > But
> > >> > > > > > async
> > >> > > > > > authorize() would support this scenario as well as
> > >> authorizers
> > >> > in
> > >> > > > > large
> > >> > > > > > organisations where an LRU cache would enable a
> > smaller
> > >> cache
> > >> > > even
> > >> > > > > > when the
> > >> > > > > > backend holds a large amount of ACLs for
> infrequently
> > used
> > >> > > > resources
> > >> > > > > or
> > >> > > > > > users who don't use the system frequently.
> > >> > > > > >
> > >> > > > > > For both cases, the built-in authorizer will
> continue
> > to be
> > >> > > > > > synchronous,
> > >> > > > > > using CompletableFuture.completedFuture() to return
> > the
> > >> actual
> > >> > > > > result.
> > >> > > > > > But
> > >> > > > > > async API will make custom authorizer
> implementations
> > more
> > >> > > > flexible.
> > >> > > > > I
> > >> > > > > > would like to know if there are any concerns with
> > these
> > >> changes
> > >> > > > > before
> > >> > > > > > updating the KIP.
> > >> > > > > >
> > >> > > > > > *Proposed API:*
> > >> > > > > > public interface Authorizer extends Configurable,
> > Closeable
> > >> {
> > >> > > > > >
> > >> > > > > > Map<Endpoint, CompletionStage<Void>>
> > >> > > start(AuthorizerServerInfo
> > >> > > > > > serverInfo);
> > >> > > > > > List<CompletionStage<AuthorizationResult>>
> > >> > > > > > authorize(AuthorizableRequestContext requestContext,
> > >> > List<Action>
> > >> > > > > > actions);
> > >> > > > > > List<CompletionStage<AclCreateResult>>
> > >> > > > > > createAcls(AuthorizableRequestContext
> requestContext,
> > >> > > > > List<AclBinding>
> > >> > > > > > aclBindings);
> > >> > > > > > List<CompletionStage<AclDeleteResult>>
> > >> > > > > > deleteAcls(AuthorizableRequestContext
> requestContext,
> > >> > > > > > List<AclBindingFilter> aclBindingFilters);
> > >> > > > > > CompletionStage<Collection<AclBinding>>
> > >> > acls(AclBindingFilter
> > >> > > > > > filter);
> > >> > > > > > }
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Thank you,
> > >> > > > > >
> > >> > > > > > Rajini
> > >> > > > > >
> > >> > > > > > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <
> > >> > > bosco@apache.org>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi Rajini
> > >> > > > > > >
> > >> > > > > > > Thanks for clarifying. I am good for now.
> > >> > > > > > >
> > >> > > > > > > Regards
> > >> > > > > > >
> > >> > > > > > > Bosco
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <
> > >> > > rajinisivaram@gmail.com
> > >> > > > >
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > Hi Don,
> > >> > > > > > >
> > >> > > > > > > That should be fine. I guess Ranger loads
> > policies
> > >> from
> > >> > the
> > >> > > > > > database
> > >> > > > > > > synchronously when the authorizer is
> configured,
> > >> similar
> > >> > to
> > >> > > > > > > SimpleAclAuthorizer loading from ZooKeeper.
> > Ranger can
> > >> > > > continue
> > >> > > > > > to load
> > >> > > > > > > synchronously from `configure()` or `start()`
> > and
> > >> return
> > >> > an
> > >> > > > > > empty map
> > >> > > > > > > from
> > >> > > > > > > `start()`. That would retain the existing
> > behaviour..
> > >> > When
> > >> > > > the
> > >> > > > > > same
> > >> > > > > > > database stores policies for all listeners and
> > the
> > >> > policies
> > >> > > > are
> > >> > > > > > not
> > >> > > > > > > stored
> > >> > > > > > > in Kafka, there is no value in making the load
> > >> > > asynchronous.
> > >> > > > > > >
> > >> > > > > > > Regards,
> > >> > > > > > >
> > >> > > > > > > Rajini
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco
> Durai
> > <
> > >> > > > > > bosco@apache.org>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi Rajini
> > >> > > > > > > >
> > >> > > > > > > > Assuming this doesn't affect custom plugins,
> > I don't
> > >> > have
> > >> > > > any
> > >> > > > > > > concerns
> > >> > > > > > > > regarding this.
> > >> > > > > > > >
> > >> > > > > > > > I do have one question regarding:
> > >> > > > > > > >
> > >> > > > > > > > "For authorizers that don’t store metadata
> in
> > >> > ZooKeeper,
> > >> > > > > > ensure that
> > >> > > > > > > > authorizer metadata for each listener is
> > available
> > >> > before
> > >> > > > > > starting
> > >> > > > > > > up the
> > >> > > > > > > > listener. This enables different
> authorization
> > >> metadata
> > >> > > > > stores
> > >> > > > > > for
> > >> > > > > > > > different listeners."
> > >> > > > > > > >
> > >> > > > > > > > Since Ranger uses its own database for
> storing
> > >> > policies,
> > >> > > do
> > >> > > > > you
> > >> > > > > > > anticipate
> > >> > > > > > > > any issues?
> > >> > > > > > > >
> > >> > > > > > > > Thanks
> > >> > > > > > > >
> > >> > > > > > > > Bosco
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
> > >> > > > > > rajinisivaram@gmail.com>
> > >> > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > Hi all,
> > >> > > > > > > >
> > >> > > > > > > > I made another change to the KIP. The
> KIP
> > was
> > >> > > > originally
> > >> > > > > > > proposing to
> > >> > > > > > > > extend SimpleAclAuthorizer to also
> > implement the
> > >> > new
> > >> > > > API
> > >> > > > > > (in
> > >> > > > > > > addition
> > >> > > > > > > > to
> > >> > > > > > > > the existing API). But since we use the
> > new API
> > >> > when
> > >> > > > > > available,
> > >> > > > > > > this
> > >> > > > > > > > can
> > >> > > > > > > > break custom authorizers that extend
> this
> > class
> > >> and
> > >> > > > > > override
> > >> > > > > > > specific
> > >> > > > > > > > methods of the old API. To avoid
> breaking
> > any
> > >> > > existing
> > >> > > > > > custom
> > >> > > > > > > > implementations that extend this class,
> > >> > particularly
> > >> > > > > > because it
> > >> > > > > > > is in
> > >> > > > > > > > the
> > >> > > > > > > > public package kafka.security.auth, the
> > KIP now
> > >> > > > proposes
> > >> > > > > to
> > >> > > > > > > retain the
> > >> > > > > > > > old
> > >> > > > > > > > authorizer as-is. SimpleAclAuthorizer
> > will
> > >> > continue
> > >> > > to
> > >> > > > > > > implement the
> > >> > > > > > > > old
> > >> > > > > > > > API, but will be deprecated. A new
> > authorizer
> > >> > > > > > implementation
> > >> > > > > > > > kafka.security.authorizer.AclAuthorizer
> > will be
> > >> > added
> > >> > > > for
> > >> > > > > > the
> > >> > > > > > > new API
> > >> > > > > > > > (this
> > >> > > > > > > > will not be in the public package).
> > >> > > > > > > >
> > >> > > > > > > > Please let me know if you have any
> > concerns.
> > >> > > > > > > >
> > >> > > > > > > > Regards,
> > >> > > > > > > >
> > >> > > > > > > > Rajini
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Fri, Aug 16, 2019 at 8:48 AM Rajini
> > Sivaram <
> > >> > > > > > > > rajinisivaram@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Thanks Colin.
> > >> > > > > > > > >
> > >> > > > > > > > > If there are no other concerns, I will
> > start
> > >> vote
> > >> > > > later
> > >> > > > > > today.
> > >> > > > > > > Many
> > >> > > > > > > > thanks
> > >> > > > > > > > > to every one for the feedback.
> > >> > > > > > > > >
> > >> > > > > > > > > Regards,
> > >> > > > > > > > >
> > >> > > > > > > > > Rajini
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Thu, Aug 15, 2019 at 11:57 PM Colin
> > McCabe
> > >> <
> > >> > > > > > > cmccabe@apache.org>
> > >> > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > >> Thanks, Rajini. It looks good to me.
> > >> > > > > > > > >>
> > >> > > > > > > > >> best,
> > >> > > > > > > > >> Colin
> > >> > > > > > > > >>
> > >> > > > > > > > >>
> > >> > > > > > > > >> On Thu, Aug 15, 2019, at 11:37,
> Rajini
> > >> Sivaram
> > >> > > > wrote:
> > >> > > > > > > > >> > Hi Colin,
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Thanks for the review. I have
> > updated the
> > >> KIP
> > >> > to
> > >> > > > > move
> > >> > > > > > the
> > >> > > > > > > > interfaces for
> > >> > > > > > > > >> > request context and server info to
> > the
> > >> > > authorizer
> > >> > > > > > package.
> > >> > > > > > > These
> > >> > > > > > > > are now
> > >> > > > > > > > >> > called AuthorizableRequestContext
> and
> > >> > > > > > AuthorizerServerInfo.
> > >> > > > > > > > Endpoint is
> > >> > > > > > > > >> now
> > >> > > > > > > > >> > a class in org.apache.kafka.common
> > to make
> > >> it
> > >> > > > > reusable
> > >> > > > > > > since we
> > >> > > > > > > > already
> > >> > > > > > > > >> > have multiple implementations of
> it.
> > I have
> > >> > > > removed
> > >> > > > > > > requestName
> > >> > > > > > > > from the
> > >> > > > > > > > >> > request context interface since
> > authorizers
> > >> > can
> > >> > > > > > distinguish
> > >> > > > > > > > follower
> > >> > > > > > > > >> fetch
> > >> > > > > > > > >> > and consumer fetch from the
> > operation being
> > >> > > > > > authorized. So
> > >> > > > > > > 16-bit
> > >> > > > > > > > >> request
> > >> > > > > > > > >> > type should be sufficient for audit
> > >> logging.
> > >> > > Also
> > >> > > > > > replaced
> > >> > > > > > > > AuditFlag
> > >> > > > > > > > >> with
> > >> > > > > > > > >> > two booleans as you suggested.
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Can you take another look and see
> if
> > the
> > >> KIP
> > >> > is
> > >> > > > > ready
> > >> > > > > > for
> > >> > > > > > > voting?
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Thanks for all your help!
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Regards,
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Rajini
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > On Wed, Aug 14, 2019 at 8:59 PM
> Colin
> > >> McCabe <
> > >> > > > > > > cmccabe@apache.org>
> > >> > > > > > > > >> wrote:
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > > Hi Rajini,
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > I think it would be good to
> rename
> > >> > > > > > KafkaRequestContext to
> > >> > > > > > > > something
> > >> > > > > > > > >> like
> > >> > > > > > > > >> > > AuthorizableRequestContext, and
> > put it in
> > >> > the
> > >> > > > > > > > >> > >
> org.apache.kafka.server.authorizer
> > >> > namespace.
> > >> > > > If
> > >> > > > > > we put
> > >> > > > > > > it in
> > >> > > > > > > > the
> > >> > > > > > > > >> > > org.apache.kafka.common
> namespace,
> > then
> > >> it's
> > >> > > not
> > >> > > > > > really
> > >> > > > > > > clear
> > >> > > > > > > > that
> > >> > > > > > > > >> it's
> > >> > > > > > > > >> > > part of the Authorizer API.
> Since
> > this
> > >> > class
> > >> > > is
> > >> > > > > > purely an
> > >> > > > > > > > interface,
> > >> > > > > > > > >> with
> > >> > > > > > > > >> > > no concrete implementation of
> > anything,
> > >> > > there's
> > >> > > > > > nothing
> > >> > > > > > > common
> > >> > > > > > > > to
> > >> > > > > > > > >> really
> > >> > > > > > > > >> > > reuse in any case. We definitely
> > don't
> > >> want
> > >> > > > > > someone to
> > >> > > > > > > > accidentally
> > >> > > > > > > > >> add or
> > >> > > > > > > > >> > > remove methods because they think
> > this is
> > >> > just
> > >> > > > > > another
> > >> > > > > > > internal
> > >> > > > > > > > class
> > >> > > > > > > > >> used
> > >> > > > > > > > >> > > for requests.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > The BrokerInfo class is a nice
> > >> improvement.
> > >> > > It
> > >> > > > > > looks
> > >> > > > > > > like it
> > >> > > > > > > > will be
> > >> > > > > > > > >> > > useful for passing in information
> > about
> > >> the
> > >> > > > > context
> > >> > > > > > we're
> > >> > > > > > > > running
> > >> > > > > > > > >> in. It
> > >> > > > > > > > >> > > would be nice to call this class
> > >> ServerInfo
> > >> > > > rather
> > >> > > > > > than
> > >> > > > > > > > BrokerInfo,
> > >> > > > > > > > >> since
> > >> > > > > > > > >> > > we will want to run the
> authorizer
> > on
> > >> > > > controllers
> > >> > > > > > as well
> > >> > > > > > > as on
> > >> > > > > > > > >> brokers,
> > >> > > > > > > > >> > > and the controller may run as a
> > separate
> > >> > > process
> > >> > > > > > post
> > >> > > > > > > KIP-500.
> > >> > > > > > > > I also
> > >> > > > > > > > >> > > think that this class should be
> in
> > the
> > >> > > > > > > > >> org.apache.kafka.server.authorizer
> > >> > > > > > > > >> > > namespace. Again, it is an
> > interface,
> > >> not a
> > >> > > > > > concrete
> > >> > > > > > > > implementation,
> > >> > > > > > > > >> and
> > >> > > > > > > > >> > > it's an interface that is very
> > >> specifically
> > >> > > for
> > >> > > > > the
> > >> > > > > > > authorizer.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > I agree that we probably don't
> have
> > >> enough
> > >> > > > > > information
> > >> > > > > > > > preserved for
> > >> > > > > > > > >> > > requests currently to always know
> > what
> > >> > entity
> > >> > > > made
> > >> > > > > > them.
> > >> > > > > > > So we
> > >> > > > > > > > can
> > >> > > > > > > > >> leave
> > >> > > > > > > > >> > > that out for now (except in the
> > special
> > >> case
> > >> > > of
> > >> > > > > > Fetch).
> > >> > > > > > > > Perhaps we
> > >> > > > > > > > >> can add
> > >> > > > > > > > >> > > this later if it's needed.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > I understand the intention behind
> > >> > > > > AuthorizationMode
> > >> > > > > > > (which is
> > >> > > > > > > > now
> > >> > > > > > > > >> called
> > >> > > > > > > > >> > > AuditFlag in the latest
> > revision). But
> > >> it
> > >> > > still
> > >> > > > > > feels
> > >> > > > > > > > complex. What
> > >> > > > > > > > >> if we
> > >> > > > > > > > >> > > just had two booleans in Action:
> > >> > logSuccesses
> > >> > > > and
> > >> > > > > > > logFailures?
> > >> > > > > > > > That
> > >> > > > > > > > >> seems
> > >> > > > > > > > >> > > to cover all the cases here.
> > >> > > > MANDATORY_AUTHORIZE
> > >> > > > > =
> > >> > > > > > true,
> > >> > > > > > > true.
> > >> > > > > > > > >> > > OPTIONAL_AUTHORIZE = true, false.
> > >> FILTER =
> > >> > > > true,
> > >> > > > > > false.
> > >> > > > > > > > >> LIST_AUTHORIZED =
> > >> > > > > > > > >> > > false, false. Would there be
> > anything
> > >> lost
> > >> > > > versus
> > >> > > > > > having
> > >> > > > > > > the
> > >> > > > > > > > enum?
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > best,
> > >> > > > > > > > >> > > Colin
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > On Wed, Aug 14, 2019, at 06:29,
> > Mickael
> > >> > Maison
> > >> > > > > > wrote:
> > >> > > > > > > > >> > > > Hi Rajini,
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > Thanks for the KIP!
> > >> > > > > > > > >> > > > I really like that authorize()
> > will be
> > >> > able
> > >> > > to
> > >> > > > > > take a
> > >> > > > > > > batch of
> > >> > > > > > > > >> > > > requests, this will speed up
> many
> > >> > > > > implementations!
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM
> > Rajini
> > >> > > > Sivaram <
> > >> > > > > > > > >> rajinisivaram@gmail.com>
> > >> > > > > > > > >> > > wrote:
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > Thanks David! I have fixed
> the
> > typo.
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > Also made a couple of changes
> > to make
> > >> > the
> > >> > > > > > context
> > >> > > > > > > > interfaces more
> > >> > > > > > > > >> > > generic.
> > >> > > > > > > > >> > > > > KafkaRequestContext now
> > returns the
> > >> > 16-bit
> > >> > > > API
> > >> > > > > > key as
> > >> > > > > > > Colin
> > >> > > > > > > > >> suggested
> > >> > > > > > > > >> > > as
> > >> > > > > > > > >> > > > > well as the friendly name
> used
> > in
> > >> > metrics
> > >> > > > > which
> > >> > > > > > are
> > >> > > > > > > useful
> > >> > > > > > > > in
> > >> > > > > > > > >> audit
> > >> > > > > > > > >> > > logs.
> > >> > > > > > > > >> > > > > `Authorizer#start` is now
> > provided a
> > >> > > generic
> > >> > > > > > > `BrokerInfo`
> > >> > > > > > > > >> interface
> > >> > > > > > > > >> > > that
> > >> > > > > > > > >> > > > > gives cluster id, broker id
> and
> > >> endpoint
> > >> > > > > > information.
> > >> > > > > > > The
> > >> > > > > > > > generic
> > >> > > > > > > > >> > > interface
> > >> > > > > > > > >> > > > > can potentially be used in
> > other
> > >> broker
> > >> > > > > plugins
> > >> > > > > > in
> > >> > > > > > > future
> > >> > > > > > > > and
> > >> > > > > > > > >> provides
> > >> > > > > > > > >> > > > > dynamically generated configs
> > like
> > >> > broker
> > >> > > id
> > >> > > > > > and ports
> > >> > > > > > > > which are
> > >> > > > > > > > >> > > currently
> > >> > > > > > > > >> > > > > not available to plugins
> > unless these
> > >> > > > configs
> > >> > > > > > are
> > >> > > > > > > statically
> > >> > > > > > > > >> > > configured.
> > >> > > > > > > > >> > > > > Please let me know if there
> > are any
> > >> > > > concerns.
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > Regards,
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > Rajini
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > On Tue, Aug 13, 2019 at 4:30
> > PM David
> > >> > > Jacot
> > >> > > > <
> > >> > > > > > > > djacot@confluent.io>
> > >> > > > > > > > >> > > wrote:
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > > Hi Rajini,
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > Thank you for the update!
> It
> > looks
> > >> > good
> > >> > > to
> > >> > > > > me.
> > >> > > > > > > There is a
> > >> > > > > > > > typo
> > >> > > > > > > > >> in the
> > >> > > > > > > > >> > > > > > `AuditFlag` enum:
> > >> > `MANDATORY_AUTHOEIZE`
> > >> > > ->
> > >> > > > > > > > >> `MANDATORY_AUTHORIZE`.
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > Regards,
> > >> > > > > > > > >> > > > > > David
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > On Mon, Aug 12, 2019 at
> 2:54
> > PM
> > >> Rajini
> > >> > > > > > Sivaram <
> > >> > > > > > > > >> > > rajinisivaram@gmail.com>
> > >> > > > > > > > >> > > > > > wrote:
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > > Hi David,
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > Thanks for reviewing the
> > KIP!
> > >> Since
> > >> > > > > > questions
> > >> > > > > > > about
> > >> > > > > > > > >> `authorization
> > >> > > > > > > > >> > > mode`
> > >> > > > > > > > >> > > > > > > and `count` have come up
> > multiple
> > >> > > > times, I
> > >> > > > > > have
> > >> > > > > > > renamed
> > >> > > > > > > > both.
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > 1) Renamed `count` to
> > >> > > > > > `resourceReferenceCount`.
> > >> > > > > > > It is
> > >> > > > > > > > the
> > >> > > > > > > > >> number
> > >> > > > > > > > >> > > of times
> > >> > > > > > > > >> > > > > > > the resource being
> > authorized is
> > >> > > > > referenced
> > >> > > > > > > within the
> > >> > > > > > > > >> request.
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > 2) Renamed
> > `AuthorizationMode` to
> > >> > > > > > `AuditFlag`. It
> > >> > > > > > > is
> > >> > > > > > > > provided
> > >> > > > > > > > >> to
> > >> > > > > > > > >> > > improve
> > >> > > > > > > > >> > > > > > > audit logging in the
> > authorizer.
> > >> The
> > >> > > > enum
> > >> > > > > > values
> > >> > > > > > > have
> > >> > > > > > > > javadoc
> > >> > > > > > > > >> which
> > >> > > > > > > > >> > > > > > > indicate how the
> > authorization
> > >> > result
> > >> > > is
> > >> > > > > > used in
> > >> > > > > > > each
> > >> > > > > > > > of the
> > >> > > > > > > > >> modes
> > >> > > > > > > > >> > > to
> > >> > > > > > > > >> > > > > > > enable authorizers to log
> > audit
> > >> > > messages
> > >> > > > > at
> > >> > > > > > the
> > >> > > > > > > right
> > >> > > > > > > > severity
> > >> > > > > > > > >> > > level.
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > Regards,
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > Rajini
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > On Mon, Aug 12, 2019 at
> > 5:54 PM
> > >> > David
> > >> > > > > Jacot
> > >> > > > > > <
> > >> > > > > > > > >> djacot@confluent.io>
> > >> > > > > > > > >> > > wrote:
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > > Hi Rajini,
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > Thank you for the KIP.
> > >> Overall, it
> > >> > > > looks
> > >> > > > > > good
> > >> > > > > > > to me.
> > >> > > > > > > > I have
> > >> > > > > > > > >> few
> > >> > > > > > > > >> > > > > > > > questions/suggestions:
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > 1. It is hard to grasp
> > what
> > >> > > > > > `Action#count` is
> > >> > > > > > > for. I
> > >> > > > > > > > guess I
> > >> > > > > > > > >> > > understand
> > >> > > > > > > > >> > > > > > > > where you want to go
> > with it
> > >> but
> > >> > it
> > >> > > > took
> > >> > > > > > me a
> > >> > > > > > > while to
> > >> > > > > > > > >> figure it
> > >> > > > > > > > >> > > out.
> > >> > > > > > > > >> > > > > > > > Perhaps, we could come
> > up with
> > >> a
> > >> > > > better
> > >> > > > > > name
> > >> > > > > > > than
> > >> > > > > > > > `count`?
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > 2. I had a hard time
> > trying to
> > >> > > > > understand
> > >> > > > > > the
> > >> > > > > > > > >> `AuthorizationMode`
> > >> > > > > > > > >> > > > > > > concept,
> > >> > > > > > > > >> > > > > > > > especially wrt. the
> > OPTIONAL
> > >> one.
> > >> > My
> > >> > > > > > > understanding is
> > >> > > > > > > > that
> > >> > > > > > > > >> an
> > >> > > > > > > > >> > > ACL is
> > >> > > > > > > > >> > > > > > > either
> > >> > > > > > > > >> > > > > > > > defined or not. Could
> you
> > >> > elaborate
> > >> > > a
> > >> > > > > bit
> > >> > > > > > more
> > >> > > > > > > on
> > >> > > > > > > > that?
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > Thanks,
> > >> > > > > > > > >> > > > > > > > David
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > On Fri, Aug 9, 2019 at
> > 10:26 PM
> > >> > Don
> > >> > > > > Bosco
> > >> > > > > > Durai
> > >> > > > > > > <
> > >> > > > > > > > >> > > bosco@apache.org>
> > >> > > > > > > > >> > > > > > > wrote:
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Hi Rajini
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > 3.2 - This makes
> sense.
> > >> Thanks
> > >> > for
> > >> > > > > > clarifying.
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Rest looks fine. Once
> > the
> > >> > > > > > implementations are
> > >> > > > > > > done,
> > >> > > > > > > > it
> > >> > > > > > > > >> will be
> > >> > > > > > > > >> > > more
> > >> > > > > > > > >> > > > > > > clear
> > >> > > > > > > > >> > > > > > > > > on the different
> values
> > >> > > RequestType
> > >> > > > > and
> > >> > > > > > Mode.
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Thanks
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Bosco
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > On 8/9/19, 5:08 AM,
> > "Rajini
> > >> > > > Sivaram"
> > >> > > > > <
> > >> > > > > > > > >> rajinisivaram@gmail.com
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > > > wrote:
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Hi Don,
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Thanks for the
> > >> suggestions.
> > >> > A
> > >> > > > few
> > >> > > > > > > responses
> > >> > > > > > > > below:
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > 3.1 Can rename
> and
> > >> improve
> > >> > > docs
> > >> > > > if
> > >> > > > > > we keep
> > >> > > > > > > > this. Let's
> > >> > > > > > > > >> > > finish the
> > >> > > > > > > > >> > > > > > > > > discussion on
> > Colin's
> > >> > > > suggestions
> > >> > > > > > > regarding this
> > >> > > > > > > > >> first.
> > >> > > > > > > > >> > > > > > > > > 3.2 No, I was
> > thinking of
> > >> > some
> > >> > > > > > requests
> > >> > > > > > > that
> > >> > > > > > > > have an
> > >> > > > > > > > >> old
> > >> > > > > > > > >> > > way of
> > >> > > > > > > > >> > > > > > > > > authorizing
> > >> > > > > > > > >> > > > > > > > > and a new way
> > where we
> > >> have
> > >> > > > > > retained the
> > >> > > > > > > old
> > >> > > > > > > > way for
> > >> > > > > > > > >> > > backward
> > >> > > > > > > > >> > > > > > > > > compatibility.
> One
> > >> example
> > >> > is
> > >> > > > > > > Cluster:Create
> > >> > > > > > > > >> permission to
> > >> > > > > > > > >> > > create
> > >> > > > > > > > >> > > > > > > > > topics.
> > >> > > > > > > > >> > > > > > > > > We have replaced
> > this
> > >> with
> > >> > > > > > fine-grained
> > >> > > > > > > topic
> > >> > > > > > > > create
> > >> > > > > > > > >> > > access using
> > >> > > >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Ismael Juma <is...@juma.me.uk>.
It's not uncommon for people to do large numbers of admin operations per
second. We've seen it many times in production. It's often a bug, but the
system must cope with it without affecting other users of the same Kafka
cluster.
Ismael
On Fri, Sep 6, 2019, 9:32 AM Don Bosco Durai <bo...@apache.org> wrote:
> +1
>
> I might be wrong here, but here are few of my assumptions..
>
> 1. Unlike transactional services, in Kafka most of the users are
> application users, so once the connection is established, then they are
> going to be long running sessions. So the caching is more predictable and
> heavy lifting can be done during initial session setup and cache can be
> frequently updated using background thread. KSQL might have a different
> flow, but I don't have enough datapoints for it.
> 2. For publish/consume authorization calls, providing async option will be
> misleading for Kafka like load
> 3. For admin calls, hopefully no one does hundreds of call in multiple
> thread within few seconds. This would be a bad design.
>
> So I feel, we should be okay with sync calls for now. And consider async
> in the future if needed. At least in Apache Ranger implementation, we would
> be okay using sync operation. I am not sure if any other plugin
> implementation would benefit from async implementation.
>
> Thanks
>
> Bosco
>
> On 9/6/19, 8:45 AM, "Ismael Juma" <is...@juma.me.uk> wrote:
>
> One more thing: if people have strong opinions that authorize has to be
> sync, we could leave it like that for now. If needed, we can later add
> an
> authorizeAsync with another method returning a Boolean indicating that
> it's
> supported. That is, there is a path to evolve the interface (even if a
> bit
> ugly).
>
> What about the other methods, is there consensus that they should be
> updated to return CompletionStage?
>
> Ismael
>
> On Fri, Sep 6, 2019, 8:32 AM Ismael Juma <is...@juma.me.uk> wrote:
>
> > I'm on the move, but what Rajini said seems reasonable. I don't think
> > using SSDs solves the issue. They can still hang for seconds when
> they
> > fail. Also, many people may not have local SSDs available (remote
> SSDs like
> > EBS hang for tens of seconds when there are issues).
> >
> > We are currently vulnerable to all of these in the normal read/write
> path,
> > but it seems suboptimal to continue assuming non blocking behavior
> for
> > things that actually do block.
> >
> > Ismael
> >
> > On Fri, Sep 6, 2019, 8:24 AM Rajini Sivaram <rajinisivaram@gmail.com
> >
> > wrote:
> >
> >> Hi Jun,
> >>
> >> For ACLs, the size is also dependent on the number of users. So in
> >> deployments with large number of users where some users are not
> active, an
> >> LRU cache may be useful. Agree that there may be other solutions
> that help
> >> avoid the requirement for an async API even for this case.
> >>
> >> I wasn't expecting our threading model to change very much. Having
> said
> >> that, I haven't figured out how the purgatory can be adapted to work
> >> efficiently for this. I was thinking:
> >> 1) If authorization future is complete when returning from
> authorize()
> >> call, we invoke the method immediately on the request thread
> >> 2) If future is not complete, the request goes into a purgatory
> >> 3) When future completes, we mark the request ready to be run. This
> can be
> >> done on ForkJoinPool.commonPool or an executor. But we don't handle
> the
> >> request on that thread
> >> 4) We handle the request on the request thread after 3).
> >>
> >> I totally agree that this adds complexity to the code. Will wait for
> >> Ismael
> >> to comment on whether he had a different model in mind.
> >>
> >> Thanks,
> >>
> >> Rajini
> >>
> >>
> >> On Thu, Sep 5, 2019 at 10:29 PM Jun Rao <ju...@confluent.io> wrote:
> >>
> >> > Hi, Ismael,
> >> >
> >> > Yes, I agree that there are 2 separate discussion points, one on
> the API
> >> > and the other on the implementation.
> >> >
> >> > First on the API, in general, my feeling is that the authorize()
> call is
> >> > expected to be fast to reduce request latency. Adding an async API
> >> could be
> >> > mis-leading since it might give users the impression that it
> would be
> >> > arbitrarily long. In the short term, currently, we cache all
> partition
> >> > level metadata on every broker. The authorization metadata is
> >> proportional
> >> > to the number of topics, which typically is less in size. So it
> we can
> >> > cache all partition level metadata, we should be able to cache all
> >> > authorization metadata. Longer term, if we want to support more
> metadata
> >> > beyond memory, another option is to put all metadata in an SSD
> backed
> >> > key/value store. The lookup time could still be under 1ms, which
> we
> >> could
> >> > potentially absorb in the request threads. So, I am not sure async
> >> > authorize() api is strictly needed even in the long term.
> >> >
> >> > Second on the implementation. I was more concerned about the
> impact on
> >> our
> >> > threading model. If we make authorize() async, some threads have
> to
> >> > complete the future when it's ready. It seems to me that those
> threads
> >> have
> >> > to be inside the Authorizer implementation since only it knows
> when to
> >> > complete a future? If so, it feels weird that all the logic of
> request
> >> > handling post authorize() is now handled by external threads
> inside a
> >> > plugin. It's not clear how we can configure and monitor them, and
> how
> >> > throttling would work. Perhaps we can provide a bit more detail
> on the
> >> new
> >> > threading model.
> >> >
> >> > Thanks,
> >> >
> >> > Jun
> >> >
> >> > On Wed, Sep 4, 2019 at 9:56 PM Ismael Juma <is...@juma.me.uk>
> wrote:
> >> >
> >> > > Hi Jun,
> >> > >
> >> > > I think it's important to discuss the pluggable API versus the
> calling
> >> > > implementation as two separate points.
> >> > >
> >> > > From an API perspective, are you suggesting that we should tell
> users
> >> > that
> >> > > they cannot block in authorize? Or are you saying that it's OK
> to
> >> block
> >> > on
> >> > > authorize on occasion and the recommendation would be for
> people to
> >> > > increase the number of threads in the request thread pool?
> >> > Architecturally,
> >> > > this feels wrong in my opinion.
> >> > >
> >> > > From the calling implementation perspective, you don't need a
> >> callback.
> >> > You
> >> > > can basically say:
> >> > >
> >> > > authorize(...).map { result =>
> >> > > ...
> >> > > }
> >> > >
> >> > > And then have a common method take that Future and handle it
> >> > synchronously
> >> > > if it's already complete or submit it to the purgatory if not.
> This is
> >> > > similar to how many modern async web libraries work.
> >> > >
> >> > > As Rajini said, this could be done later. The initial
> implementation
> >> > could
> >> > > simply do `toCompletableFuture().get()` to do it synchronously.
> But it
> >> > > would mean that we could add this functionality without
> changing the
> >> > > pluggable interface.
> >> > >
> >> > > Ismael
> >> > >
> >> > > On Wed, Sep 4, 2019 at 5:29 AM Jun Rao <ju...@confluent.io>
> wrote:
> >> > >
> >> > > > Hi, Rajini,
> >> > > >
> >> > > > Thanks for the KIP. I was concerned about #4 too. If we
> change the
> >> > > handling
> >> > > > of all requests to use an async authorize() api, will that
> cause the
> >> > code
> >> > > > much harder to understand? There are quite a few callbacks
> already.
> >> I
> >> > am
> >> > > > not sure that we want to introduce more of those. The benefit
> from
> >> > async
> >> > > > authorize() api seems limited.
> >> > > >
> >> > > > Jun
> >> > > >
> >> > > > On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <
> >> rajinisivaram@gmail.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Don,
> >> > > > >
> >> > > > > Thanks for your note.
> >> > > > >
> >> > > > > 1) The intention is to avoid blocking in the calling
> thread. We
> >> > already
> >> > > > > have several requests that are put into a purgatory when
> waiting
> >> for
> >> > > > remote
> >> > > > > communication, for example produce request waiting for
> >> replication.
> >> > > Since
> >> > > > > we have a limited number of request threads in the broker,
> we
> >> want to
> >> > > > make
> >> > > > > progress with other requests, while one is waiting on any
> form of
> >> > > remote
> >> > > > > communication.
> >> > > > >
> >> > > > > 2) Async management calls will be useful with the default
> >> authorizer
> >> > > when
> >> > > > > KIP-500 removes ZK and we rely on Kafka instead. Our current
> >> ZK-based
> >> > > > > implementation as well as any custom implementations that
> don't
> >> want
> >> > to
> >> > > > be
> >> > > > > async will just need to return a sync'ed value. So instead
> of
> >> > > returning `
> >> > > > > value`, the code would just return
> >> > > > > `CompletableFuture.completedFuture(value)
> >> > > > > `. So it would be just a single line change in the
> implementation
> >> > with
> >> > > > the
> >> > > > > new API. The caller would treat completedFuture exactly as
> it does
> >> > > today,
> >> > > > > processing the request synchronously without using a
> purgatory.
> >> > > > >
> >> > > > > 3) For implementations that return a completedFuture as
> described
> >> in
> >> > > 2),
> >> > > > > the behaviour would remain exactly the same. No additional
> >> threads or
> >> > > > > purgatory will be used for this case. So there would be no
> >> > performance
> >> > > > > penalty. For implementations that return a future that is
> not
> >> > complete,
> >> > > > we
> >> > > > > prioritise running more requests concurrently. So in a
> deployment
> >> > with
> >> > > a
> >> > > > > large number of clients, we would improve performance by
> allowing
> >> > other
> >> > > > > requests to be processed on the request threads while some
> are
> >> > waiting
> >> > > > for
> >> > > > > authorization metadata.
> >> > > > >
> >> > > > > 4) I was concerned about this too. The goal is to make the
> API
> >> > flexible
> >> > > > > enough to handle large scale deployments in future when
> caching
> >> all
> >> > > > > authorization metadata in each broker is not viable. Using
> an
> >> async
> >> > API
> >> > > > > that returns CompletionStage, the caller has the option to
> handle
> >> the
> >> > > > > result synchronously or asynchronously, so we don't
> necessarily
> >> need
> >> > to
> >> > > > > update the calling code right away. Custom authorizers
> using the
> >> > async
> >> > > > API
> >> > > > > have full control over whether authorization is performed
> in-line
> >> > since
> >> > > > > completedFuture will always be handled synchronously. We do
> need
> >> to
> >> > > > update
> >> > > > > KafkaApis to take advantage of the asynchronous API to
> improve
> >> scale.
> >> > > > Even
> >> > > > > though this is a big change, since we will be doing the
> same for
> >> all
> >> > > > > requests, it shouldn't be too hard to maintain since the
> same
> >> pattern
> >> > > > will
> >> > > > > be used for all requests.
> >> > > > >
> >> > > > > Regards,
> >> > > > >
> >> > > > > Rajini
> >> > > > >
> >> > > > > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <
> bosco@apache.org
> >> >
> >> > > > wrote:
> >> > > > >
> >> > > > > > Hi Rajini
> >> > > > > >
> >> > > > > > Help me understand this a bit more.
> >> > > > > >
> >> > > > > > 1. For all practical purpose, without authorization you
> can't
> >> go to
> >> > > the
> >> > > > > > next step. The calling code needs to block anyway. So
> should we
> >> > just
> >> > > > let
> >> > > > > > the implementation code do the async part?
> >> > > > > > 2. If you feel management calls need to be async, then we
> should
> >> > > > consider
> >> > > > > > another set of async APIs. I don't feel we should
> complicate it
> >> > > > further (
> >> > > > > > 3. Another concern I have is wrt performance. Kafka has
> been
> >> built
> >> > to
> >> > > > > > handle 1000s per second requests. Not sure whether making
> it
> >> async
> >> > > will
> >> > > > > add
> >> > > > > > any unnecessary overhead.
> >> > > > > > 4. How much complication would this add on the calling
> side?
> >> And is
> >> > > it
> >> > > > > > worth it?
> >> > > > > >
> >> > > > > > Thanks
> >> > > > > >
> >> > > > > > Bosco
> >> > > > > >
> >> > > > > >
> >> > > > > > On 9/3/19, 8:50 AM, "Rajini Sivaram" <
> rajinisivaram@gmail.com>
> >> > > wrote:
> >> > > > > >
> >> > > > > > Hi all,
> >> > > > > >
> >> > > > > > Ismael brought up a point that it will be good to
> make the
> >> > > > Authorizer
> >> > > > > > interface asynchronous to avoid blocking request
> threads
> >> during
> >> > > > > remote
> >> > > > > > operations.
> >> > > > > >
> >> > > > > > 1) Since we want to support different backends for
> >> > authorization
> >> > > > > > metadata,
> >> > > > > > making createAcls() and deleteAcls() asynchronous
> makes
> >> sense
> >> > > since
> >> > > > > > these
> >> > > > > > always involve remote operations. When KIP-500 removes
> >> > ZooKeeper,
> >> > > > we
> >> > > > > > would
> >> > > > > > want to move ACLs to Kafka and async updates will
> avoid
> >> > > unnecessary
> >> > > > > > blocking.
> >> > > > > > 2) For authorize() method, we currently use cached
> ACLs in
> >> the
> >> > > > > built-in
> >> > > > > > authorizers, so synchronous authorise operations work
> well
> >> now.
> >> > > But
> >> > > > > > async
> >> > > > > > authorize() would support this scenario as well as
> >> authorizers
> >> > in
> >> > > > > large
> >> > > > > > organisations where an LRU cache would enable a
> smaller
> >> cache
> >> > > even
> >> > > > > > when the
> >> > > > > > backend holds a large amount of ACLs for infrequently
> used
> >> > > > resources
> >> > > > > or
> >> > > > > > users who don't use the system frequently.
> >> > > > > >
> >> > > > > > For both cases, the built-in authorizer will continue
> to be
> >> > > > > > synchronous,
> >> > > > > > using CompletableFuture.completedFuture() to return
> the
> >> actual
> >> > > > > result.
> >> > > > > > But
> >> > > > > > async API will make custom authorizer implementations
> more
> >> > > > flexible.
> >> > > > > I
> >> > > > > > would like to know if there are any concerns with
> these
> >> changes
> >> > > > > before
> >> > > > > > updating the KIP.
> >> > > > > >
> >> > > > > > *Proposed API:*
> >> > > > > > public interface Authorizer extends Configurable,
> Closeable
> >> {
> >> > > > > >
> >> > > > > > Map<Endpoint, CompletionStage<Void>>
> >> > > start(AuthorizerServerInfo
> >> > > > > > serverInfo);
> >> > > > > > List<CompletionStage<AuthorizationResult>>
> >> > > > > > authorize(AuthorizableRequestContext requestContext,
> >> > List<Action>
> >> > > > > > actions);
> >> > > > > > List<CompletionStage<AclCreateResult>>
> >> > > > > > createAcls(AuthorizableRequestContext requestContext,
> >> > > > > List<AclBinding>
> >> > > > > > aclBindings);
> >> > > > > > List<CompletionStage<AclDeleteResult>>
> >> > > > > > deleteAcls(AuthorizableRequestContext requestContext,
> >> > > > > > List<AclBindingFilter> aclBindingFilters);
> >> > > > > > CompletionStage<Collection<AclBinding>>
> >> > acls(AclBindingFilter
> >> > > > > > filter);
> >> > > > > > }
> >> > > > > >
> >> > > > > >
> >> > > > > > Thank you,
> >> > > > > >
> >> > > > > > Rajini
> >> > > > > >
> >> > > > > > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <
> >> > > bosco@apache.org>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Rajini
> >> > > > > > >
> >> > > > > > > Thanks for clarifying. I am good for now.
> >> > > > > > >
> >> > > > > > > Regards
> >> > > > > > >
> >> > > > > > > Bosco
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <
> >> > > rajinisivaram@gmail.com
> >> > > > >
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > Hi Don,
> >> > > > > > >
> >> > > > > > > That should be fine. I guess Ranger loads
> policies
> >> from
> >> > the
> >> > > > > > database
> >> > > > > > > synchronously when the authorizer is configured,
> >> similar
> >> > to
> >> > > > > > > SimpleAclAuthorizer loading from ZooKeeper.
> Ranger can
> >> > > > continue
> >> > > > > > to load
> >> > > > > > > synchronously from `configure()` or `start()`
> and
> >> return
> >> > an
> >> > > > > > empty map
> >> > > > > > > from
> >> > > > > > > `start()`. That would retain the existing
> behaviour..
> >> > When
> >> > > > the
> >> > > > > > same
> >> > > > > > > database stores policies for all listeners and
> the
> >> > policies
> >> > > > are
> >> > > > > > not
> >> > > > > > > stored
> >> > > > > > > in Kafka, there is no value in making the load
> >> > > asynchronous.
> >> > > > > > >
> >> > > > > > > Regards,
> >> > > > > > >
> >> > > > > > > Rajini
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai
> <
> >> > > > > > bosco@apache.org>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Hi Rajini
> >> > > > > > > >
> >> > > > > > > > Assuming this doesn't affect custom plugins,
> I don't
> >> > have
> >> > > > any
> >> > > > > > > concerns
> >> > > > > > > > regarding this.
> >> > > > > > > >
> >> > > > > > > > I do have one question regarding:
> >> > > > > > > >
> >> > > > > > > > "For authorizers that don’t store metadata in
> >> > ZooKeeper,
> >> > > > > > ensure that
> >> > > > > > > > authorizer metadata for each listener is
> available
> >> > before
> >> > > > > > starting
> >> > > > > > > up the
> >> > > > > > > > listener. This enables different authorization
> >> metadata
> >> > > > > stores
> >> > > > > > for
> >> > > > > > > > different listeners."
> >> > > > > > > >
> >> > > > > > > > Since Ranger uses its own database for storing
> >> > policies,
> >> > > do
> >> > > > > you
> >> > > > > > > anticipate
> >> > > > > > > > any issues?
> >> > > > > > > >
> >> > > > > > > > Thanks
> >> > > > > > > >
> >> > > > > > > > Bosco
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
> >> > > > > > rajinisivaram@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > Hi all,
> >> > > > > > > >
> >> > > > > > > > I made another change to the KIP. The KIP
> was
> >> > > > originally
> >> > > > > > > proposing to
> >> > > > > > > > extend SimpleAclAuthorizer to also
> implement the
> >> > new
> >> > > > API
> >> > > > > > (in
> >> > > > > > > addition
> >> > > > > > > > to
> >> > > > > > > > the existing API). But since we use the
> new API
> >> > when
> >> > > > > > available,
> >> > > > > > > this
> >> > > > > > > > can
> >> > > > > > > > break custom authorizers that extend this
> class
> >> and
> >> > > > > > override
> >> > > > > > > specific
> >> > > > > > > > methods of the old API. To avoid breaking
> any
> >> > > existing
> >> > > > > > custom
> >> > > > > > > > implementations that extend this class,
> >> > particularly
> >> > > > > > because it
> >> > > > > > > is in
> >> > > > > > > > the
> >> > > > > > > > public package kafka.security.auth, the
> KIP now
> >> > > > proposes
> >> > > > > to
> >> > > > > > > retain the
> >> > > > > > > > old
> >> > > > > > > > authorizer as-is. SimpleAclAuthorizer
> will
> >> > continue
> >> > > to
> >> > > > > > > implement the
> >> > > > > > > > old
> >> > > > > > > > API, but will be deprecated. A new
> authorizer
> >> > > > > > implementation
> >> > > > > > > > kafka.security.authorizer.AclAuthorizer
> will be
> >> > added
> >> > > > for
> >> > > > > > the
> >> > > > > > > new API
> >> > > > > > > > (this
> >> > > > > > > > will not be in the public package).
> >> > > > > > > >
> >> > > > > > > > Please let me know if you have any
> concerns.
> >> > > > > > > >
> >> > > > > > > > Regards,
> >> > > > > > > >
> >> > > > > > > > Rajini
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Fri, Aug 16, 2019 at 8:48 AM Rajini
> Sivaram <
> >> > > > > > > > rajinisivaram@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Thanks Colin.
> >> > > > > > > > >
> >> > > > > > > > > If there are no other concerns, I will
> start
> >> vote
> >> > > > later
> >> > > > > > today.
> >> > > > > > > Many
> >> > > > > > > > thanks
> >> > > > > > > > > to every one for the feedback.
> >> > > > > > > > >
> >> > > > > > > > > Regards,
> >> > > > > > > > >
> >> > > > > > > > > Rajini
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Thu, Aug 15, 2019 at 11:57 PM Colin
> McCabe
> >> <
> >> > > > > > > cmccabe@apache.org>
> >> > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > >> Thanks, Rajini. It looks good to me.
> >> > > > > > > > >>
> >> > > > > > > > >> best,
> >> > > > > > > > >> Colin
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > > > >> On Thu, Aug 15, 2019, at 11:37, Rajini
> >> Sivaram
> >> > > > wrote:
> >> > > > > > > > >> > Hi Colin,
> >> > > > > > > > >> >
> >> > > > > > > > >> > Thanks for the review. I have
> updated the
> >> KIP
> >> > to
> >> > > > > move
> >> > > > > > the
> >> > > > > > > > interfaces for
> >> > > > > > > > >> > request context and server info to
> the
> >> > > authorizer
> >> > > > > > package.
> >> > > > > > > These
> >> > > > > > > > are now
> >> > > > > > > > >> > called AuthorizableRequestContext and
> >> > > > > > AuthorizerServerInfo.
> >> > > > > > > > Endpoint is
> >> > > > > > > > >> now
> >> > > > > > > > >> > a class in org.apache.kafka.common
> to make
> >> it
> >> > > > > reusable
> >> > > > > > > since we
> >> > > > > > > > already
> >> > > > > > > > >> > have multiple implementations of it.
> I have
> >> > > > removed
> >> > > > > > > requestName
> >> > > > > > > > from the
> >> > > > > > > > >> > request context interface since
> authorizers
> >> > can
> >> > > > > > distinguish
> >> > > > > > > > follower
> >> > > > > > > > >> fetch
> >> > > > > > > > >> > and consumer fetch from the
> operation being
> >> > > > > > authorized. So
> >> > > > > > > 16-bit
> >> > > > > > > > >> request
> >> > > > > > > > >> > type should be sufficient for audit
> >> logging.
> >> > > Also
> >> > > > > > replaced
> >> > > > > > > > AuditFlag
> >> > > > > > > > >> with
> >> > > > > > > > >> > two booleans as you suggested.
> >> > > > > > > > >> >
> >> > > > > > > > >> > Can you take another look and see if
> the
> >> KIP
> >> > is
> >> > > > > ready
> >> > > > > > for
> >> > > > > > > voting?
> >> > > > > > > > >> >
> >> > > > > > > > >> > Thanks for all your help!
> >> > > > > > > > >> >
> >> > > > > > > > >> > Regards,
> >> > > > > > > > >> >
> >> > > > > > > > >> > Rajini
> >> > > > > > > > >> >
> >> > > > > > > > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin
> >> McCabe <
> >> > > > > > > cmccabe@apache.org>
> >> > > > > > > > >> wrote:
> >> > > > > > > > >> >
> >> > > > > > > > >> > > Hi Rajini,
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > I think it would be good to rename
> >> > > > > > KafkaRequestContext to
> >> > > > > > > > something
> >> > > > > > > > >> like
> >> > > > > > > > >> > > AuthorizableRequestContext, and
> put it in
> >> > the
> >> > > > > > > > >> > > org.apache.kafka.server.authorizer
> >> > namespace.
> >> > > > If
> >> > > > > > we put
> >> > > > > > > it in
> >> > > > > > > > the
> >> > > > > > > > >> > > org.apache.kafka.common namespace,
> then
> >> it's
> >> > > not
> >> > > > > > really
> >> > > > > > > clear
> >> > > > > > > > that
> >> > > > > > > > >> it's
> >> > > > > > > > >> > > part of the Authorizer API. Since
> this
> >> > class
> >> > > is
> >> > > > > > purely an
> >> > > > > > > > interface,
> >> > > > > > > > >> with
> >> > > > > > > > >> > > no concrete implementation of
> anything,
> >> > > there's
> >> > > > > > nothing
> >> > > > > > > common
> >> > > > > > > > to
> >> > > > > > > > >> really
> >> > > > > > > > >> > > reuse in any case. We definitely
> don't
> >> want
> >> > > > > > someone to
> >> > > > > > > > accidentally
> >> > > > > > > > >> add or
> >> > > > > > > > >> > > remove methods because they think
> this is
> >> > just
> >> > > > > > another
> >> > > > > > > internal
> >> > > > > > > > class
> >> > > > > > > > >> used
> >> > > > > > > > >> > > for requests.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > The BrokerInfo class is a nice
> >> improvement.
> >> > > It
> >> > > > > > looks
> >> > > > > > > like it
> >> > > > > > > > will be
> >> > > > > > > > >> > > useful for passing in information
> about
> >> the
> >> > > > > context
> >> > > > > > we're
> >> > > > > > > > running
> >> > > > > > > > >> in. It
> >> > > > > > > > >> > > would be nice to call this class
> >> ServerInfo
> >> > > > rather
> >> > > > > > than
> >> > > > > > > > BrokerInfo,
> >> > > > > > > > >> since
> >> > > > > > > > >> > > we will want to run the authorizer
> on
> >> > > > controllers
> >> > > > > > as well
> >> > > > > > > as on
> >> > > > > > > > >> brokers,
> >> > > > > > > > >> > > and the controller may run as a
> separate
> >> > > process
> >> > > > > > post
> >> > > > > > > KIP-500.
> >> > > > > > > > I also
> >> > > > > > > > >> > > think that this class should be in
> the
> >> > > > > > > > >> org.apache.kafka.server.authorizer
> >> > > > > > > > >> > > namespace. Again, it is an
> interface,
> >> not a
> >> > > > > > concrete
> >> > > > > > > > implementation,
> >> > > > > > > > >> and
> >> > > > > > > > >> > > it's an interface that is very
> >> specifically
> >> > > for
> >> > > > > the
> >> > > > > > > authorizer.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > I agree that we probably don't have
> >> enough
> >> > > > > > information
> >> > > > > > > > preserved for
> >> > > > > > > > >> > > requests currently to always know
> what
> >> > entity
> >> > > > made
> >> > > > > > them.
> >> > > > > > > So we
> >> > > > > > > > can
> >> > > > > > > > >> leave
> >> > > > > > > > >> > > that out for now (except in the
> special
> >> case
> >> > > of
> >> > > > > > Fetch).
> >> > > > > > > > Perhaps we
> >> > > > > > > > >> can add
> >> > > > > > > > >> > > this later if it's needed.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > I understand the intention behind
> >> > > > > AuthorizationMode
> >> > > > > > > (which is
> >> > > > > > > > now
> >> > > > > > > > >> called
> >> > > > > > > > >> > > AuditFlag in the latest
> revision). But
> >> it
> >> > > still
> >> > > > > > feels
> >> > > > > > > > complex. What
> >> > > > > > > > >> if we
> >> > > > > > > > >> > > just had two booleans in Action:
> >> > logSuccesses
> >> > > > and
> >> > > > > > > logFailures?
> >> > > > > > > > That
> >> > > > > > > > >> seems
> >> > > > > > > > >> > > to cover all the cases here.
> >> > > > MANDATORY_AUTHORIZE
> >> > > > > =
> >> > > > > > true,
> >> > > > > > > true.
> >> > > > > > > > >> > > OPTIONAL_AUTHORIZE = true, false.
> >> FILTER =
> >> > > > true,
> >> > > > > > false.
> >> > > > > > > > >> LIST_AUTHORIZED =
> >> > > > > > > > >> > > false, false. Would there be
> anything
> >> lost
> >> > > > versus
> >> > > > > > having
> >> > > > > > > the
> >> > > > > > > > enum?
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > best,
> >> > > > > > > > >> > > Colin
> >> > > > > > > > >> > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > On Wed, Aug 14, 2019, at 06:29,
> Mickael
> >> > Maison
> >> > > > > > wrote:
> >> > > > > > > > >> > > > Hi Rajini,
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > Thanks for the KIP!
> >> > > > > > > > >> > > > I really like that authorize()
> will be
> >> > able
> >> > > to
> >> > > > > > take a
> >> > > > > > > batch of
> >> > > > > > > > >> > > > requests, this will speed up many
> >> > > > > implementations!
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM
> Rajini
> >> > > > Sivaram <
> >> > > > > > > > >> rajinisivaram@gmail.com>
> >> > > > > > > > >> > > wrote:
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > Thanks David! I have fixed the
> typo.
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > Also made a couple of changes
> to make
> >> > the
> >> > > > > > context
> >> > > > > > > > interfaces more
> >> > > > > > > > >> > > generic.
> >> > > > > > > > >> > > > > KafkaRequestContext now
> returns the
> >> > 16-bit
> >> > > > API
> >> > > > > > key as
> >> > > > > > > Colin
> >> > > > > > > > >> suggested
> >> > > > > > > > >> > > as
> >> > > > > > > > >> > > > > well as the friendly name used
> in
> >> > metrics
> >> > > > > which
> >> > > > > > are
> >> > > > > > > useful
> >> > > > > > > > in
> >> > > > > > > > >> audit
> >> > > > > > > > >> > > logs.
> >> > > > > > > > >> > > > > `Authorizer#start` is now
> provided a
> >> > > generic
> >> > > > > > > `BrokerInfo`
> >> > > > > > > > >> interface
> >> > > > > > > > >> > > that
> >> > > > > > > > >> > > > > gives cluster id, broker id and
> >> endpoint
> >> > > > > > information.
> >> > > > > > > The
> >> > > > > > > > generic
> >> > > > > > > > >> > > interface
> >> > > > > > > > >> > > > > can potentially be used in
> other
> >> broker
> >> > > > > plugins
> >> > > > > > in
> >> > > > > > > future
> >> > > > > > > > and
> >> > > > > > > > >> provides
> >> > > > > > > > >> > > > > dynamically generated configs
> like
> >> > broker
> >> > > id
> >> > > > > > and ports
> >> > > > > > > > which are
> >> > > > > > > > >> > > currently
> >> > > > > > > > >> > > > > not available to plugins
> unless these
> >> > > > configs
> >> > > > > > are
> >> > > > > > > statically
> >> > > > > > > > >> > > configured.
> >> > > > > > > > >> > > > > Please let me know if there
> are any
> >> > > > concerns.
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > Regards,
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > Rajini
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > On Tue, Aug 13, 2019 at 4:30
> PM David
> >> > > Jacot
> >> > > > <
> >> > > > > > > > djacot@confluent.io>
> >> > > > > > > > >> > > wrote:
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > > Hi Rajini,
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > Thank you for the update! It
> looks
> >> > good
> >> > > to
> >> > > > > me.
> >> > > > > > > There is a
> >> > > > > > > > typo
> >> > > > > > > > >> in the
> >> > > > > > > > >> > > > > > `AuditFlag` enum:
> >> > `MANDATORY_AUTHOEIZE`
> >> > > ->
> >> > > > > > > > >> `MANDATORY_AUTHORIZE`.
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > Regards,
> >> > > > > > > > >> > > > > > David
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > On Mon, Aug 12, 2019 at 2:54
> PM
> >> Rajini
> >> > > > > > Sivaram <
> >> > > > > > > > >> > > rajinisivaram@gmail.com>
> >> > > > > > > > >> > > > > > wrote:
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > > Hi David,
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > Thanks for reviewing the
> KIP!
> >> Since
> >> > > > > > questions
> >> > > > > > > about
> >> > > > > > > > >> `authorization
> >> > > > > > > > >> > > mode`
> >> > > > > > > > >> > > > > > > and `count` have come up
> multiple
> >> > > > times, I
> >> > > > > > have
> >> > > > > > > renamed
> >> > > > > > > > both.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > 1) Renamed `count` to
> >> > > > > > `resourceReferenceCount`.
> >> > > > > > > It is
> >> > > > > > > > the
> >> > > > > > > > >> number
> >> > > > > > > > >> > > of times
> >> > > > > > > > >> > > > > > > the resource being
> authorized is
> >> > > > > referenced
> >> > > > > > > within the
> >> > > > > > > > >> request.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > 2) Renamed
> `AuthorizationMode` to
> >> > > > > > `AuditFlag`. It
> >> > > > > > > is
> >> > > > > > > > provided
> >> > > > > > > > >> to
> >> > > > > > > > >> > > improve
> >> > > > > > > > >> > > > > > > audit logging in the
> authorizer.
> >> The
> >> > > > enum
> >> > > > > > values
> >> > > > > > > have
> >> > > > > > > > javadoc
> >> > > > > > > > >> which
> >> > > > > > > > >> > > > > > > indicate how the
> authorization
> >> > result
> >> > > is
> >> > > > > > used in
> >> > > > > > > each
> >> > > > > > > > of the
> >> > > > > > > > >> modes
> >> > > > > > > > >> > > to
> >> > > > > > > > >> > > > > > > enable authorizers to log
> audit
> >> > > messages
> >> > > > > at
> >> > > > > > the
> >> > > > > > > right
> >> > > > > > > > severity
> >> > > > > > > > >> > > level.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > Regards,
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > Rajini
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > On Mon, Aug 12, 2019 at
> 5:54 PM
> >> > David
> >> > > > > Jacot
> >> > > > > > <
> >> > > > > > > > >> djacot@confluent.io>
> >> > > > > > > > >> > > wrote:
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > > Hi Rajini,
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > Thank you for the KIP.
> >> Overall, it
> >> > > > looks
> >> > > > > > good
> >> > > > > > > to me.
> >> > > > > > > > I have
> >> > > > > > > > >> few
> >> > > > > > > > >> > > > > > > > questions/suggestions:
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > 1. It is hard to grasp
> what
> >> > > > > > `Action#count` is
> >> > > > > > > for. I
> >> > > > > > > > guess I
> >> > > > > > > > >> > > understand
> >> > > > > > > > >> > > > > > > > where you want to go
> with it
> >> but
> >> > it
> >> > > > took
> >> > > > > > me a
> >> > > > > > > while to
> >> > > > > > > > >> figure it
> >> > > > > > > > >> > > out.
> >> > > > > > > > >> > > > > > > > Perhaps, we could come
> up with
> >> a
> >> > > > better
> >> > > > > > name
> >> > > > > > > than
> >> > > > > > > > `count`?
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > 2. I had a hard time
> trying to
> >> > > > > understand
> >> > > > > > the
> >> > > > > > > > >> `AuthorizationMode`
> >> > > > > > > > >> > > > > > > concept,
> >> > > > > > > > >> > > > > > > > especially wrt. the
> OPTIONAL
> >> one.
> >> > My
> >> > > > > > > understanding is
> >> > > > > > > > that
> >> > > > > > > > >> an
> >> > > > > > > > >> > > ACL is
> >> > > > > > > > >> > > > > > > either
> >> > > > > > > > >> > > > > > > > defined or not. Could you
> >> > elaborate
> >> > > a
> >> > > > > bit
> >> > > > > > more
> >> > > > > > > on
> >> > > > > > > > that?
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > Thanks,
> >> > > > > > > > >> > > > > > > > David
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > On Fri, Aug 9, 2019 at
> 10:26 PM
> >> > Don
> >> > > > > Bosco
> >> > > > > > Durai
> >> > > > > > > <
> >> > > > > > > > >> > > bosco@apache.org>
> >> > > > > > > > >> > > > > > > wrote:
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > > Hi Rajini
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > 3.2 - This makes sense.
> >> Thanks
> >> > for
> >> > > > > > clarifying.
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > Rest looks fine. Once
> the
> >> > > > > > implementations are
> >> > > > > > > done,
> >> > > > > > > > it
> >> > > > > > > > >> will be
> >> > > > > > > > >> > > more
> >> > > > > > > > >> > > > > > > clear
> >> > > > > > > > >> > > > > > > > > on the different values
> >> > > RequestType
> >> > > > > and
> >> > > > > > Mode.
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > Thanks
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > Bosco
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > On 8/9/19, 5:08 AM,
> "Rajini
> >> > > > Sivaram"
> >> > > > > <
> >> > > > > > > > >> rajinisivaram@gmail.com
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > > > wrote:
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > Hi Don,
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > Thanks for the
> >> suggestions.
> >> > A
> >> > > > few
> >> > > > > > > responses
> >> > > > > > > > below:
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > 3.1 Can rename and
> >> improve
> >> > > docs
> >> > > > if
> >> > > > > > we keep
> >> > > > > > > > this. Let's
> >> > > > > > > > >> > > finish the
> >> > > > > > > > >> > > > > > > > > discussion on
> Colin's
> >> > > > suggestions
> >> > > > > > > regarding this
> >> > > > > > > > >> first.
> >> > > > > > > > >> > > > > > > > > 3.2 No, I was
> thinking of
> >> > some
> >> > > > > > requests
> >> > > > > > > that
> >> > > > > > > > have an
> >> > > > > > > > >> old
> >> > > > > > > > >> > > way of
> >> > > > > > > > >> > > > > > > > > authorizing
> >> > > > > > > > >> > > > > > > > > and a new way
> where we
> >> have
> >> > > > > > retained the
> >> > > > > > > old
> >> > > > > > > > way for
> >> > > > > > > > >> > > backward
> >> > > > > > > > >> > > > > > > > > compatibility. One
> >> example
> >> > is
> >> > > > > > > Cluster:Create
> >> > > > > > > > >> permission to
> >> > > > > > > > >> > > create
> >> > > > > > > > >> > > > > > > > > topics.
> >> > > > > > > > >> > > > > > > > > We have replaced
> this
> >> with
> >> > > > > > fine-grained
> >> > > > > > > topic
> >> > > > > > > > create
> >> > > > > > > > >> > > access using
> >> > > >
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Don Bosco Durai <bo...@apache.org>.
Sounds good to me.
Thanks
Bosco
On 9/6/19, 11:47 AM, "Colin McCabe" <cm...@apache.org> wrote:
Hi Rajini,
I agree that there is probably some value in async createAcls and deleteAcls methods. It's also a smaller change to use a futures-based interface there, since it really only affects the CreateAcls and DeleteAcls operations. It's probably fine for the other methods to be sync for now.... we would need a longer discussion about how to manage and monitor the authorization purgatory to make this async, and it's not clear that we need it (yet?)
best,
Colin
On Fri, Sep 6, 2019, at 11:32, Rajini Sivaram wrote:
> I would suggest that we go with synchronous authorize() and acls() method
> to describe ACLs, but asynchronous createAcls() and deleteAcls() that
> return CompletionStage. With KIP-500, I think async create/delete methods
> will be useful. For now, we can assume that brokers are able to cache all
> ACLs. That keeps the API and implementation simple, while still avoiding
> blocking request threads for the expensive update operations. Even though
> we don't expect large rates of concurrent ACL updates, blocking request
> threads during updates (e.g. to a database) isn't ideal. Since custom
> authorizers can still do synchronous updates by returning completed
> futures, would this be a reasonable change to the API?
>
> Thanks,
>
> Rajini
>
>
> On Fri, Sep 6, 2019 at 5:32 PM Don Bosco Durai <bo...@apache.org> wrote:
>
> > +1
> >
> > I might be wrong here, but here are few of my assumptions..
> >
> > 1. Unlike transactional services, in Kafka most of the users are
> > application users, so once the connection is established, then they are
> > going to be long running sessions. So the caching is more predictable and
> > heavy lifting can be done during initial session setup and cache can be
> > frequently updated using background thread. KSQL might have a different
> > flow, but I don't have enough datapoints for it.
> > 2. For publish/consume authorization calls, providing async option will be
> > misleading for Kafka like load
> > 3. For admin calls, hopefully no one does hundreds of call in multiple
> > thread within few seconds. This would be a bad design.
> >
> > So I feel, we should be okay with sync calls for now. And consider async
> > in the future if needed. At least in Apache Ranger implementation, we would
> > be okay using sync operation. I am not sure if any other plugin
> > implementation would benefit from async implementation.
> >
> > Thanks
> >
> > Bosco
> >
> > On 9/6/19, 8:45 AM, "Ismael Juma" <is...@juma.me.uk> wrote:
> >
> > One more thing: if people have strong opinions that authorize has to be
> > sync, we could leave it like that for now. If needed, we can later add
> > an
> > authorizeAsync with another method returning a Boolean indicating that
> > it's
> > supported. That is, there is a path to evolve the interface (even if a
> > bit
> > ugly).
> >
> > What about the other methods, is there consensus that they should be
> > updated to return CompletionStage?
> >
> > Ismael
> >
> > On Fri, Sep 6, 2019, 8:32 AM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > I'm on the move, but what Rajini said seems reasonable. I don't think
> > > using SSDs solves the issue. They can still hang for seconds when
> > they
> > > fail. Also, many people may not have local SSDs available (remote
> > SSDs like
> > > EBS hang for tens of seconds when there are issues).
> > >
> > > We are currently vulnerable to all of these in the normal read/write
> > path,
> > > but it seems suboptimal to continue assuming non blocking behavior
> > for
> > > things that actually do block.
> > >
> > > Ismael
> > >
> > > On Fri, Sep 6, 2019, 8:24 AM Rajini Sivaram <rajinisivaram@gmail.com
> > >
> > > wrote:
> > >
> > >> Hi Jun,
> > >>
> > >> For ACLs, the size is also dependent on the number of users. So in
> > >> deployments with large number of users where some users are not
> > active, an
> > >> LRU cache may be useful. Agree that there may be other solutions
> > that help
> > >> avoid the requirement for an async API even for this case.
> > >>
> > >> I wasn't expecting our threading model to change very much. Having
> > said
> > >> that, I haven't figured out how the purgatory can be adapted to work
> > >> efficiently for this. I was thinking:
> > >> 1) If authorization future is complete when returning from
> > authorize()
> > >> call, we invoke the method immediately on the request thread
> > >> 2) If future is not complete, the request goes into a purgatory
> > >> 3) When future completes, we mark the request ready to be run. This
> > can be
> > >> done on ForkJoinPool.commonPool or an executor. But we don't handle
> > the
> > >> request on that thread
> > >> 4) We handle the request on the request thread after 3).
> > >>
> > >> I totally agree that this adds complexity to the code. Will wait for
> > >> Ismael
> > >> to comment on whether he had a different model in mind.
> > >>
> > >> Thanks,
> > >>
> > >> Rajini
> > >>
> > >>
> > >> On Thu, Sep 5, 2019 at 10:29 PM Jun Rao <ju...@confluent.io> wrote:
> > >>
> > >> > Hi, Ismael,
> > >> >
> > >> > Yes, I agree that there are 2 separate discussion points, one on
> > the API
> > >> > and the other on the implementation.
> > >> >
> > >> > First on the API, in general, my feeling is that the authorize()
> > call is
> > >> > expected to be fast to reduce request latency. Adding an async API
> > >> could be
> > >> > mis-leading since it might give users the impression that it
> > would be
> > >> > arbitrarily long. In the short term, currently, we cache all
> > partition
> > >> > level metadata on every broker. The authorization metadata is
> > >> proportional
> > >> > to the number of topics, which typically is less in size. So it
> > we can
> > >> > cache all partition level metadata, we should be able to cache all
> > >> > authorization metadata. Longer term, if we want to support more
> > metadata
> > >> > beyond memory, another option is to put all metadata in an SSD
> > backed
> > >> > key/value store. The lookup time could still be under 1ms, which
> > we
> > >> could
> > >> > potentially absorb in the request threads. So, I am not sure async
> > >> > authorize() api is strictly needed even in the long term.
> > >> >
> > >> > Second on the implementation. I was more concerned about the
> > impact on
> > >> our
> > >> > threading model. If we make authorize() async, some threads have
> > to
> > >> > complete the future when it's ready. It seems to me that those
> > threads
> > >> have
> > >> > to be inside the Authorizer implementation since only it knows
> > when to
> > >> > complete a future? If so, it feels weird that all the logic of
> > request
> > >> > handling post authorize() is now handled by external threads
> > inside a
> > >> > plugin. It's not clear how we can configure and monitor them, and
> > how
> > >> > throttling would work. Perhaps we can provide a bit more detail
> > on the
> > >> new
> > >> > threading model.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Jun
> > >> >
> > >> > On Wed, Sep 4, 2019 at 9:56 PM Ismael Juma <is...@juma.me.uk>
> > wrote:
> > >> >
> > >> > > Hi Jun,
> > >> > >
> > >> > > I think it's important to discuss the pluggable API versus the
> > calling
> > >> > > implementation as two separate points.
> > >> > >
> > >> > > From an API perspective, are you suggesting that we should tell
> > users
> > >> > that
> > >> > > they cannot block in authorize? Or are you saying that it's OK
> > to
> > >> block
> > >> > on
> > >> > > authorize on occasion and the recommendation would be for
> > people to
> > >> > > increase the number of threads in the request thread pool?
> > >> > Architecturally,
> > >> > > this feels wrong in my opinion.
> > >> > >
> > >> > > From the calling implementation perspective, you don't need a
> > >> callback.
> > >> > You
> > >> > > can basically say:
> > >> > >
> > >> > > authorize(...).map { result =>
> > >> > > ...
> > >> > > }
> > >> > >
> > >> > > And then have a common method take that Future and handle it
> > >> > synchronously
> > >> > > if it's already complete or submit it to the purgatory if not.
> > This is
> > >> > > similar to how many modern async web libraries work.
> > >> > >
> > >> > > As Rajini said, this could be done later. The initial
> > implementation
> > >> > could
> > >> > > simply do `toCompletableFuture().get()` to do it synchronously.
> > But it
> > >> > > would mean that we could add this functionality without
> > changing the
> > >> > > pluggable interface.
> > >> > >
> > >> > > Ismael
> > >> > >
> > >> > > On Wed, Sep 4, 2019 at 5:29 AM Jun Rao <ju...@confluent.io>
> > wrote:
> > >> > >
> > >> > > > Hi, Rajini,
> > >> > > >
> > >> > > > Thanks for the KIP. I was concerned about #4 too. If we
> > change the
> > >> > > handling
> > >> > > > of all requests to use an async authorize() api, will that
> > cause the
> > >> > code
> > >> > > > much harder to understand? There are quite a few callbacks
> > already.
> > >> I
> > >> > am
> > >> > > > not sure that we want to introduce more of those. The benefit
> > from
> > >> > async
> > >> > > > authorize() api seems limited.
> > >> > > >
> > >> > > > Jun
> > >> > > >
> > >> > > > On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <
> > >> rajinisivaram@gmail.com
> > >> > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi Don,
> > >> > > > >
> > >> > > > > Thanks for your note.
> > >> > > > >
> > >> > > > > 1) The intention is to avoid blocking in the calling
> > thread. We
> > >> > already
> > >> > > > > have several requests that are put into a purgatory when
> > waiting
> > >> for
> > >> > > > remote
> > >> > > > > communication, for example produce request waiting for
> > >> replication.
> > >> > > Since
> > >> > > > > we have a limited number of request threads in the broker,
> > we
> > >> want to
> > >> > > > make
> > >> > > > > progress with other requests, while one is waiting on any
> > form of
> > >> > > remote
> > >> > > > > communication.
> > >> > > > >
> > >> > > > > 2) Async management calls will be useful with the default
> > >> authorizer
> > >> > > when
> > >> > > > > KIP-500 removes ZK and we rely on Kafka instead. Our current
> > >> ZK-based
> > >> > > > > implementation as well as any custom implementations that
> > don't
> > >> want
> > >> > to
> > >> > > > be
> > >> > > > > async will just need to return a sync'ed value. So instead
> > of
> > >> > > returning `
> > >> > > > > value`, the code would just return
> > >> > > > > `CompletableFuture.completedFuture(value)
> > >> > > > > `. So it would be just a single line change in the
> > implementation
> > >> > with
> > >> > > > the
> > >> > > > > new API. The caller would treat completedFuture exactly as
> > it does
> > >> > > today,
> > >> > > > > processing the request synchronously without using a
> > purgatory.
> > >> > > > >
> > >> > > > > 3) For implementations that return a completedFuture as
> > described
> > >> in
> > >> > > 2),
> > >> > > > > the behaviour would remain exactly the same. No additional
> > >> threads or
> > >> > > > > purgatory will be used for this case. So there would be no
> > >> > performance
> > >> > > > > penalty. For implementations that return a future that is
> > not
> > >> > complete,
> > >> > > > we
> > >> > > > > prioritise running more requests concurrently. So in a
> > deployment
> > >> > with
> > >> > > a
> > >> > > > > large number of clients, we would improve performance by
> > allowing
> > >> > other
> > >> > > > > requests to be processed on the request threads while some
> > are
> > >> > waiting
> > >> > > > for
> > >> > > > > authorization metadata.
> > >> > > > >
> > >> > > > > 4) I was concerned about this too. The goal is to make the
> > API
> > >> > flexible
> > >> > > > > enough to handle large scale deployments in future when
> > caching
> > >> all
> > >> > > > > authorization metadata in each broker is not viable. Using
> > an
> > >> async
> > >> > API
> > >> > > > > that returns CompletionStage, the caller has the option to
> > handle
> > >> the
> > >> > > > > result synchronously or asynchronously, so we don't
> > necessarily
> > >> need
> > >> > to
> > >> > > > > update the calling code right away. Custom authorizers
> > using the
> > >> > async
> > >> > > > API
> > >> > > > > have full control over whether authorization is performed
> > in-line
> > >> > since
> > >> > > > > completedFuture will always be handled synchronously. We do
> > need
> > >> to
> > >> > > > update
> > >> > > > > KafkaApis to take advantage of the asynchronous API to
> > improve
> > >> scale.
> > >> > > > Even
> > >> > > > > though this is a big change, since we will be doing the
> > same for
> > >> all
> > >> > > > > requests, it shouldn't be too hard to maintain since the
> > same
> > >> pattern
> > >> > > > will
> > >> > > > > be used for all requests.
> > >> > > > >
> > >> > > > > Regards,
> > >> > > > >
> > >> > > > > Rajini
> > >> > > > >
> > >> > > > > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <
> > bosco@apache.org
> > >> >
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > > Hi Rajini
> > >> > > > > >
> > >> > > > > > Help me understand this a bit more.
> > >> > > > > >
> > >> > > > > > 1. For all practical purpose, without authorization you
> > can't
> > >> go to
> > >> > > the
> > >> > > > > > next step. The calling code needs to block anyway. So
> > should we
> > >> > just
> > >> > > > let
> > >> > > > > > the implementation code do the async part?
> > >> > > > > > 2. If you feel management calls need to be async, then we
> > should
> > >> > > > consider
> > >> > > > > > another set of async APIs. I don't feel we should
> > complicate it
> > >> > > > further (
> > >> > > > > > 3. Another concern I have is wrt performance. Kafka has
> > been
> > >> built
> > >> > to
> > >> > > > > > handle 1000s per second requests. Not sure whether making
> > it
> > >> async
> > >> > > will
> > >> > > > > add
> > >> > > > > > any unnecessary overhead.
> > >> > > > > > 4. How much complication would this add on the calling
> > side?
> > >> And is
> > >> > > it
> > >> > > > > > worth it?
> > >> > > > > >
> > >> > > > > > Thanks
> > >> > > > > >
> > >> > > > > > Bosco
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On 9/3/19, 8:50 AM, "Rajini Sivaram" <
> > rajinisivaram@gmail.com>
> > >> > > wrote:
> > >> > > > > >
> > >> > > > > > Hi all,
> > >> > > > > >
> > >> > > > > > Ismael brought up a point that it will be good to
> > make the
> > >> > > > Authorizer
> > >> > > > > > interface asynchronous to avoid blocking request
> > threads
> > >> during
> > >> > > > > remote
> > >> > > > > > operations.
> > >> > > > > >
> > >> > > > > > 1) Since we want to support different backends for
> > >> > authorization
> > >> > > > > > metadata,
> > >> > > > > > making createAcls() and deleteAcls() asynchronous
> > makes
> > >> sense
> > >> > > since
> > >> > > > > > these
> > >> > > > > > always involve remote operations. When KIP-500 removes
> > >> > ZooKeeper,
> > >> > > > we
> > >> > > > > > would
> > >> > > > > > want to move ACLs to Kafka and async updates will
> > avoid
> > >> > > unnecessary
> > >> > > > > > blocking.
> > >> > > > > > 2) For authorize() method, we currently use cached
> > ACLs in
> > >> the
> > >> > > > > built-in
> > >> > > > > > authorizers, so synchronous authorise operations work
> > well
> > >> now.
> > >> > > But
> > >> > > > > > async
> > >> > > > > > authorize() would support this scenario as well as
> > >> authorizers
> > >> > in
> > >> > > > > large
> > >> > > > > > organisations where an LRU cache would enable a
> > smaller
> > >> cache
> > >> > > even
> > >> > > > > > when the
> > >> > > > > > backend holds a large amount of ACLs for infrequently
> > used
> > >> > > > resources
> > >> > > > > or
> > >> > > > > > users who don't use the system frequently.
> > >> > > > > >
> > >> > > > > > For both cases, the built-in authorizer will continue
> > to be
> > >> > > > > > synchronous,
> > >> > > > > > using CompletableFuture.completedFuture() to return
> > the
> > >> actual
> > >> > > > > result.
> > >> > > > > > But
> > >> > > > > > async API will make custom authorizer implementations
> > more
> > >> > > > flexible.
> > >> > > > > I
> > >> > > > > > would like to know if there are any concerns with
> > these
> > >> changes
> > >> > > > > before
> > >> > > > > > updating the KIP.
> > >> > > > > >
> > >> > > > > > *Proposed API:*
> > >> > > > > > public interface Authorizer extends Configurable,
> > Closeable
> > >> {
> > >> > > > > >
> > >> > > > > > Map<Endpoint, CompletionStage<Void>>
> > >> > > start(AuthorizerServerInfo
> > >> > > > > > serverInfo);
> > >> > > > > > List<CompletionStage<AuthorizationResult>>
> > >> > > > > > authorize(AuthorizableRequestContext requestContext,
> > >> > List<Action>
> > >> > > > > > actions);
> > >> > > > > > List<CompletionStage<AclCreateResult>>
> > >> > > > > > createAcls(AuthorizableRequestContext requestContext,
> > >> > > > > List<AclBinding>
> > >> > > > > > aclBindings);
> > >> > > > > > List<CompletionStage<AclDeleteResult>>
> > >> > > > > > deleteAcls(AuthorizableRequestContext requestContext,
> > >> > > > > > List<AclBindingFilter> aclBindingFilters);
> > >> > > > > > CompletionStage<Collection<AclBinding>>
> > >> > acls(AclBindingFilter
> > >> > > > > > filter);
> > >> > > > > > }
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Thank you,
> > >> > > > > >
> > >> > > > > > Rajini
> > >> > > > > >
> > >> > > > > > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <
> > >> > > bosco@apache.org>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi Rajini
> > >> > > > > > >
> > >> > > > > > > Thanks for clarifying. I am good for now.
> > >> > > > > > >
> > >> > > > > > > Regards
> > >> > > > > > >
> > >> > > > > > > Bosco
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <
> > >> > > rajinisivaram@gmail.com
> > >> > > > >
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > Hi Don,
> > >> > > > > > >
> > >> > > > > > > That should be fine. I guess Ranger loads
> > policies
> > >> from
> > >> > the
> > >> > > > > > database
> > >> > > > > > > synchronously when the authorizer is configured,
> > >> similar
> > >> > to
> > >> > > > > > > SimpleAclAuthorizer loading from ZooKeeper.
> > Ranger can
> > >> > > > continue
> > >> > > > > > to load
> > >> > > > > > > synchronously from `configure()` or `start()`
> > and
> > >> return
> > >> > an
> > >> > > > > > empty map
> > >> > > > > > > from
> > >> > > > > > > `start()`. That would retain the existing
> > behaviour..
> > >> > When
> > >> > > > the
> > >> > > > > > same
> > >> > > > > > > database stores policies for all listeners and
> > the
> > >> > policies
> > >> > > > are
> > >> > > > > > not
> > >> > > > > > > stored
> > >> > > > > > > in Kafka, there is no value in making the load
> > >> > > asynchronous.
> > >> > > > > > >
> > >> > > > > > > Regards,
> > >> > > > > > >
> > >> > > > > > > Rajini
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai
> > <
> > >> > > > > > bosco@apache.org>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi Rajini
> > >> > > > > > > >
> > >> > > > > > > > Assuming this doesn't affect custom plugins,
> > I don't
> > >> > have
> > >> > > > any
> > >> > > > > > > concerns
> > >> > > > > > > > regarding this.
> > >> > > > > > > >
> > >> > > > > > > > I do have one question regarding:
> > >> > > > > > > >
> > >> > > > > > > > "For authorizers that don’t store metadata in
> > >> > ZooKeeper,
> > >> > > > > > ensure that
> > >> > > > > > > > authorizer metadata for each listener is
> > available
> > >> > before
> > >> > > > > > starting
> > >> > > > > > > up the
> > >> > > > > > > > listener. This enables different authorization
> > >> metadata
> > >> > > > > stores
> > >> > > > > > for
> > >> > > > > > > > different listeners."
> > >> > > > > > > >
> > >> > > > > > > > Since Ranger uses its own database for storing
> > >> > policies,
> > >> > > do
> > >> > > > > you
> > >> > > > > > > anticipate
> > >> > > > > > > > any issues?
> > >> > > > > > > >
> > >> > > > > > > > Thanks
> > >> > > > > > > >
> > >> > > > > > > > Bosco
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
> > >> > > > > > rajinisivaram@gmail.com>
> > >> > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > Hi all,
> > >> > > > > > > >
> > >> > > > > > > > I made another change to the KIP. The KIP
> > was
> > >> > > > originally
> > >> > > > > > > proposing to
> > >> > > > > > > > extend SimpleAclAuthorizer to also
> > implement the
> > >> > new
> > >> > > > API
> > >> > > > > > (in
> > >> > > > > > > addition
> > >> > > > > > > > to
> > >> > > > > > > > the existing API). But since we use the
> > new API
> > >> > when
> > >> > > > > > available,
> > >> > > > > > > this
> > >> > > > > > > > can
> > >> > > > > > > > break custom authorizers that extend this
> > class
> > >> and
> > >> > > > > > override
> > >> > > > > > > specific
> > >> > > > > > > > methods of the old API. To avoid breaking
> > any
> > >> > > existing
> > >> > > > > > custom
> > >> > > > > > > > implementations that extend this class,
> > >> > particularly
> > >> > > > > > because it
> > >> > > > > > > is in
> > >> > > > > > > > the
> > >> > > > > > > > public package kafka.security.auth, the
> > KIP now
> > >> > > > proposes
> > >> > > > > to
> > >> > > > > > > retain the
> > >> > > > > > > > old
> > >> > > > > > > > authorizer as-is. SimpleAclAuthorizer
> > will
> > >> > continue
> > >> > > to
> > >> > > > > > > implement the
> > >> > > > > > > > old
> > >> > > > > > > > API, but will be deprecated. A new
> > authorizer
> > >> > > > > > implementation
> > >> > > > > > > > kafka.security.authorizer.AclAuthorizer
> > will be
> > >> > added
> > >> > > > for
> > >> > > > > > the
> > >> > > > > > > new API
> > >> > > > > > > > (this
> > >> > > > > > > > will not be in the public package).
> > >> > > > > > > >
> > >> > > > > > > > Please let me know if you have any
> > concerns.
> > >> > > > > > > >
> > >> > > > > > > > Regards,
> > >> > > > > > > >
> > >> > > > > > > > Rajini
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Fri, Aug 16, 2019 at 8:48 AM Rajini
> > Sivaram <
> > >> > > > > > > > rajinisivaram@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Thanks Colin.
> > >> > > > > > > > >
> > >> > > > > > > > > If there are no other concerns, I will
> > start
> > >> vote
> > >> > > > later
> > >> > > > > > today.
> > >> > > > > > > Many
> > >> > > > > > > > thanks
> > >> > > > > > > > > to every one for the feedback.
> > >> > > > > > > > >
> > >> > > > > > > > > Regards,
> > >> > > > > > > > >
> > >> > > > > > > > > Rajini
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Thu, Aug 15, 2019 at 11:57 PM Colin
> > McCabe
> > >> <
> > >> > > > > > > cmccabe@apache.org>
> > >> > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > >> Thanks, Rajini. It looks good to me.
> > >> > > > > > > > >>
> > >> > > > > > > > >> best,
> > >> > > > > > > > >> Colin
> > >> > > > > > > > >>
> > >> > > > > > > > >>
> > >> > > > > > > > >> On Thu, Aug 15, 2019, at 11:37, Rajini
> > >> Sivaram
> > >> > > > wrote:
> > >> > > > > > > > >> > Hi Colin,
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Thanks for the review. I have
> > updated the
> > >> KIP
> > >> > to
> > >> > > > > move
> > >> > > > > > the
> > >> > > > > > > > interfaces for
> > >> > > > > > > > >> > request context and server info to
> > the
> > >> > > authorizer
> > >> > > > > > package.
> > >> > > > > > > These
> > >> > > > > > > > are now
> > >> > > > > > > > >> > called AuthorizableRequestContext and
> > >> > > > > > AuthorizerServerInfo.
> > >> > > > > > > > Endpoint is
> > >> > > > > > > > >> now
> > >> > > > > > > > >> > a class in org.apache.kafka.common
> > to make
> > >> it
> > >> > > > > reusable
> > >> > > > > > > since we
> > >> > > > > > > > already
> > >> > > > > > > > >> > have multiple implementations of it.
> > I have
> > >> > > > removed
> > >> > > > > > > requestName
> > >> > > > > > > > from the
> > >> > > > > > > > >> > request context interface since
> > authorizers
> > >> > can
> > >> > > > > > distinguish
> > >> > > > > > > > follower
> > >> > > > > > > > >> fetch
> > >> > > > > > > > >> > and consumer fetch from the
> > operation being
> > >> > > > > > authorized. So
> > >> > > > > > > 16-bit
> > >> > > > > > > > >> request
> > >> > > > > > > > >> > type should be sufficient for audit
> > >> logging.
> > >> > > Also
> > >> > > > > > replaced
> > >> > > > > > > > AuditFlag
> > >> > > > > > > > >> with
> > >> > > > > > > > >> > two booleans as you suggested.
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Can you take another look and see if
> > the
> > >> KIP
> > >> > is
> > >> > > > > ready
> > >> > > > > > for
> > >> > > > > > > voting?
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Thanks for all your help!
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Regards,
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Rajini
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin
> > >> McCabe <
> > >> > > > > > > cmccabe@apache.org>
> > >> > > > > > > > >> wrote:
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > > Hi Rajini,
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > I think it would be good to rename
> > >> > > > > > KafkaRequestContext to
> > >> > > > > > > > something
> > >> > > > > > > > >> like
> > >> > > > > > > > >> > > AuthorizableRequestContext, and
> > put it in
> > >> > the
> > >> > > > > > > > >> > > org.apache.kafka.server.authorizer
> > >> > namespace.
> > >> > > > If
> > >> > > > > > we put
> > >> > > > > > > it in
> > >> > > > > > > > the
> > >> > > > > > > > >> > > org.apache.kafka.common namespace,
> > then
> > >> it's
> > >> > > not
> > >> > > > > > really
> > >> > > > > > > clear
> > >> > > > > > > > that
> > >> > > > > > > > >> it's
> > >> > > > > > > > >> > > part of the Authorizer API. Since
> > this
> > >> > class
> > >> > > is
> > >> > > > > > purely an
> > >> > > > > > > > interface,
> > >> > > > > > > > >> with
> > >> > > > > > > > >> > > no concrete implementation of
> > anything,
> > >> > > there's
> > >> > > > > > nothing
> > >> > > > > > > common
> > >> > > > > > > > to
> > >> > > > > > > > >> really
> > >> > > > > > > > >> > > reuse in any case. We definitely
> > don't
> > >> want
> > >> > > > > > someone to
> > >> > > > > > > > accidentally
> > >> > > > > > > > >> add or
> > >> > > > > > > > >> > > remove methods because they think
> > this is
> > >> > just
> > >> > > > > > another
> > >> > > > > > > internal
> > >> > > > > > > > class
> > >> > > > > > > > >> used
> > >> > > > > > > > >> > > for requests.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > The BrokerInfo class is a nice
> > >> improvement.
> > >> > > It
> > >> > > > > > looks
> > >> > > > > > > like it
> > >> > > > > > > > will be
> > >> > > > > > > > >> > > useful for passing in information
> > about
> > >> the
> > >> > > > > context
> > >> > > > > > we're
> > >> > > > > > > > running
> > >> > > > > > > > >> in. It
> > >> > > > > > > > >> > > would be nice to call this class
> > >> ServerInfo
> > >> > > > rather
> > >> > > > > > than
> > >> > > > > > > > BrokerInfo,
> > >> > > > > > > > >> since
> > >> > > > > > > > >> > > we will want to run the authorizer
> > on
> > >> > > > controllers
> > >> > > > > > as well
> > >> > > > > > > as on
> > >> > > > > > > > >> brokers,
> > >> > > > > > > > >> > > and the controller may run as a
> > separate
> > >> > > process
> > >> > > > > > post
> > >> > > > > > > KIP-500.
> > >> > > > > > > > I also
> > >> > > > > > > > >> > > think that this class should be in
> > the
> > >> > > > > > > > >> org.apache.kafka.server.authorizer
> > >> > > > > > > > >> > > namespace. Again, it is an
> > interface,
> > >> not a
> > >> > > > > > concrete
> > >> > > > > > > > implementation,
> > >> > > > > > > > >> and
> > >> > > > > > > > >> > > it's an interface that is very
> > >> specifically
> > >> > > for
> > >> > > > > the
> > >> > > > > > > authorizer.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > I agree that we probably don't have
> > >> enough
> > >> > > > > > information
> > >> > > > > > > > preserved for
> > >> > > > > > > > >> > > requests currently to always know
> > what
> > >> > entity
> > >> > > > made
> > >> > > > > > them.
> > >> > > > > > > So we
> > >> > > > > > > > can
> > >> > > > > > > > >> leave
> > >> > > > > > > > >> > > that out for now (except in the
> > special
> > >> case
> > >> > > of
> > >> > > > > > Fetch).
> > >> > > > > > > > Perhaps we
> > >> > > > > > > > >> can add
> > >> > > > > > > > >> > > this later if it's needed.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > I understand the intention behind
> > >> > > > > AuthorizationMode
> > >> > > > > > > (which is
> > >> > > > > > > > now
> > >> > > > > > > > >> called
> > >> > > > > > > > >> > > AuditFlag in the latest
> > revision). But
> > >> it
> > >> > > still
> > >> > > > > > feels
> > >> > > > > > > > complex. What
> > >> > > > > > > > >> if we
> > >> > > > > > > > >> > > just had two booleans in Action:
> > >> > logSuccesses
> > >> > > > and
> > >> > > > > > > logFailures?
> > >> > > > > > > > That
> > >> > > > > > > > >> seems
> > >> > > > > > > > >> > > to cover all the cases here.
> > >> > > > MANDATORY_AUTHORIZE
> > >> > > > > =
> > >> > > > > > true,
> > >> > > > > > > true.
> > >> > > > > > > > >> > > OPTIONAL_AUTHORIZE = true, false.
> > >> FILTER =
> > >> > > > true,
> > >> > > > > > false.
> > >> > > > > > > > >> LIST_AUTHORIZED =
> > >> > > > > > > > >> > > false, false. Would there be
> > anything
> > >> lost
> > >> > > > versus
> > >> > > > > > having
> > >> > > > > > > the
> > >> > > > > > > > enum?
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > best,
> > >> > > > > > > > >> > > Colin
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > On Wed, Aug 14, 2019, at 06:29,
> > Mickael
> > >> > Maison
> > >> > > > > > wrote:
> > >> > > > > > > > >> > > > Hi Rajini,
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > Thanks for the KIP!
> > >> > > > > > > > >> > > > I really like that authorize()
> > will be
> > >> > able
> > >> > > to
> > >> > > > > > take a
> > >> > > > > > > batch of
> > >> > > > > > > > >> > > > requests, this will speed up many
> > >> > > > > implementations!
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM
> > Rajini
> > >> > > > Sivaram <
> > >> > > > > > > > >> rajinisivaram@gmail.com>
> > >> > > > > > > > >> > > wrote:
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > Thanks David! I have fixed the
> > typo.
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > Also made a couple of changes
> > to make
> > >> > the
> > >> > > > > > context
> > >> > > > > > > > interfaces more
> > >> > > > > > > > >> > > generic.
> > >> > > > > > > > >> > > > > KafkaRequestContext now
> > returns the
> > >> > 16-bit
> > >> > > > API
> > >> > > > > > key as
> > >> > > > > > > Colin
> > >> > > > > > > > >> suggested
> > >> > > > > > > > >> > > as
> > >> > > > > > > > >> > > > > well as the friendly name used
> > in
> > >> > metrics
> > >> > > > > which
> > >> > > > > > are
> > >> > > > > > > useful
> > >> > > > > > > > in
> > >> > > > > > > > >> audit
> > >> > > > > > > > >> > > logs.
> > >> > > > > > > > >> > > > > `Authorizer#start` is now
> > provided a
> > >> > > generic
> > >> > > > > > > `BrokerInfo`
> > >> > > > > > > > >> interface
> > >> > > > > > > > >> > > that
> > >> > > > > > > > >> > > > > gives cluster id, broker id and
> > >> endpoint
> > >> > > > > > information.
> > >> > > > > > > The
> > >> > > > > > > > generic
> > >> > > > > > > > >> > > interface
> > >> > > > > > > > >> > > > > can potentially be used in
> > other
> > >> broker
> > >> > > > > plugins
> > >> > > > > > in
> > >> > > > > > > future
> > >> > > > > > > > and
> > >> > > > > > > > >> provides
> > >> > > > > > > > >> > > > > dynamically generated configs
> > like
> > >> > broker
> > >> > > id
> > >> > > > > > and ports
> > >> > > > > > > > which are
> > >> > > > > > > > >> > > currently
> > >> > > > > > > > >> > > > > not available to plugins
> > unless these
> > >> > > > configs
> > >> > > > > > are
> > >> > > > > > > statically
> > >> > > > > > > > >> > > configured.
> > >> > > > > > > > >> > > > > Please let me know if there
> > are any
> > >> > > > concerns.
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > Regards,
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > Rajini
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > On Tue, Aug 13, 2019 at 4:30
> > PM David
> > >> > > Jacot
> > >> > > > <
> > >> > > > > > > > djacot@confluent.io>
> > >> > > > > > > > >> > > wrote:
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > > Hi Rajini,
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > Thank you for the update! It
> > looks
> > >> > good
> > >> > > to
> > >> > > > > me.
> > >> > > > > > > There is a
> > >> > > > > > > > typo
> > >> > > > > > > > >> in the
> > >> > > > > > > > >> > > > > > `AuditFlag` enum:
> > >> > `MANDATORY_AUTHOEIZE`
> > >> > > ->
> > >> > > > > > > > >> `MANDATORY_AUTHORIZE`.
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > Regards,
> > >> > > > > > > > >> > > > > > David
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > On Mon, Aug 12, 2019 at 2:54
> > PM
> > >> Rajini
> > >> > > > > > Sivaram <
> > >> > > > > > > > >> > > rajinisivaram@gmail.com>
> > >> > > > > > > > >> > > > > > wrote:
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > > Hi David,
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > Thanks for reviewing the
> > KIP!
> > >> Since
> > >> > > > > > questions
> > >> > > > > > > about
> > >> > > > > > > > >> `authorization
> > >> > > > > > > > >> > > mode`
> > >> > > > > > > > >> > > > > > > and `count` have come up
> > multiple
> > >> > > > times, I
> > >> > > > > > have
> > >> > > > > > > renamed
> > >> > > > > > > > both.
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > 1) Renamed `count` to
> > >> > > > > > `resourceReferenceCount`.
> > >> > > > > > > It is
> > >> > > > > > > > the
> > >> > > > > > > > >> number
> > >> > > > > > > > >> > > of times
> > >> > > > > > > > >> > > > > > > the resource being
> > authorized is
> > >> > > > > referenced
> > >> > > > > > > within the
> > >> > > > > > > > >> request.
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > 2) Renamed
> > `AuthorizationMode` to
> > >> > > > > > `AuditFlag`. It
> > >> > > > > > > is
> > >> > > > > > > > provided
> > >> > > > > > > > >> to
> > >> > > > > > > > >> > > improve
> > >> > > > > > > > >> > > > > > > audit logging in the
> > authorizer.
> > >> The
> > >> > > > enum
> > >> > > > > > values
> > >> > > > > > > have
> > >> > > > > > > > javadoc
> > >> > > > > > > > >> which
> > >> > > > > > > > >> > > > > > > indicate how the
> > authorization
> > >> > result
> > >> > > is
> > >> > > > > > used in
> > >> > > > > > > each
> > >> > > > > > > > of the
> > >> > > > > > > > >> modes
> > >> > > > > > > > >> > > to
> > >> > > > > > > > >> > > > > > > enable authorizers to log
> > audit
> > >> > > messages
> > >> > > > > at
> > >> > > > > > the
> > >> > > > > > > right
> > >> > > > > > > > severity
> > >> > > > > > > > >> > > level.
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > Regards,
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > Rajini
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > On Mon, Aug 12, 2019 at
> > 5:54 PM
> > >> > David
> > >> > > > > Jacot
> > >> > > > > > <
> > >> > > > > > > > >> djacot@confluent.io>
> > >> > > > > > > > >> > > wrote:
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > > Hi Rajini,
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > Thank you for the KIP.
> > >> Overall, it
> > >> > > > looks
> > >> > > > > > good
> > >> > > > > > > to me.
> > >> > > > > > > > I have
> > >> > > > > > > > >> few
> > >> > > > > > > > >> > > > > > > > questions/suggestions:
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > 1. It is hard to grasp
> > what
> > >> > > > > > `Action#count` is
> > >> > > > > > > for. I
> > >> > > > > > > > guess I
> > >> > > > > > > > >> > > understand
> > >> > > > > > > > >> > > > > > > > where you want to go
> > with it
> > >> but
> > >> > it
> > >> > > > took
> > >> > > > > > me a
> > >> > > > > > > while to
> > >> > > > > > > > >> figure it
> > >> > > > > > > > >> > > out.
> > >> > > > > > > > >> > > > > > > > Perhaps, we could come
> > up with
> > >> a
> > >> > > > better
> > >> > > > > > name
> > >> > > > > > > than
> > >> > > > > > > > `count`?
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > 2. I had a hard time
> > trying to
> > >> > > > > understand
> > >> > > > > > the
> > >> > > > > > > > >> `AuthorizationMode`
> > >> > > > > > > > >> > > > > > > concept,
> > >> > > > > > > > >> > > > > > > > especially wrt. the
> > OPTIONAL
> > >> one.
> > >> > My
> > >> > > > > > > understanding is
> > >> > > > > > > > that
> > >> > > > > > > > >> an
> > >> > > > > > > > >> > > ACL is
> > >> > > > > > > > >> > > > > > > either
> > >> > > > > > > > >> > > > > > > > defined or not. Could you
> > >> > elaborate
> > >> > > a
> > >> > > > > bit
> > >> > > > > > more
> > >> > > > > > > on
> > >> > > > > > > > that?
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > Thanks,
> > >> > > > > > > > >> > > > > > > > David
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > On Fri, Aug 9, 2019 at
> > 10:26 PM
> > >> > Don
> > >> > > > > Bosco
> > >> > > > > > Durai
> > >> > > > > > > <
> > >> > > > > > > > >> > > bosco@apache.org>
> > >> > > > > > > > >> > > > > > > wrote:
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Hi Rajini
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > 3.2 - This makes sense.
> > >> Thanks
> > >> > for
> > >> > > > > > clarifying.
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Rest looks fine. Once
> > the
> > >> > > > > > implementations are
> > >> > > > > > > done,
> > >> > > > > > > > it
> > >> > > > > > > > >> will be
> > >> > > > > > > > >> > > more
> > >> > > > > > > > >> > > > > > > clear
> > >> > > > > > > > >> > > > > > > > > on the different values
> > >> > > RequestType
> > >> > > > > and
> > >> > > > > > Mode.
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Thanks
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Bosco
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > On 8/9/19, 5:08 AM,
> > "Rajini
> > >> > > > Sivaram"
> > >> > > > > <
> > >> > > > > > > > >> rajinisivaram@gmail.com
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > > > wrote:
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Hi Don,
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Thanks for the
> > >> suggestions.
> > >> > A
> > >> > > > few
> > >> > > > > > > responses
> > >> > > > > > > > below:
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > 3.1 Can rename and
> > >> improve
> > >> > > docs
> > >> > > > if
> > >> > > > > > we keep
> > >> > > > > > > > this. Let's
> > >> > > > > > > > >> > > finish the
> > >> > > > > > > > >> > > > > > > > > discussion on
> > Colin's
> > >> > > > suggestions
> > >> > > > > > > regarding this
> > >> > > > > > > > >> first.
> > >> > > > > > > > >> > > > > > > > > 3.2 No, I was
> > thinking of
> > >> > some
> > >> > > > > > requests
> > >> > > > > > > that
> > >> > > > > > > > have an
> > >> > > > > > > > >> old
> > >> > > > > > > > >> > > way of
> > >> > > > > > > > >> > > > > > > > > authorizing
> > >> > > > > > > > >> > > > > > > > > and a new way
> > where we
> > >> have
> > >> > > > > > retained the
> > >> > > > > > > old
> > >> > > > > > > > way for
> > >> > > > > > > > >> > > backward
> > >> > > > > > > > >> > > > > > > > > compatibility. One
> > >> example
> > >> > is
> > >> > > > > > > Cluster:Create
> > >> > > > > > > > >> permission to
> > >> > > > > > > > >> > > create
> > >> > > > > > > > >> > > > > > > > > topics.
> > >> > > > > > > > >> > > > > > > > > We have replaced
> > this
> > >> with
> > >> > > > > > fine-grained
> > >> > > > > > > topic
> > >> > > > > > > > create
> > >> > > > > > > > >> > > access using
> > >> > > > > > > > >> > > > > > > > > Topic:Create
> > >> > > > > > > > >> > > > > > > > > for topic
> > patterns. But
> > >> we
> > >> > > still
> > >> > > > > > check if
> > >> > > > > > > user
> > >> > > > > > > > has
> > >> > > > > > > > >> > > Cluster:Create
> > >> > > > > > > > >> > > > > > > > > first. If
> > >> > > > > > > > >> > > > > > > > > Denied, the deny is
> > >> ignored
> > >> > > and
> > >> > > > we
> > >> > > > > > check
> > >> > > > > > > > >> Topic:Create. We
> > >> > > > > > > > >> > > dont
> > >> > > > > > > > >> > > > > > want
> > >> > > > > > > > >> > > > > > > > to
> > >> > > > > > > > >> > > > > > > > > log
> > >> > > > > > > > >> > > > > > > > > DENY for
> > Cluster:Create
> > >> at
> > >> > > INFO
> > >> > > > > > level for
> > >> > > > > > > this,
> > >> > > > > > > > since
> > >> > > > > > > > >> this
> > >> > > > > > > > >> > > is
> > >> > > > > > > > >> > > > > > not a
> > >> > > > > > > > >> > > > > > > > > mandatory ACL for
> > >> creating
> > >> > > > topics.
> > >> > > > > > We
> > >> > > > > > > will get
> > >> > > > > > > > >> appropriate
> > >> > > > > > > > >> > > logs
> > >> > > > > > > > >> > > > > > > from
> > >> > > > > > > > >> > > > > > > > > the
> > >> > > > > > > > >> > > > > > > > > subsequent
> > Topic:Create
> > >> in
> > >> > > this
> > >> > > > > > case.
> > >> > > > > > > > >> > > > > > > > > 3.3 They are not
> > quite
> > >> the
> > >> > > same.
> > >> > > > > > FILTER
> > >> > > > > > > implies
> > >> > > > > > > > that
> > >> > > > > > > > >> user
> > >> > > > > > > > >> > > > > > actually
> > >> > > > > > > > >> > > > > > > > > requested access
> > to and
> > >> > > > performed
> > >> > > > > > some
> > >> > > > > > > > operation on
> > >> > > > > > > > >> the
> > >> > > > > > > > >> > > filtered
> > >> > > > > > > > >> > > > > > > > > resources.
> > >> > > > > > > > >> > > > > > > > > LIST_AUTHORZED did
> > not
> > >> > result
> > >> > > in
> > >> > > > > any
> > >> > > > > > > actual
> > >> > > > > > > > access.
> > >> > > > > > > > >> User
> > >> > > > > > > > >> > > simply
> > >> > > > > > > > >> > > > > > > > wanted
> > >> > > > > > > > >> > > > > > > > > to
> > >> > > > > > > > >> > > > > > > > > know what they are
> > >> allowed
> > >> > to
> > >> > > > > > access.
> > >> > > > > > > > >> > > > > > > > > 3.4 Request types
> > are
> > >> > Produce,
> > >> > > > > > JoinGroup,
> > >> > > > > > > > OffsetCommit
> > >> > > > > > > > >> > > etc. So
> > >> > > > > > > > >> > > > > > that
> > >> > > > > > > > >> > > > > > > > is
> > >> > > > > > > > >> > > > > > > > > different from
> > >> authorization
> > >> > > > mode,
> > >> > > > > > > operation
> > >> > > > > > > > etc.
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > On Thu, Aug 8,
> > 2019 at
> > >> 11:36
> > >> > > PM
> > >> > > > > Don
> > >> > > > > > Bosco
> > >> > > > > > > Durai
> > >> > > > > > > > <
> > >> > > > > > > > >> > > > > > bosco@apache.org>
> > >> > > > > > > > >> > > > > > > > > wrote:
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > > Hi Rajini
> > >> > > > > > > > >> > > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > > Thanks for
> > clarifying.
> > >> > This
> > >> > > is
> > >> > > > > > very
> > >> > > > > > > helpful...
> > >> > > > > > > > >> > > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > > #1 - On the
> > Ranger
> > >> side,
> > >> > we
> > >> > > > > > should be
> > >> > > > > > > able to
> > >> > > > > > > > handle
> > >> > > > > > > > >> > > multiple
> > >> > > > > > > > >> > > > > > > > > requests at
> > >> > > > > > > > >> > > > > > > > > > the same time. I
> > was
> > >> just
> > >> > > not
> > >> > > > > > sure how
> > >> > > > > > > much
> > >> > > > > > > > >> processing
> > >> > > > > > > > >> > > overhead
> > >> > > > > > > > >> > > > > > > > will
> > >> > > > > > > > >> > > > > > > > > be
> > >> > > > > > > > >> > > > > > > > > > there on the
> > Broker
> > >> side
> > >> > to
> > >> > > > > split
> > >> > > > > > and
> > >> > > > > > > then
> > >> > > > > > > > >> consolidate
> > >> > > > > > > > >> > > the
> > >> > > > > > > > >> > > > > > > results.
> > >> > > > > > > > >> > > > > > > > > If it
> > >> > > > > > > > >> > > > > > > > > > is negligible,
> > then
> > >> this
> > >> > is
> > >> > > > the
> > >> > > > > > better
> > >> > > > > > > way.
> > >> > > > > > > > It will
> > >> > > > > > > > >> > > make it
> > >> > > > > > > > >> > > > > > > future
> > >> > > > > > > > >> > > > > > > > > proof.
> > >> > > > > > > > >> > > > > > > > > > #2 - I agree,
> > having a
> > >> > > single
> > >> > > > > > "start"
> > >> > > > > > > call
> > >> > > > > > > > makes it
> > >> > > > > > > > >> > > cleaner.
> > >> > > > > > > > >> > > > > > The
> > >> > > > > > > > >> > > > > > > > > > Authorization
> > >> > implementation
> > >> > > > > will
> > >> > > > > > only
> > >> > > > > > > have
> > >> > > > > > > > to do
> > >> > > > > > > > >> the
> > >> > > > > > > > >> > > > > > > > initialization
> > >> > > > > > > > >> > > > > > > > > only
> > >> > > > > > > > >> > > > > > > > > > once.
> > >> > > > > > > > >> > > > > > > > > > #3.1 - Thanks
> > for the
> > >> > > > > > clarification. I
> > >> > > > > > > think
> > >> > > > > > > > it
> > >> > > > > > > > >> makes
> > >> > > > > > > > >> > > sense to
> > >> > > > > > > > >> > > > > > > have
> > >> > > > > > > > >> > > > > > > > > this.
> > >> > > > > > > > >> > > > > > > > > > The term "Mode"
> > might
> > >> not
> > >> > be
> > >> > > > > > explicit
> > >> > > > > > > enough.
> > >> > > > > > > > >> > > Essentially it
> > >> > > > > > > > >> > > > > > > seems
> > >> > > > > > > > >> > > > > > > > > you want
> > >> > > > > > > > >> > > > > > > > > > the Authorizer
> > to know
> > >> the
> > >> > > > > > > Intent/Purpose of
> > >> > > > > > > > the
> > >> > > > > > > > >> > > authorize call
> > >> > > > > > > > >> > > > > > > and
> > >> > > > > > > > >> > > > > > > > > let the
> > >> > > > > > > > >> > > > > > > > > > Authorizer
> > decide what
> > >> to
> > >> > > log
> > >> > > > as
> > >> > > > > > Audit
> > >> > > > > > > event.
> > >> > > > > > > > >> Changing
> > >> > > > > > > > >> > > the
> > >> > > > > > > > >> > > > > > > > > class/field name
> > >> > > > > > > > >> > > > > > > > > > or giving more
> > >> > documentation
> > >> > > > > will
> > >> > > > > > do.
> > >> > > > > > > > >> > > > > > > > > > #3.2 - Regarding
> > the
> > >> > option
> > >> > > > > > "OPTIONAL".
> > >> > > > > > > Are
> > >> > > > > > > > you
> > >> > > > > > > > >> thinking
> > >> > > > > > > > >> > > from
> > >> > > > > > > > >> > > > > > > > > chaining
> > >> > > > > > > > >> > > > > > > > > > multiple
> > Authorizer? If
> > >> > > so, I
> > >> > > > > am
> > >> > > > > > not
> > >> > > > > > > sure
> > >> > > > > > > > whether
> > >> > > > > > > > >> the
> > >> > > > > > > > >> > > Broker
> > >> > > > > > > > >> > > > > > > would
> > >> > > > > > > > >> > > > > > > > > have
> > >> > > > > > > > >> > > > > > > > > > enough
> > information to
> > >> make
> > >> > > > that
> > >> > > > > > > decision. I
> > >> > > > > > > > feel the
> > >> > > > > > > > >> > > Authorizer
> > >> > > > > > > > >> > > > > > > > will
> > >> > > > > > > > >> > > > > > > > > be the
> > >> > > > > > > > >> > > > > > > > > > one who would
> > have that
> > >> > > > > > knowledge. E.g.
> > >> > > > > > > in
> > >> > > > > > > > Ranger
> > >> > > > > > > > >> we have
> > >> > > > > > > > >> > > > > > > explicit
> > >> > > > > > > > >> > > > > > > > > deny,
> > >> > > > > > > > >> > > > > > > > > > which means no
> > matter
> > >> > what,
> > >> > > > the
> > >> > > > > > request
> > >> > > > > > > > should be
> > >> > > > > > > > >> denied
> > >> > > > > > > > >> > > for
> > >> > > > > > > > >> > > > > > the
> > >> > > > > > > > >> > > > > > > > > user/group
> > >> > > > > > > > >> > > > > > > > > > or condition. So
> > if you
> > >> > are
> > >> > > > > > thinking of
> > >> > > > > > > > chaining
> > >> > > > > > > > >> > > Authorizers,
> > >> > > > > > > > >> > > > > > > then
> > >> > > > > > > > >> > > > > > > > > > responses should
> > have
> > >> the
> > >> > > > third
> > >> > > > > > state,
> > >> > > > > > > e.g.
> > >> > > > > > > > >> > > "DENIED_FINAL", in
> > >> > > > > > > > >> > > > > > > > which
> > >> > > > > > > > >> > > > > > > > > case
> > >> > > > > > > > >> > > > > > > > > > if there is an
> > >> > Authorization
> > >> > > > > > chain, it
> > >> > > > > > > will
> > >> > > > > > > > be stop
> > >> > > > > > > > >> and
> > >> > > > > > > > >> > > the
> > >> > > > > > > > >> > > > > > > request
> > >> > > > > > > > >> > > > > > > > > will be
> > >> > > > > > > > >> > > > > > > > > > denied and if it
> > is
> > >> just
> > >> > > > denied,
> > >> > > > > > then
> > >> > > > > > > you
> > >> > > > > > > > might fall
> > >> > > > > > > > >> > > back to
> > >> > > > > > > > >> > > > > > next
> > >> > > > > > > > >> > > > > > > > > > authorizer. If
> > we don't
> > >> > have
> > >> > > > > > chaining of
> > >> > > > > > > > >> Authorizing in
> > >> > > > > > > > >> > > mind,
> > >> > > > > > > > >> > > > > > > then
> > >> > > > > > > > >> > > > > > > > we
> > >> > > > > > > > >> > > > > > > > > > should reconsider
> > >> OPTIONAL
> > >> > > for
> > >> > > > > > now. Or
> > >> > > > > > > > clarify under
> > >> > > > > > > > >> > > which
> > >> > > > > > > > >> > > > > > > scenario
> > >> > > > > > > > >> > > > > > > > > > OPTIONAL will be
> > >> called.
> > >> > > > > > > > >> > > > > > > > > > #3.3 Regarding,
> > FILTER
> > >> v/s
> > >> > > > > > > LIST_AUTHORIZED,
> > >> > > > > > > > isn't
> > >> > > > > > > > >> > > > > > > LIST_AUTHORIZED a
> > >> > > > > > > > >> > > > > > > > > > special case for
> > >> "FILTER
> > >
> > >
> >
> >
> >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Colin McCabe <cm...@apache.org>.
Hi Rajini,
I agree that there is probably some value in async createAcls and deleteAcls methods. It's also a smaller change to use a futures-based interface there, since it really only affects the CreateAcls and DeleteAcls operations. It's probably fine for the other methods to be sync for now.... we would need a longer discussion about how to manage and monitor the authorization purgatory to make this async, and it's not clear that we need it (yet?)
best,
Colin
On Fri, Sep 6, 2019, at 11:32, Rajini Sivaram wrote:
> I would suggest that we go with synchronous authorize() and acls() method
> to describe ACLs, but asynchronous createAcls() and deleteAcls() that
> return CompletionStage. With KIP-500, I think async create/delete methods
> will be useful. For now, we can assume that brokers are able to cache all
> ACLs. That keeps the API and implementation simple, while still avoiding
> blocking request threads for the expensive update operations. Even though
> we don't expect large rates of concurrent ACL updates, blocking request
> threads during updates (e.g. to a database) isn't ideal. Since custom
> authorizers can still do synchronous updates by returning completed
> futures, would this be a reasonable change to the API?
>
> Thanks,
>
> Rajini
>
>
> On Fri, Sep 6, 2019 at 5:32 PM Don Bosco Durai <bo...@apache.org> wrote:
>
> > +1
> >
> > I might be wrong here, but here are few of my assumptions..
> >
> > 1. Unlike transactional services, in Kafka most of the users are
> > application users, so once the connection is established, then they are
> > going to be long running sessions. So the caching is more predictable and
> > heavy lifting can be done during initial session setup and cache can be
> > frequently updated using background thread. KSQL might have a different
> > flow, but I don't have enough datapoints for it.
> > 2. For publish/consume authorization calls, providing async option will be
> > misleading for Kafka like load
> > 3. For admin calls, hopefully no one does hundreds of call in multiple
> > thread within few seconds. This would be a bad design.
> >
> > So I feel, we should be okay with sync calls for now. And consider async
> > in the future if needed. At least in Apache Ranger implementation, we would
> > be okay using sync operation. I am not sure if any other plugin
> > implementation would benefit from async implementation.
> >
> > Thanks
> >
> > Bosco
> >
> > On 9/6/19, 8:45 AM, "Ismael Juma" <is...@juma.me.uk> wrote:
> >
> > One more thing: if people have strong opinions that authorize has to be
> > sync, we could leave it like that for now. If needed, we can later add
> > an
> > authorizeAsync with another method returning a Boolean indicating that
> > it's
> > supported. That is, there is a path to evolve the interface (even if a
> > bit
> > ugly).
> >
> > What about the other methods, is there consensus that they should be
> > updated to return CompletionStage?
> >
> > Ismael
> >
> > On Fri, Sep 6, 2019, 8:32 AM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > I'm on the move, but what Rajini said seems reasonable. I don't think
> > > using SSDs solves the issue. They can still hang for seconds when
> > they
> > > fail. Also, many people may not have local SSDs available (remote
> > SSDs like
> > > EBS hang for tens of seconds when there are issues).
> > >
> > > We are currently vulnerable to all of these in the normal read/write
> > path,
> > > but it seems suboptimal to continue assuming non blocking behavior
> > for
> > > things that actually do block.
> > >
> > > Ismael
> > >
> > > On Fri, Sep 6, 2019, 8:24 AM Rajini Sivaram <rajinisivaram@gmail.com
> > >
> > > wrote:
> > >
> > >> Hi Jun,
> > >>
> > >> For ACLs, the size is also dependent on the number of users. So in
> > >> deployments with large number of users where some users are not
> > active, an
> > >> LRU cache may be useful. Agree that there may be other solutions
> > that help
> > >> avoid the requirement for an async API even for this case.
> > >>
> > >> I wasn't expecting our threading model to change very much. Having
> > said
> > >> that, I haven't figured out how the purgatory can be adapted to work
> > >> efficiently for this. I was thinking:
> > >> 1) If authorization future is complete when returning from
> > authorize()
> > >> call, we invoke the method immediately on the request thread
> > >> 2) If future is not complete, the request goes into a purgatory
> > >> 3) When future completes, we mark the request ready to be run. This
> > can be
> > >> done on ForkJoinPool.commonPool or an executor. But we don't handle
> > the
> > >> request on that thread
> > >> 4) We handle the request on the request thread after 3).
> > >>
> > >> I totally agree that this adds complexity to the code. Will wait for
> > >> Ismael
> > >> to comment on whether he had a different model in mind.
> > >>
> > >> Thanks,
> > >>
> > >> Rajini
> > >>
> > >>
> > >> On Thu, Sep 5, 2019 at 10:29 PM Jun Rao <ju...@confluent.io> wrote:
> > >>
> > >> > Hi, Ismael,
> > >> >
> > >> > Yes, I agree that there are 2 separate discussion points, one on
> > the API
> > >> > and the other on the implementation.
> > >> >
> > >> > First on the API, in general, my feeling is that the authorize()
> > call is
> > >> > expected to be fast to reduce request latency. Adding an async API
> > >> could be
> > >> > mis-leading since it might give users the impression that it
> > would be
> > >> > arbitrarily long. In the short term, currently, we cache all
> > partition
> > >> > level metadata on every broker. The authorization metadata is
> > >> proportional
> > >> > to the number of topics, which typically is less in size. So it
> > we can
> > >> > cache all partition level metadata, we should be able to cache all
> > >> > authorization metadata. Longer term, if we want to support more
> > metadata
> > >> > beyond memory, another option is to put all metadata in an SSD
> > backed
> > >> > key/value store. The lookup time could still be under 1ms, which
> > we
> > >> could
> > >> > potentially absorb in the request threads. So, I am not sure async
> > >> > authorize() api is strictly needed even in the long term.
> > >> >
> > >> > Second on the implementation. I was more concerned about the
> > impact on
> > >> our
> > >> > threading model. If we make authorize() async, some threads have
> > to
> > >> > complete the future when it's ready. It seems to me that those
> > threads
> > >> have
> > >> > to be inside the Authorizer implementation since only it knows
> > when to
> > >> > complete a future? If so, it feels weird that all the logic of
> > request
> > >> > handling post authorize() is now handled by external threads
> > inside a
> > >> > plugin. It's not clear how we can configure and monitor them, and
> > how
> > >> > throttling would work. Perhaps we can provide a bit more detail
> > on the
> > >> new
> > >> > threading model.
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Jun
> > >> >
> > >> > On Wed, Sep 4, 2019 at 9:56 PM Ismael Juma <is...@juma.me.uk>
> > wrote:
> > >> >
> > >> > > Hi Jun,
> > >> > >
> > >> > > I think it's important to discuss the pluggable API versus the
> > calling
> > >> > > implementation as two separate points.
> > >> > >
> > >> > > From an API perspective, are you suggesting that we should tell
> > users
> > >> > that
> > >> > > they cannot block in authorize? Or are you saying that it's OK
> > to
> > >> block
> > >> > on
> > >> > > authorize on occasion and the recommendation would be for
> > people to
> > >> > > increase the number of threads in the request thread pool?
> > >> > Architecturally,
> > >> > > this feels wrong in my opinion.
> > >> > >
> > >> > > From the calling implementation perspective, you don't need a
> > >> callback.
> > >> > You
> > >> > > can basically say:
> > >> > >
> > >> > > authorize(...).map { result =>
> > >> > > ...
> > >> > > }
> > >> > >
> > >> > > And then have a common method take that Future and handle it
> > >> > synchronously
> > >> > > if it's already complete or submit it to the purgatory if not.
> > This is
> > >> > > similar to how many modern async web libraries work.
> > >> > >
> > >> > > As Rajini said, this could be done later. The initial
> > implementation
> > >> > could
> > >> > > simply do `toCompletableFuture().get()` to do it synchronously.
> > But it
> > >> > > would mean that we could add this functionality without
> > changing the
> > >> > > pluggable interface.
> > >> > >
> > >> > > Ismael
> > >> > >
> > >> > > On Wed, Sep 4, 2019 at 5:29 AM Jun Rao <ju...@confluent.io>
> > wrote:
> > >> > >
> > >> > > > Hi, Rajini,
> > >> > > >
> > >> > > > Thanks for the KIP. I was concerned about #4 too. If we
> > change the
> > >> > > handling
> > >> > > > of all requests to use an async authorize() api, will that
> > cause the
> > >> > code
> > >> > > > much harder to understand? There are quite a few callbacks
> > already.
> > >> I
> > >> > am
> > >> > > > not sure that we want to introduce more of those. The benefit
> > from
> > >> > async
> > >> > > > authorize() api seems limited.
> > >> > > >
> > >> > > > Jun
> > >> > > >
> > >> > > > On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <
> > >> rajinisivaram@gmail.com
> > >> > >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Hi Don,
> > >> > > > >
> > >> > > > > Thanks for your note.
> > >> > > > >
> > >> > > > > 1) The intention is to avoid blocking in the calling
> > thread. We
> > >> > already
> > >> > > > > have several requests that are put into a purgatory when
> > waiting
> > >> for
> > >> > > > remote
> > >> > > > > communication, for example produce request waiting for
> > >> replication.
> > >> > > Since
> > >> > > > > we have a limited number of request threads in the broker,
> > we
> > >> want to
> > >> > > > make
> > >> > > > > progress with other requests, while one is waiting on any
> > form of
> > >> > > remote
> > >> > > > > communication.
> > >> > > > >
> > >> > > > > 2) Async management calls will be useful with the default
> > >> authorizer
> > >> > > when
> > >> > > > > KIP-500 removes ZK and we rely on Kafka instead. Our current
> > >> ZK-based
> > >> > > > > implementation as well as any custom implementations that
> > don't
> > >> want
> > >> > to
> > >> > > > be
> > >> > > > > async will just need to return a sync'ed value. So instead
> > of
> > >> > > returning `
> > >> > > > > value`, the code would just return
> > >> > > > > `CompletableFuture.completedFuture(value)
> > >> > > > > `. So it would be just a single line change in the
> > implementation
> > >> > with
> > >> > > > the
> > >> > > > > new API. The caller would treat completedFuture exactly as
> > it does
> > >> > > today,
> > >> > > > > processing the request synchronously without using a
> > purgatory.
> > >> > > > >
> > >> > > > > 3) For implementations that return a completedFuture as
> > described
> > >> in
> > >> > > 2),
> > >> > > > > the behaviour would remain exactly the same. No additional
> > >> threads or
> > >> > > > > purgatory will be used for this case. So there would be no
> > >> > performance
> > >> > > > > penalty. For implementations that return a future that is
> > not
> > >> > complete,
> > >> > > > we
> > >> > > > > prioritise running more requests concurrently. So in a
> > deployment
> > >> > with
> > >> > > a
> > >> > > > > large number of clients, we would improve performance by
> > allowing
> > >> > other
> > >> > > > > requests to be processed on the request threads while some
> > are
> > >> > waiting
> > >> > > > for
> > >> > > > > authorization metadata.
> > >> > > > >
> > >> > > > > 4) I was concerned about this too. The goal is to make the
> > API
> > >> > flexible
> > >> > > > > enough to handle large scale deployments in future when
> > caching
> > >> all
> > >> > > > > authorization metadata in each broker is not viable. Using
> > an
> > >> async
> > >> > API
> > >> > > > > that returns CompletionStage, the caller has the option to
> > handle
> > >> the
> > >> > > > > result synchronously or asynchronously, so we don't
> > necessarily
> > >> need
> > >> > to
> > >> > > > > update the calling code right away. Custom authorizers
> > using the
> > >> > async
> > >> > > > API
> > >> > > > > have full control over whether authorization is performed
> > in-line
> > >> > since
> > >> > > > > completedFuture will always be handled synchronously. We do
> > need
> > >> to
> > >> > > > update
> > >> > > > > KafkaApis to take advantage of the asynchronous API to
> > improve
> > >> scale.
> > >> > > > Even
> > >> > > > > though this is a big change, since we will be doing the
> > same for
> > >> all
> > >> > > > > requests, it shouldn't be too hard to maintain since the
> > same
> > >> pattern
> > >> > > > will
> > >> > > > > be used for all requests.
> > >> > > > >
> > >> > > > > Regards,
> > >> > > > >
> > >> > > > > Rajini
> > >> > > > >
> > >> > > > > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <
> > bosco@apache.org
> > >> >
> > >> > > > wrote:
> > >> > > > >
> > >> > > > > > Hi Rajini
> > >> > > > > >
> > >> > > > > > Help me understand this a bit more.
> > >> > > > > >
> > >> > > > > > 1. For all practical purpose, without authorization you
> > can't
> > >> go to
> > >> > > the
> > >> > > > > > next step. The calling code needs to block anyway. So
> > should we
> > >> > just
> > >> > > > let
> > >> > > > > > the implementation code do the async part?
> > >> > > > > > 2. If you feel management calls need to be async, then we
> > should
> > >> > > > consider
> > >> > > > > > another set of async APIs. I don't feel we should
> > complicate it
> > >> > > > further (
> > >> > > > > > 3. Another concern I have is wrt performance. Kafka has
> > been
> > >> built
> > >> > to
> > >> > > > > > handle 1000s per second requests. Not sure whether making
> > it
> > >> async
> > >> > > will
> > >> > > > > add
> > >> > > > > > any unnecessary overhead.
> > >> > > > > > 4. How much complication would this add on the calling
> > side?
> > >> And is
> > >> > > it
> > >> > > > > > worth it?
> > >> > > > > >
> > >> > > > > > Thanks
> > >> > > > > >
> > >> > > > > > Bosco
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On 9/3/19, 8:50 AM, "Rajini Sivaram" <
> > rajinisivaram@gmail.com>
> > >> > > wrote:
> > >> > > > > >
> > >> > > > > > Hi all,
> > >> > > > > >
> > >> > > > > > Ismael brought up a point that it will be good to
> > make the
> > >> > > > Authorizer
> > >> > > > > > interface asynchronous to avoid blocking request
> > threads
> > >> during
> > >> > > > > remote
> > >> > > > > > operations.
> > >> > > > > >
> > >> > > > > > 1) Since we want to support different backends for
> > >> > authorization
> > >> > > > > > metadata,
> > >> > > > > > making createAcls() and deleteAcls() asynchronous
> > makes
> > >> sense
> > >> > > since
> > >> > > > > > these
> > >> > > > > > always involve remote operations. When KIP-500 removes
> > >> > ZooKeeper,
> > >> > > > we
> > >> > > > > > would
> > >> > > > > > want to move ACLs to Kafka and async updates will
> > avoid
> > >> > > unnecessary
> > >> > > > > > blocking.
> > >> > > > > > 2) For authorize() method, we currently use cached
> > ACLs in
> > >> the
> > >> > > > > built-in
> > >> > > > > > authorizers, so synchronous authorise operations work
> > well
> > >> now.
> > >> > > But
> > >> > > > > > async
> > >> > > > > > authorize() would support this scenario as well as
> > >> authorizers
> > >> > in
> > >> > > > > large
> > >> > > > > > organisations where an LRU cache would enable a
> > smaller
> > >> cache
> > >> > > even
> > >> > > > > > when the
> > >> > > > > > backend holds a large amount of ACLs for infrequently
> > used
> > >> > > > resources
> > >> > > > > or
> > >> > > > > > users who don't use the system frequently.
> > >> > > > > >
> > >> > > > > > For both cases, the built-in authorizer will continue
> > to be
> > >> > > > > > synchronous,
> > >> > > > > > using CompletableFuture.completedFuture() to return
> > the
> > >> actual
> > >> > > > > result.
> > >> > > > > > But
> > >> > > > > > async API will make custom authorizer implementations
> > more
> > >> > > > flexible.
> > >> > > > > I
> > >> > > > > > would like to know if there are any concerns with
> > these
> > >> changes
> > >> > > > > before
> > >> > > > > > updating the KIP.
> > >> > > > > >
> > >> > > > > > *Proposed API:*
> > >> > > > > > public interface Authorizer extends Configurable,
> > Closeable
> > >> {
> > >> > > > > >
> > >> > > > > > Map<Endpoint, CompletionStage<Void>>
> > >> > > start(AuthorizerServerInfo
> > >> > > > > > serverInfo);
> > >> > > > > > List<CompletionStage<AuthorizationResult>>
> > >> > > > > > authorize(AuthorizableRequestContext requestContext,
> > >> > List<Action>
> > >> > > > > > actions);
> > >> > > > > > List<CompletionStage<AclCreateResult>>
> > >> > > > > > createAcls(AuthorizableRequestContext requestContext,
> > >> > > > > List<AclBinding>
> > >> > > > > > aclBindings);
> > >> > > > > > List<CompletionStage<AclDeleteResult>>
> > >> > > > > > deleteAcls(AuthorizableRequestContext requestContext,
> > >> > > > > > List<AclBindingFilter> aclBindingFilters);
> > >> > > > > > CompletionStage<Collection<AclBinding>>
> > >> > acls(AclBindingFilter
> > >> > > > > > filter);
> > >> > > > > > }
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > Thank you,
> > >> > > > > >
> > >> > > > > > Rajini
> > >> > > > > >
> > >> > > > > > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <
> > >> > > bosco@apache.org>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi Rajini
> > >> > > > > > >
> > >> > > > > > > Thanks for clarifying. I am good for now.
> > >> > > > > > >
> > >> > > > > > > Regards
> > >> > > > > > >
> > >> > > > > > > Bosco
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <
> > >> > > rajinisivaram@gmail.com
> > >> > > > >
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > Hi Don,
> > >> > > > > > >
> > >> > > > > > > That should be fine. I guess Ranger loads
> > policies
> > >> from
> > >> > the
> > >> > > > > > database
> > >> > > > > > > synchronously when the authorizer is configured,
> > >> similar
> > >> > to
> > >> > > > > > > SimpleAclAuthorizer loading from ZooKeeper.
> > Ranger can
> > >> > > > continue
> > >> > > > > > to load
> > >> > > > > > > synchronously from `configure()` or `start()`
> > and
> > >> return
> > >> > an
> > >> > > > > > empty map
> > >> > > > > > > from
> > >> > > > > > > `start()`. That would retain the existing
> > behaviour..
> > >> > When
> > >> > > > the
> > >> > > > > > same
> > >> > > > > > > database stores policies for all listeners and
> > the
> > >> > policies
> > >> > > > are
> > >> > > > > > not
> > >> > > > > > > stored
> > >> > > > > > > in Kafka, there is no value in making the load
> > >> > > asynchronous.
> > >> > > > > > >
> > >> > > > > > > Regards,
> > >> > > > > > >
> > >> > > > > > > Rajini
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai
> > <
> > >> > > > > > bosco@apache.org>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi Rajini
> > >> > > > > > > >
> > >> > > > > > > > Assuming this doesn't affect custom plugins,
> > I don't
> > >> > have
> > >> > > > any
> > >> > > > > > > concerns
> > >> > > > > > > > regarding this.
> > >> > > > > > > >
> > >> > > > > > > > I do have one question regarding:
> > >> > > > > > > >
> > >> > > > > > > > "For authorizers that don’t store metadata in
> > >> > ZooKeeper,
> > >> > > > > > ensure that
> > >> > > > > > > > authorizer metadata for each listener is
> > available
> > >> > before
> > >> > > > > > starting
> > >> > > > > > > up the
> > >> > > > > > > > listener. This enables different authorization
> > >> metadata
> > >> > > > > stores
> > >> > > > > > for
> > >> > > > > > > > different listeners."
> > >> > > > > > > >
> > >> > > > > > > > Since Ranger uses its own database for storing
> > >> > policies,
> > >> > > do
> > >> > > > > you
> > >> > > > > > > anticipate
> > >> > > > > > > > any issues?
> > >> > > > > > > >
> > >> > > > > > > > Thanks
> > >> > > > > > > >
> > >> > > > > > > > Bosco
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
> > >> > > > > > rajinisivaram@gmail.com>
> > >> > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > Hi all,
> > >> > > > > > > >
> > >> > > > > > > > I made another change to the KIP. The KIP
> > was
> > >> > > > originally
> > >> > > > > > > proposing to
> > >> > > > > > > > extend SimpleAclAuthorizer to also
> > implement the
> > >> > new
> > >> > > > API
> > >> > > > > > (in
> > >> > > > > > > addition
> > >> > > > > > > > to
> > >> > > > > > > > the existing API). But since we use the
> > new API
> > >> > when
> > >> > > > > > available,
> > >> > > > > > > this
> > >> > > > > > > > can
> > >> > > > > > > > break custom authorizers that extend this
> > class
> > >> and
> > >> > > > > > override
> > >> > > > > > > specific
> > >> > > > > > > > methods of the old API. To avoid breaking
> > any
> > >> > > existing
> > >> > > > > > custom
> > >> > > > > > > > implementations that extend this class,
> > >> > particularly
> > >> > > > > > because it
> > >> > > > > > > is in
> > >> > > > > > > > the
> > >> > > > > > > > public package kafka.security.auth, the
> > KIP now
> > >> > > > proposes
> > >> > > > > to
> > >> > > > > > > retain the
> > >> > > > > > > > old
> > >> > > > > > > > authorizer as-is. SimpleAclAuthorizer
> > will
> > >> > continue
> > >> > > to
> > >> > > > > > > implement the
> > >> > > > > > > > old
> > >> > > > > > > > API, but will be deprecated. A new
> > authorizer
> > >> > > > > > implementation
> > >> > > > > > > > kafka.security.authorizer.AclAuthorizer
> > will be
> > >> > added
> > >> > > > for
> > >> > > > > > the
> > >> > > > > > > new API
> > >> > > > > > > > (this
> > >> > > > > > > > will not be in the public package).
> > >> > > > > > > >
> > >> > > > > > > > Please let me know if you have any
> > concerns.
> > >> > > > > > > >
> > >> > > > > > > > Regards,
> > >> > > > > > > >
> > >> > > > > > > > Rajini
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Fri, Aug 16, 2019 at 8:48 AM Rajini
> > Sivaram <
> > >> > > > > > > > rajinisivaram@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Thanks Colin.
> > >> > > > > > > > >
> > >> > > > > > > > > If there are no other concerns, I will
> > start
> > >> vote
> > >> > > > later
> > >> > > > > > today.
> > >> > > > > > > Many
> > >> > > > > > > > thanks
> > >> > > > > > > > > to every one for the feedback.
> > >> > > > > > > > >
> > >> > > > > > > > > Regards,
> > >> > > > > > > > >
> > >> > > > > > > > > Rajini
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Thu, Aug 15, 2019 at 11:57 PM Colin
> > McCabe
> > >> <
> > >> > > > > > > cmccabe@apache.org>
> > >> > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > >> Thanks, Rajini. It looks good to me.
> > >> > > > > > > > >>
> > >> > > > > > > > >> best,
> > >> > > > > > > > >> Colin
> > >> > > > > > > > >>
> > >> > > > > > > > >>
> > >> > > > > > > > >> On Thu, Aug 15, 2019, at 11:37, Rajini
> > >> Sivaram
> > >> > > > wrote:
> > >> > > > > > > > >> > Hi Colin,
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Thanks for the review. I have
> > updated the
> > >> KIP
> > >> > to
> > >> > > > > move
> > >> > > > > > the
> > >> > > > > > > > interfaces for
> > >> > > > > > > > >> > request context and server info to
> > the
> > >> > > authorizer
> > >> > > > > > package.
> > >> > > > > > > These
> > >> > > > > > > > are now
> > >> > > > > > > > >> > called AuthorizableRequestContext and
> > >> > > > > > AuthorizerServerInfo.
> > >> > > > > > > > Endpoint is
> > >> > > > > > > > >> now
> > >> > > > > > > > >> > a class in org.apache.kafka.common
> > to make
> > >> it
> > >> > > > > reusable
> > >> > > > > > > since we
> > >> > > > > > > > already
> > >> > > > > > > > >> > have multiple implementations of it.
> > I have
> > >> > > > removed
> > >> > > > > > > requestName
> > >> > > > > > > > from the
> > >> > > > > > > > >> > request context interface since
> > authorizers
> > >> > can
> > >> > > > > > distinguish
> > >> > > > > > > > follower
> > >> > > > > > > > >> fetch
> > >> > > > > > > > >> > and consumer fetch from the
> > operation being
> > >> > > > > > authorized. So
> > >> > > > > > > 16-bit
> > >> > > > > > > > >> request
> > >> > > > > > > > >> > type should be sufficient for audit
> > >> logging.
> > >> > > Also
> > >> > > > > > replaced
> > >> > > > > > > > AuditFlag
> > >> > > > > > > > >> with
> > >> > > > > > > > >> > two booleans as you suggested.
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Can you take another look and see if
> > the
> > >> KIP
> > >> > is
> > >> > > > > ready
> > >> > > > > > for
> > >> > > > > > > voting?
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Thanks for all your help!
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Regards,
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > Rajini
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin
> > >> McCabe <
> > >> > > > > > > cmccabe@apache.org>
> > >> > > > > > > > >> wrote:
> > >> > > > > > > > >> >
> > >> > > > > > > > >> > > Hi Rajini,
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > I think it would be good to rename
> > >> > > > > > KafkaRequestContext to
> > >> > > > > > > > something
> > >> > > > > > > > >> like
> > >> > > > > > > > >> > > AuthorizableRequestContext, and
> > put it in
> > >> > the
> > >> > > > > > > > >> > > org.apache.kafka.server.authorizer
> > >> > namespace.
> > >> > > > If
> > >> > > > > > we put
> > >> > > > > > > it in
> > >> > > > > > > > the
> > >> > > > > > > > >> > > org.apache.kafka.common namespace,
> > then
> > >> it's
> > >> > > not
> > >> > > > > > really
> > >> > > > > > > clear
> > >> > > > > > > > that
> > >> > > > > > > > >> it's
> > >> > > > > > > > >> > > part of the Authorizer API. Since
> > this
> > >> > class
> > >> > > is
> > >> > > > > > purely an
> > >> > > > > > > > interface,
> > >> > > > > > > > >> with
> > >> > > > > > > > >> > > no concrete implementation of
> > anything,
> > >> > > there's
> > >> > > > > > nothing
> > >> > > > > > > common
> > >> > > > > > > > to
> > >> > > > > > > > >> really
> > >> > > > > > > > >> > > reuse in any case. We definitely
> > don't
> > >> want
> > >> > > > > > someone to
> > >> > > > > > > > accidentally
> > >> > > > > > > > >> add or
> > >> > > > > > > > >> > > remove methods because they think
> > this is
> > >> > just
> > >> > > > > > another
> > >> > > > > > > internal
> > >> > > > > > > > class
> > >> > > > > > > > >> used
> > >> > > > > > > > >> > > for requests.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > The BrokerInfo class is a nice
> > >> improvement.
> > >> > > It
> > >> > > > > > looks
> > >> > > > > > > like it
> > >> > > > > > > > will be
> > >> > > > > > > > >> > > useful for passing in information
> > about
> > >> the
> > >> > > > > context
> > >> > > > > > we're
> > >> > > > > > > > running
> > >> > > > > > > > >> in. It
> > >> > > > > > > > >> > > would be nice to call this class
> > >> ServerInfo
> > >> > > > rather
> > >> > > > > > than
> > >> > > > > > > > BrokerInfo,
> > >> > > > > > > > >> since
> > >> > > > > > > > >> > > we will want to run the authorizer
> > on
> > >> > > > controllers
> > >> > > > > > as well
> > >> > > > > > > as on
> > >> > > > > > > > >> brokers,
> > >> > > > > > > > >> > > and the controller may run as a
> > separate
> > >> > > process
> > >> > > > > > post
> > >> > > > > > > KIP-500.
> > >> > > > > > > > I also
> > >> > > > > > > > >> > > think that this class should be in
> > the
> > >> > > > > > > > >> org.apache.kafka.server.authorizer
> > >> > > > > > > > >> > > namespace. Again, it is an
> > interface,
> > >> not a
> > >> > > > > > concrete
> > >> > > > > > > > implementation,
> > >> > > > > > > > >> and
> > >> > > > > > > > >> > > it's an interface that is very
> > >> specifically
> > >> > > for
> > >> > > > > the
> > >> > > > > > > authorizer.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > I agree that we probably don't have
> > >> enough
> > >> > > > > > information
> > >> > > > > > > > preserved for
> > >> > > > > > > > >> > > requests currently to always know
> > what
> > >> > entity
> > >> > > > made
> > >> > > > > > them.
> > >> > > > > > > So we
> > >> > > > > > > > can
> > >> > > > > > > > >> leave
> > >> > > > > > > > >> > > that out for now (except in the
> > special
> > >> case
> > >> > > of
> > >> > > > > > Fetch).
> > >> > > > > > > > Perhaps we
> > >> > > > > > > > >> can add
> > >> > > > > > > > >> > > this later if it's needed.
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > I understand the intention behind
> > >> > > > > AuthorizationMode
> > >> > > > > > > (which is
> > >> > > > > > > > now
> > >> > > > > > > > >> called
> > >> > > > > > > > >> > > AuditFlag in the latest
> > revision). But
> > >> it
> > >> > > still
> > >> > > > > > feels
> > >> > > > > > > > complex. What
> > >> > > > > > > > >> if we
> > >> > > > > > > > >> > > just had two booleans in Action:
> > >> > logSuccesses
> > >> > > > and
> > >> > > > > > > logFailures?
> > >> > > > > > > > That
> > >> > > > > > > > >> seems
> > >> > > > > > > > >> > > to cover all the cases here.
> > >> > > > MANDATORY_AUTHORIZE
> > >> > > > > =
> > >> > > > > > true,
> > >> > > > > > > true.
> > >> > > > > > > > >> > > OPTIONAL_AUTHORIZE = true, false.
> > >> FILTER =
> > >> > > > true,
> > >> > > > > > false.
> > >> > > > > > > > >> LIST_AUTHORIZED =
> > >> > > > > > > > >> > > false, false. Would there be
> > anything
> > >> lost
> > >> > > > versus
> > >> > > > > > having
> > >> > > > > > > the
> > >> > > > > > > > enum?
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > best,
> > >> > > > > > > > >> > > Colin
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > >
> > >> > > > > > > > >> > > On Wed, Aug 14, 2019, at 06:29,
> > Mickael
> > >> > Maison
> > >> > > > > > wrote:
> > >> > > > > > > > >> > > > Hi Rajini,
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > Thanks for the KIP!
> > >> > > > > > > > >> > > > I really like that authorize()
> > will be
> > >> > able
> > >> > > to
> > >> > > > > > take a
> > >> > > > > > > batch of
> > >> > > > > > > > >> > > > requests, this will speed up many
> > >> > > > > implementations!
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM
> > Rajini
> > >> > > > Sivaram <
> > >> > > > > > > > >> rajinisivaram@gmail.com>
> > >> > > > > > > > >> > > wrote:
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > Thanks David! I have fixed the
> > typo.
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > Also made a couple of changes
> > to make
> > >> > the
> > >> > > > > > context
> > >> > > > > > > > interfaces more
> > >> > > > > > > > >> > > generic.
> > >> > > > > > > > >> > > > > KafkaRequestContext now
> > returns the
> > >> > 16-bit
> > >> > > > API
> > >> > > > > > key as
> > >> > > > > > > Colin
> > >> > > > > > > > >> suggested
> > >> > > > > > > > >> > > as
> > >> > > > > > > > >> > > > > well as the friendly name used
> > in
> > >> > metrics
> > >> > > > > which
> > >> > > > > > are
> > >> > > > > > > useful
> > >> > > > > > > > in
> > >> > > > > > > > >> audit
> > >> > > > > > > > >> > > logs.
> > >> > > > > > > > >> > > > > `Authorizer#start` is now
> > provided a
> > >> > > generic
> > >> > > > > > > `BrokerInfo`
> > >> > > > > > > > >> interface
> > >> > > > > > > > >> > > that
> > >> > > > > > > > >> > > > > gives cluster id, broker id and
> > >> endpoint
> > >> > > > > > information.
> > >> > > > > > > The
> > >> > > > > > > > generic
> > >> > > > > > > > >> > > interface
> > >> > > > > > > > >> > > > > can potentially be used in
> > other
> > >> broker
> > >> > > > > plugins
> > >> > > > > > in
> > >> > > > > > > future
> > >> > > > > > > > and
> > >> > > > > > > > >> provides
> > >> > > > > > > > >> > > > > dynamically generated configs
> > like
> > >> > broker
> > >> > > id
> > >> > > > > > and ports
> > >> > > > > > > > which are
> > >> > > > > > > > >> > > currently
> > >> > > > > > > > >> > > > > not available to plugins
> > unless these
> > >> > > > configs
> > >> > > > > > are
> > >> > > > > > > statically
> > >> > > > > > > > >> > > configured.
> > >> > > > > > > > >> > > > > Please let me know if there
> > are any
> > >> > > > concerns.
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > Regards,
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > Rajini
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > On Tue, Aug 13, 2019 at 4:30
> > PM David
> > >> > > Jacot
> > >> > > > <
> > >> > > > > > > > djacot@confluent.io>
> > >> > > > > > > > >> > > wrote:
> > >> > > > > > > > >> > > > >
> > >> > > > > > > > >> > > > > > Hi Rajini,
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > Thank you for the update! It
> > looks
> > >> > good
> > >> > > to
> > >> > > > > me.
> > >> > > > > > > There is a
> > >> > > > > > > > typo
> > >> > > > > > > > >> in the
> > >> > > > > > > > >> > > > > > `AuditFlag` enum:
> > >> > `MANDATORY_AUTHOEIZE`
> > >> > > ->
> > >> > > > > > > > >> `MANDATORY_AUTHORIZE`.
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > Regards,
> > >> > > > > > > > >> > > > > > David
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > On Mon, Aug 12, 2019 at 2:54
> > PM
> > >> Rajini
> > >> > > > > > Sivaram <
> > >> > > > > > > > >> > > rajinisivaram@gmail.com>
> > >> > > > > > > > >> > > > > > wrote:
> > >> > > > > > > > >> > > > > >
> > >> > > > > > > > >> > > > > > > Hi David,
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > Thanks for reviewing the
> > KIP!
> > >> Since
> > >> > > > > > questions
> > >> > > > > > > about
> > >> > > > > > > > >> `authorization
> > >> > > > > > > > >> > > mode`
> > >> > > > > > > > >> > > > > > > and `count` have come up
> > multiple
> > >> > > > times, I
> > >> > > > > > have
> > >> > > > > > > renamed
> > >> > > > > > > > both.
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > 1) Renamed `count` to
> > >> > > > > > `resourceReferenceCount`.
> > >> > > > > > > It is
> > >> > > > > > > > the
> > >> > > > > > > > >> number
> > >> > > > > > > > >> > > of times
> > >> > > > > > > > >> > > > > > > the resource being
> > authorized is
> > >> > > > > referenced
> > >> > > > > > > within the
> > >> > > > > > > > >> request.
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > 2) Renamed
> > `AuthorizationMode` to
> > >> > > > > > `AuditFlag`. It
> > >> > > > > > > is
> > >> > > > > > > > provided
> > >> > > > > > > > >> to
> > >> > > > > > > > >> > > improve
> > >> > > > > > > > >> > > > > > > audit logging in the
> > authorizer.
> > >> The
> > >> > > > enum
> > >> > > > > > values
> > >> > > > > > > have
> > >> > > > > > > > javadoc
> > >> > > > > > > > >> which
> > >> > > > > > > > >> > > > > > > indicate how the
> > authorization
> > >> > result
> > >> > > is
> > >> > > > > > used in
> > >> > > > > > > each
> > >> > > > > > > > of the
> > >> > > > > > > > >> modes
> > >> > > > > > > > >> > > to
> > >> > > > > > > > >> > > > > > > enable authorizers to log
> > audit
> > >> > > messages
> > >> > > > > at
> > >> > > > > > the
> > >> > > > > > > right
> > >> > > > > > > > severity
> > >> > > > > > > > >> > > level.
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > Regards,
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > Rajini
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > On Mon, Aug 12, 2019 at
> > 5:54 PM
> > >> > David
> > >> > > > > Jacot
> > >> > > > > > <
> > >> > > > > > > > >> djacot@confluent.io>
> > >> > > > > > > > >> > > wrote:
> > >> > > > > > > > >> > > > > > >
> > >> > > > > > > > >> > > > > > > > Hi Rajini,
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > Thank you for the KIP.
> > >> Overall, it
> > >> > > > looks
> > >> > > > > > good
> > >> > > > > > > to me.
> > >> > > > > > > > I have
> > >> > > > > > > > >> few
> > >> > > > > > > > >> > > > > > > > questions/suggestions:
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > 1. It is hard to grasp
> > what
> > >> > > > > > `Action#count` is
> > >> > > > > > > for. I
> > >> > > > > > > > guess I
> > >> > > > > > > > >> > > understand
> > >> > > > > > > > >> > > > > > > > where you want to go
> > with it
> > >> but
> > >> > it
> > >> > > > took
> > >> > > > > > me a
> > >> > > > > > > while to
> > >> > > > > > > > >> figure it
> > >> > > > > > > > >> > > out.
> > >> > > > > > > > >> > > > > > > > Perhaps, we could come
> > up with
> > >> a
> > >> > > > better
> > >> > > > > > name
> > >> > > > > > > than
> > >> > > > > > > > `count`?
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > 2. I had a hard time
> > trying to
> > >> > > > > understand
> > >> > > > > > the
> > >> > > > > > > > >> `AuthorizationMode`
> > >> > > > > > > > >> > > > > > > concept,
> > >> > > > > > > > >> > > > > > > > especially wrt. the
> > OPTIONAL
> > >> one.
> > >> > My
> > >> > > > > > > understanding is
> > >> > > > > > > > that
> > >> > > > > > > > >> an
> > >> > > > > > > > >> > > ACL is
> > >> > > > > > > > >> > > > > > > either
> > >> > > > > > > > >> > > > > > > > defined or not. Could you
> > >> > elaborate
> > >> > > a
> > >> > > > > bit
> > >> > > > > > more
> > >> > > > > > > on
> > >> > > > > > > > that?
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > Thanks,
> > >> > > > > > > > >> > > > > > > > David
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > On Fri, Aug 9, 2019 at
> > 10:26 PM
> > >> > Don
> > >> > > > > Bosco
> > >> > > > > > Durai
> > >> > > > > > > <
> > >> > > > > > > > >> > > bosco@apache.org>
> > >> > > > > > > > >> > > > > > > wrote:
> > >> > > > > > > > >> > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Hi Rajini
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > 3.2 - This makes sense.
> > >> Thanks
> > >> > for
> > >> > > > > > clarifying.
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Rest looks fine. Once
> > the
> > >> > > > > > implementations are
> > >> > > > > > > done,
> > >> > > > > > > > it
> > >> > > > > > > > >> will be
> > >> > > > > > > > >> > > more
> > >> > > > > > > > >> > > > > > > clear
> > >> > > > > > > > >> > > > > > > > > on the different values
> > >> > > RequestType
> > >> > > > > and
> > >> > > > > > Mode.
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Thanks
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Bosco
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > On 8/9/19, 5:08 AM,
> > "Rajini
> > >> > > > Sivaram"
> > >> > > > > <
> > >> > > > > > > > >> rajinisivaram@gmail.com
> > >> > > > > > > > >> > > >
> > >> > > > > > > > >> > > > > > wrote:
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Hi Don,
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > Thanks for the
> > >> suggestions.
> > >> > A
> > >> > > > few
> > >> > > > > > > responses
> > >> > > > > > > > below:
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > 3.1 Can rename and
> > >> improve
> > >> > > docs
> > >> > > > if
> > >> > > > > > we keep
> > >> > > > > > > > this. Let's
> > >> > > > > > > > >> > > finish the
> > >> > > > > > > > >> > > > > > > > > discussion on
> > Colin's
> > >> > > > suggestions
> > >> > > > > > > regarding this
> > >> > > > > > > > >> first.
> > >> > > > > > > > >> > > > > > > > > 3.2 No, I was
> > thinking of
> > >> > some
> > >> > > > > > requests
> > >> > > > > > > that
> > >> > > > > > > > have an
> > >> > > > > > > > >> old
> > >> > > > > > > > >> > > way of
> > >> > > > > > > > >> > > > > > > > > authorizing
> > >> > > > > > > > >> > > > > > > > > and a new way
> > where we
> > >> have
> > >> > > > > > retained the
> > >> > > > > > > old
> > >> > > > > > > > way for
> > >> > > > > > > > >> > > backward
> > >> > > > > > > > >> > > > > > > > > compatibility. One
> > >> example
> > >> > is
> > >> > > > > > > Cluster:Create
> > >> > > > > > > > >> permission to
> > >> > > > > > > > >> > > create
> > >> > > > > > > > >> > > > > > > > > topics.
> > >> > > > > > > > >> > > > > > > > > We have replaced
> > this
> > >> with
> > >> > > > > > fine-grained
> > >> > > > > > > topic
> > >> > > > > > > > create
> > >> > > > > > > > >> > > access using
> > >> > > > > > > > >> > > > > > > > > Topic:Create
> > >> > > > > > > > >> > > > > > > > > for topic
> > patterns. But
> > >> we
> > >> > > still
> > >> > > > > > check if
> > >> > > > > > > user
> > >> > > > > > > > has
> > >> > > > > > > > >> > > Cluster:Create
> > >> > > > > > > > >> > > > > > > > > first. If
> > >> > > > > > > > >> > > > > > > > > Denied, the deny is
> > >> ignored
> > >> > > and
> > >> > > > we
> > >> > > > > > check
> > >> > > > > > > > >> Topic:Create. We
> > >> > > > > > > > >> > > dont
> > >> > > > > > > > >> > > > > > want
> > >> > > > > > > > >> > > > > > > > to
> > >> > > > > > > > >> > > > > > > > > log
> > >> > > > > > > > >> > > > > > > > > DENY for
> > Cluster:Create
> > >> at
> > >> > > INFO
> > >> > > > > > level for
> > >> > > > > > > this,
> > >> > > > > > > > since
> > >> > > > > > > > >> this
> > >> > > > > > > > >> > > is
> > >> > > > > > > > >> > > > > > not a
> > >> > > > > > > > >> > > > > > > > > mandatory ACL for
> > >> creating
> > >> > > > topics.
> > >> > > > > > We
> > >> > > > > > > will get
> > >> > > > > > > > >> appropriate
> > >> > > > > > > > >> > > logs
> > >> > > > > > > > >> > > > > > > from
> > >> > > > > > > > >> > > > > > > > > the
> > >> > > > > > > > >> > > > > > > > > subsequent
> > Topic:Create
> > >> in
> > >> > > this
> > >> > > > > > case.
> > >> > > > > > > > >> > > > > > > > > 3.3 They are not
> > quite
> > >> the
> > >> > > same.
> > >> > > > > > FILTER
> > >> > > > > > > implies
> > >> > > > > > > > that
> > >> > > > > > > > >> user
> > >> > > > > > > > >> > > > > > actually
> > >> > > > > > > > >> > > > > > > > > requested access
> > to and
> > >> > > > performed
> > >> > > > > > some
> > >> > > > > > > > operation on
> > >> > > > > > > > >> the
> > >> > > > > > > > >> > > filtered
> > >> > > > > > > > >> > > > > > > > > resources.
> > >> > > > > > > > >> > > > > > > > > LIST_AUTHORZED did
> > not
> > >> > result
> > >> > > in
> > >> > > > > any
> > >> > > > > > > actual
> > >> > > > > > > > access.
> > >> > > > > > > > >> User
> > >> > > > > > > > >> > > simply
> > >> > > > > > > > >> > > > > > > > wanted
> > >> > > > > > > > >> > > > > > > > > to
> > >> > > > > > > > >> > > > > > > > > know what they are
> > >> allowed
> > >> > to
> > >> > > > > > access.
> > >> > > > > > > > >> > > > > > > > > 3.4 Request types
> > are
> > >> > Produce,
> > >> > > > > > JoinGroup,
> > >> > > > > > > > OffsetCommit
> > >> > > > > > > > >> > > etc. So
> > >> > > > > > > > >> > > > > > that
> > >> > > > > > > > >> > > > > > > > is
> > >> > > > > > > > >> > > > > > > > > different from
> > >> authorization
> > >> > > > mode,
> > >> > > > > > > operation
> > >> > > > > > > > etc.
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > On Thu, Aug 8,
> > 2019 at
> > >> 11:36
> > >> > > PM
> > >> > > > > Don
> > >> > > > > > Bosco
> > >> > > > > > > Durai
> > >> > > > > > > > <
> > >> > > > > > > > >> > > > > > bosco@apache.org>
> > >> > > > > > > > >> > > > > > > > > wrote:
> > >> > > > > > > > >> > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > > Hi Rajini
> > >> > > > > > > > >> > > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > > Thanks for
> > clarifying.
> > >> > This
> > >> > > is
> > >> > > > > > very
> > >> > > > > > > helpful...
> > >> > > > > > > > >> > > > > > > > > >
> > >> > > > > > > > >> > > > > > > > > > #1 - On the
> > Ranger
> > >> side,
> > >> > we
> > >> > > > > > should be
> > >> > > > > > > able to
> > >> > > > > > > > handle
> > >> > > > > > > > >> > > multiple
> > >> > > > > > > > >> > > > > > > > > requests at
> > >> > > > > > > > >> > > > > > > > > > the same time. I
> > was
> > >> just
> > >> > > not
> > >> > > > > > sure how
> > >> > > > > > > much
> > >> > > > > > > > >> processing
> > >> > > > > > > > >> > > overhead
> > >> > > > > > > > >> > > > > > > > will
> > >> > > > > > > > >> > > > > > > > > be
> > >> > > > > > > > >> > > > > > > > > > there on the
> > Broker
> > >> side
> > >> > to
> > >> > > > > split
> > >> > > > > > and
> > >> > > > > > > then
> > >> > > > > > > > >> consolidate
> > >> > > > > > > > >> > > the
> > >> > > > > > > > >> > > > > > > results.
> > >> > > > > > > > >> > > > > > > > > If it
> > >> > > > > > > > >> > > > > > > > > > is negligible,
> > then
> > >> this
> > >> > is
> > >> > > > the
> > >> > > > > > better
> > >> > > > > > > way.
> > >> > > > > > > > It will
> > >> > > > > > > > >> > > make it
> > >> > > > > > > > >> > > > > > > future
> > >> > > > > > > > >> > > > > > > > > proof.
> > >> > > > > > > > >> > > > > > > > > > #2 - I agree,
> > having a
> > >> > > single
> > >> > > > > > "start"
> > >> > > > > > > call
> > >> > > > > > > > makes it
> > >> > > > > > > > >> > > cleaner.
> > >> > > > > > > > >> > > > > > The
> > >> > > > > > > > >> > > > > > > > > > Authorization
> > >> > implementation
> > >> > > > > will
> > >> > > > > > only
> > >> > > > > > > have
> > >> > > > > > > > to do
> > >> > > > > > > > >> the
> > >> > > > > > > > >> > > > > > > > initialization
> > >> > > > > > > > >> > > > > > > > > only
> > >> > > > > > > > >> > > > > > > > > > once.
> > >> > > > > > > > >> > > > > > > > > > #3.1 - Thanks
> > for the
> > >> > > > > > clarification. I
> > >> > > > > > > think
> > >> > > > > > > > it
> > >> > > > > > > > >> makes
> > >> > > > > > > > >> > > sense to
> > >> > > > > > > > >> > > > > > > have
> > >> > > > > > > > >> > > > > > > > > this.
> > >> > > > > > > > >> > > > > > > > > > The term "Mode"
> > might
> > >> not
> > >> > be
> > >> > > > > > explicit
> > >> > > > > > > enough.
> > >> > > > > > > > >> > > Essentially it
> > >> > > > > > > > >> > > > > > > seems
> > >> > > > > > > > >> > > > > > > > > you want
> > >> > > > > > > > >> > > > > > > > > > the Authorizer
> > to know
> > >> the
> > >> > > > > > > Intent/Purpose of
> > >> > > > > > > > the
> > >> > > > > > > > >> > > authorize call
> > >> > > > > > > > >> > > > > > > and
> > >> > > > > > > > >> > > > > > > > > let the
> > >> > > > > > > > >> > > > > > > > > > Authorizer
> > decide what
> > >> to
> > >> > > log
> > >> > > > as
> > >> > > > > > Audit
> > >> > > > > > > event.
> > >> > > > > > > > >> Changing
> > >> > > > > > > > >> > > the
> > >> > > > > > > > >> > > > > > > > > class/field name
> > >> > > > > > > > >> > > > > > > > > > or giving more
> > >> > documentation
> > >> > > > > will
> > >> > > > > > do.
> > >> > > > > > > > >> > > > > > > > > > #3.2 - Regarding
> > the
> > >> > option
> > >> > > > > > "OPTIONAL".
> > >> > > > > > > Are
> > >> > > > > > > > you
> > >> > > > > > > > >> thinking
> > >> > > > > > > > >> > > from
> > >> > > > > > > > >> > > > > > > > > chaining
> > >> > > > > > > > >> > > > > > > > > > multiple
> > Authorizer? If
> > >> > > so, I
> > >> > > > > am
> > >> > > > > > not
> > >> > > > > > > sure
> > >> > > > > > > > whether
> > >> > > > > > > > >> the
> > >> > > > > > > > >> > > Broker
> > >> > > > > > > > >> > > > > > > would
> > >> > > > > > > > >> > > > > > > > > have
> > >> > > > > > > > >> > > > > > > > > > enough
> > information to
> > >> make
> > >> > > > that
> > >> > > > > > > decision. I
> > >> > > > > > > > feel the
> > >> > > > > > > > >> > > Authorizer
> > >> > > > > > > > >> > > > > > > > will
> > >> > > > > > > > >> > > > > > > > > be the
> > >> > > > > > > > >> > > > > > > > > > one who would
> > have that
> > >> > > > > > knowledge. E.g.
> > >> > > > > > > in
> > >> > > > > > > > Ranger
> > >> > > > > > > > >> we have
> > >> > > > > > > > >> > > > > > > explicit
> > >> > > > > > > > >> > > > > > > > > deny,
> > >> > > > > > > > >> > > > > > > > > > which means no
> > matter
> > >> > what,
> > >> > > > the
> > >> > > > > > request
> > >> > > > > > > > should be
> > >> > > > > > > > >> denied
> > >> > > > > > > > >> > > for
> > >> > > > > > > > >> > > > > > the
> > >> > > > > > > > >> > > > > > > > > user/group
> > >> > > > > > > > >> > > > > > > > > > or condition. So
> > if you
> > >> > are
> > >> > > > > > thinking of
> > >> > > > > > > > chaining
> > >> > > > > > > > >> > > Authorizers,
> > >> > > > > > > > >> > > > > > > then
> > >> > > > > > > > >> > > > > > > > > > responses should
> > have
> > >> the
> > >> > > > third
> > >> > > > > > state,
> > >> > > > > > > e.g.
> > >> > > > > > > > >> > > "DENIED_FINAL", in
> > >> > > > > > > > >> > > > > > > > which
> > >> > > > > > > > >> > > > > > > > > case
> > >> > > > > > > > >> > > > > > > > > > if there is an
> > >> > Authorization
> > >> > > > > > chain, it
> > >> > > > > > > will
> > >> > > > > > > > be stop
> > >> > > > > > > > >> and
> > >> > > > > > > > >> > > the
> > >> > > > > > > > >> > > > > > > request
> > >> > > > > > > > >> > > > > > > > > will be
> > >> > > > > > > > >> > > > > > > > > > denied and if it
> > is
> > >> just
> > >> > > > denied,
> > >> > > > > > then
> > >> > > > > > > you
> > >> > > > > > > > might fall
> > >> > > > > > > > >> > > back to
> > >> > > > > > > > >> > > > > > next
> > >> > > > > > > > >> > > > > > > > > > authorizer. If
> > we don't
> > >> > have
> > >> > > > > > chaining of
> > >> > > > > > > > >> Authorizing in
> > >> > > > > > > > >> > > mind,
> > >> > > > > > > > >> > > > > > > then
> > >> > > > > > > > >> > > > > > > > we
> > >> > > > > > > > >> > > > > > > > > > should reconsider
> > >> OPTIONAL
> > >> > > for
> > >> > > > > > now. Or
> > >> > > > > > > > clarify under
> > >> > > > > > > > >> > > which
> > >> > > > > > > > >> > > > > > > scenario
> > >> > > > > > > > >> > > > > > > > > > OPTIONAL will be
> > >> called.
> > >> > > > > > > > >> > > > > > > > > > #3.3 Regarding,
> > FILTER
> > >> v/s
> > >> > > > > > > LIST_AUTHORIZED,
> > >> > > > > > > > isn't
> > >> > > > > > > > >> > > > > > > LIST_AUTHORIZED a
> > >> > > > > > > > >> > > > > > > > > > special case for
> > >> "FILTER
> > >
> > >
> >
> >
> >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
I would suggest that we go with synchronous authorize() and acls() method
to describe ACLs, but asynchronous createAcls() and deleteAcls() that
return CompletionStage. With KIP-500, I think async create/delete methods
will be useful. For now, we can assume that brokers are able to cache all
ACLs. That keeps the API and implementation simple, while still avoiding
blocking request threads for the expensive update operations. Even though
we don't expect large rates of concurrent ACL updates, blocking request
threads during updates (e.g. to a database) isn't ideal. Since custom
authorizers can still do synchronous updates by returning completed
futures, would this be a reasonable change to the API?
Thanks,
Rajini
On Fri, Sep 6, 2019 at 5:32 PM Don Bosco Durai <bo...@apache.org> wrote:
> +1
>
> I might be wrong here, but here are few of my assumptions..
>
> 1. Unlike transactional services, in Kafka most of the users are
> application users, so once the connection is established, then they are
> going to be long running sessions. So the caching is more predictable and
> heavy lifting can be done during initial session setup and cache can be
> frequently updated using background thread. KSQL might have a different
> flow, but I don't have enough datapoints for it.
> 2. For publish/consume authorization calls, providing async option will be
> misleading for Kafka like load
> 3. For admin calls, hopefully no one does hundreds of call in multiple
> thread within few seconds. This would be a bad design.
>
> So I feel, we should be okay with sync calls for now. And consider async
> in the future if needed. At least in Apache Ranger implementation, we would
> be okay using sync operation. I am not sure if any other plugin
> implementation would benefit from async implementation.
>
> Thanks
>
> Bosco
>
> On 9/6/19, 8:45 AM, "Ismael Juma" <is...@juma.me.uk> wrote:
>
> One more thing: if people have strong opinions that authorize has to be
> sync, we could leave it like that for now. If needed, we can later add
> an
> authorizeAsync with another method returning a Boolean indicating that
> it's
> supported. That is, there is a path to evolve the interface (even if a
> bit
> ugly).
>
> What about the other methods, is there consensus that they should be
> updated to return CompletionStage?
>
> Ismael
>
> On Fri, Sep 6, 2019, 8:32 AM Ismael Juma <is...@juma.me.uk> wrote:
>
> > I'm on the move, but what Rajini said seems reasonable. I don't think
> > using SSDs solves the issue. They can still hang for seconds when
> they
> > fail. Also, many people may not have local SSDs available (remote
> SSDs like
> > EBS hang for tens of seconds when there are issues).
> >
> > We are currently vulnerable to all of these in the normal read/write
> path,
> > but it seems suboptimal to continue assuming non blocking behavior
> for
> > things that actually do block.
> >
> > Ismael
> >
> > On Fri, Sep 6, 2019, 8:24 AM Rajini Sivaram <rajinisivaram@gmail.com
> >
> > wrote:
> >
> >> Hi Jun,
> >>
> >> For ACLs, the size is also dependent on the number of users. So in
> >> deployments with large number of users where some users are not
> active, an
> >> LRU cache may be useful. Agree that there may be other solutions
> that help
> >> avoid the requirement for an async API even for this case.
> >>
> >> I wasn't expecting our threading model to change very much. Having
> said
> >> that, I haven't figured out how the purgatory can be adapted to work
> >> efficiently for this. I was thinking:
> >> 1) If authorization future is complete when returning from
> authorize()
> >> call, we invoke the method immediately on the request thread
> >> 2) If future is not complete, the request goes into a purgatory
> >> 3) When future completes, we mark the request ready to be run. This
> can be
> >> done on ForkJoinPool.commonPool or an executor. But we don't handle
> the
> >> request on that thread
> >> 4) We handle the request on the request thread after 3).
> >>
> >> I totally agree that this adds complexity to the code. Will wait for
> >> Ismael
> >> to comment on whether he had a different model in mind.
> >>
> >> Thanks,
> >>
> >> Rajini
> >>
> >>
> >> On Thu, Sep 5, 2019 at 10:29 PM Jun Rao <ju...@confluent.io> wrote:
> >>
> >> > Hi, Ismael,
> >> >
> >> > Yes, I agree that there are 2 separate discussion points, one on
> the API
> >> > and the other on the implementation.
> >> >
> >> > First on the API, in general, my feeling is that the authorize()
> call is
> >> > expected to be fast to reduce request latency. Adding an async API
> >> could be
> >> > mis-leading since it might give users the impression that it
> would be
> >> > arbitrarily long. In the short term, currently, we cache all
> partition
> >> > level metadata on every broker. The authorization metadata is
> >> proportional
> >> > to the number of topics, which typically is less in size. So it
> we can
> >> > cache all partition level metadata, we should be able to cache all
> >> > authorization metadata. Longer term, if we want to support more
> metadata
> >> > beyond memory, another option is to put all metadata in an SSD
> backed
> >> > key/value store. The lookup time could still be under 1ms, which
> we
> >> could
> >> > potentially absorb in the request threads. So, I am not sure async
> >> > authorize() api is strictly needed even in the long term.
> >> >
> >> > Second on the implementation. I was more concerned about the
> impact on
> >> our
> >> > threading model. If we make authorize() async, some threads have
> to
> >> > complete the future when it's ready. It seems to me that those
> threads
> >> have
> >> > to be inside the Authorizer implementation since only it knows
> when to
> >> > complete a future? If so, it feels weird that all the logic of
> request
> >> > handling post authorize() is now handled by external threads
> inside a
> >> > plugin. It's not clear how we can configure and monitor them, and
> how
> >> > throttling would work. Perhaps we can provide a bit more detail
> on the
> >> new
> >> > threading model.
> >> >
> >> > Thanks,
> >> >
> >> > Jun
> >> >
> >> > On Wed, Sep 4, 2019 at 9:56 PM Ismael Juma <is...@juma.me.uk>
> wrote:
> >> >
> >> > > Hi Jun,
> >> > >
> >> > > I think it's important to discuss the pluggable API versus the
> calling
> >> > > implementation as two separate points.
> >> > >
> >> > > From an API perspective, are you suggesting that we should tell
> users
> >> > that
> >> > > they cannot block in authorize? Or are you saying that it's OK
> to
> >> block
> >> > on
> >> > > authorize on occasion and the recommendation would be for
> people to
> >> > > increase the number of threads in the request thread pool?
> >> > Architecturally,
> >> > > this feels wrong in my opinion.
> >> > >
> >> > > From the calling implementation perspective, you don't need a
> >> callback.
> >> > You
> >> > > can basically say:
> >> > >
> >> > > authorize(...).map { result =>
> >> > > ...
> >> > > }
> >> > >
> >> > > And then have a common method take that Future and handle it
> >> > synchronously
> >> > > if it's already complete or submit it to the purgatory if not.
> This is
> >> > > similar to how many modern async web libraries work.
> >> > >
> >> > > As Rajini said, this could be done later. The initial
> implementation
> >> > could
> >> > > simply do `toCompletableFuture().get()` to do it synchronously.
> But it
> >> > > would mean that we could add this functionality without
> changing the
> >> > > pluggable interface.
> >> > >
> >> > > Ismael
> >> > >
> >> > > On Wed, Sep 4, 2019 at 5:29 AM Jun Rao <ju...@confluent.io>
> wrote:
> >> > >
> >> > > > Hi, Rajini,
> >> > > >
> >> > > > Thanks for the KIP. I was concerned about #4 too. If we
> change the
> >> > > handling
> >> > > > of all requests to use an async authorize() api, will that
> cause the
> >> > code
> >> > > > much harder to understand? There are quite a few callbacks
> already.
> >> I
> >> > am
> >> > > > not sure that we want to introduce more of those. The benefit
> from
> >> > async
> >> > > > authorize() api seems limited.
> >> > > >
> >> > > > Jun
> >> > > >
> >> > > > On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <
> >> rajinisivaram@gmail.com
> >> > >
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Don,
> >> > > > >
> >> > > > > Thanks for your note.
> >> > > > >
> >> > > > > 1) The intention is to avoid blocking in the calling
> thread. We
> >> > already
> >> > > > > have several requests that are put into a purgatory when
> waiting
> >> for
> >> > > > remote
> >> > > > > communication, for example produce request waiting for
> >> replication.
> >> > > Since
> >> > > > > we have a limited number of request threads in the broker,
> we
> >> want to
> >> > > > make
> >> > > > > progress with other requests, while one is waiting on any
> form of
> >> > > remote
> >> > > > > communication.
> >> > > > >
> >> > > > > 2) Async management calls will be useful with the default
> >> authorizer
> >> > > when
> >> > > > > KIP-500 removes ZK and we rely on Kafka instead. Our current
> >> ZK-based
> >> > > > > implementation as well as any custom implementations that
> don't
> >> want
> >> > to
> >> > > > be
> >> > > > > async will just need to return a sync'ed value. So instead
> of
> >> > > returning `
> >> > > > > value`, the code would just return
> >> > > > > `CompletableFuture.completedFuture(value)
> >> > > > > `. So it would be just a single line change in the
> implementation
> >> > with
> >> > > > the
> >> > > > > new API. The caller would treat completedFuture exactly as
> it does
> >> > > today,
> >> > > > > processing the request synchronously without using a
> purgatory.
> >> > > > >
> >> > > > > 3) For implementations that return a completedFuture as
> described
> >> in
> >> > > 2),
> >> > > > > the behaviour would remain exactly the same. No additional
> >> threads or
> >> > > > > purgatory will be used for this case. So there would be no
> >> > performance
> >> > > > > penalty. For implementations that return a future that is
> not
> >> > complete,
> >> > > > we
> >> > > > > prioritise running more requests concurrently. So in a
> deployment
> >> > with
> >> > > a
> >> > > > > large number of clients, we would improve performance by
> allowing
> >> > other
> >> > > > > requests to be processed on the request threads while some
> are
> >> > waiting
> >> > > > for
> >> > > > > authorization metadata.
> >> > > > >
> >> > > > > 4) I was concerned about this too. The goal is to make the
> API
> >> > flexible
> >> > > > > enough to handle large scale deployments in future when
> caching
> >> all
> >> > > > > authorization metadata in each broker is not viable. Using
> an
> >> async
> >> > API
> >> > > > > that returns CompletionStage, the caller has the option to
> handle
> >> the
> >> > > > > result synchronously or asynchronously, so we don't
> necessarily
> >> need
> >> > to
> >> > > > > update the calling code right away. Custom authorizers
> using the
> >> > async
> >> > > > API
> >> > > > > have full control over whether authorization is performed
> in-line
> >> > since
> >> > > > > completedFuture will always be handled synchronously. We do
> need
> >> to
> >> > > > update
> >> > > > > KafkaApis to take advantage of the asynchronous API to
> improve
> >> scale.
> >> > > > Even
> >> > > > > though this is a big change, since we will be doing the
> same for
> >> all
> >> > > > > requests, it shouldn't be too hard to maintain since the
> same
> >> pattern
> >> > > > will
> >> > > > > be used for all requests.
> >> > > > >
> >> > > > > Regards,
> >> > > > >
> >> > > > > Rajini
> >> > > > >
> >> > > > > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <
> bosco@apache.org
> >> >
> >> > > > wrote:
> >> > > > >
> >> > > > > > Hi Rajini
> >> > > > > >
> >> > > > > > Help me understand this a bit more.
> >> > > > > >
> >> > > > > > 1. For all practical purpose, without authorization you
> can't
> >> go to
> >> > > the
> >> > > > > > next step. The calling code needs to block anyway. So
> should we
> >> > just
> >> > > > let
> >> > > > > > the implementation code do the async part?
> >> > > > > > 2. If you feel management calls need to be async, then we
> should
> >> > > > consider
> >> > > > > > another set of async APIs. I don't feel we should
> complicate it
> >> > > > further (
> >> > > > > > 3. Another concern I have is wrt performance. Kafka has
> been
> >> built
> >> > to
> >> > > > > > handle 1000s per second requests. Not sure whether making
> it
> >> async
> >> > > will
> >> > > > > add
> >> > > > > > any unnecessary overhead.
> >> > > > > > 4. How much complication would this add on the calling
> side?
> >> And is
> >> > > it
> >> > > > > > worth it?
> >> > > > > >
> >> > > > > > Thanks
> >> > > > > >
> >> > > > > > Bosco
> >> > > > > >
> >> > > > > >
> >> > > > > > On 9/3/19, 8:50 AM, "Rajini Sivaram" <
> rajinisivaram@gmail.com>
> >> > > wrote:
> >> > > > > >
> >> > > > > > Hi all,
> >> > > > > >
> >> > > > > > Ismael brought up a point that it will be good to
> make the
> >> > > > Authorizer
> >> > > > > > interface asynchronous to avoid blocking request
> threads
> >> during
> >> > > > > remote
> >> > > > > > operations.
> >> > > > > >
> >> > > > > > 1) Since we want to support different backends for
> >> > authorization
> >> > > > > > metadata,
> >> > > > > > making createAcls() and deleteAcls() asynchronous
> makes
> >> sense
> >> > > since
> >> > > > > > these
> >> > > > > > always involve remote operations. When KIP-500 removes
> >> > ZooKeeper,
> >> > > > we
> >> > > > > > would
> >> > > > > > want to move ACLs to Kafka and async updates will
> avoid
> >> > > unnecessary
> >> > > > > > blocking.
> >> > > > > > 2) For authorize() method, we currently use cached
> ACLs in
> >> the
> >> > > > > built-in
> >> > > > > > authorizers, so synchronous authorise operations work
> well
> >> now.
> >> > > But
> >> > > > > > async
> >> > > > > > authorize() would support this scenario as well as
> >> authorizers
> >> > in
> >> > > > > large
> >> > > > > > organisations where an LRU cache would enable a
> smaller
> >> cache
> >> > > even
> >> > > > > > when the
> >> > > > > > backend holds a large amount of ACLs for infrequently
> used
> >> > > > resources
> >> > > > > or
> >> > > > > > users who don't use the system frequently.
> >> > > > > >
> >> > > > > > For both cases, the built-in authorizer will continue
> to be
> >> > > > > > synchronous,
> >> > > > > > using CompletableFuture.completedFuture() to return
> the
> >> actual
> >> > > > > result.
> >> > > > > > But
> >> > > > > > async API will make custom authorizer implementations
> more
> >> > > > flexible.
> >> > > > > I
> >> > > > > > would like to know if there are any concerns with
> these
> >> changes
> >> > > > > before
> >> > > > > > updating the KIP.
> >> > > > > >
> >> > > > > > *Proposed API:*
> >> > > > > > public interface Authorizer extends Configurable,
> Closeable
> >> {
> >> > > > > >
> >> > > > > > Map<Endpoint, CompletionStage<Void>>
> >> > > start(AuthorizerServerInfo
> >> > > > > > serverInfo);
> >> > > > > > List<CompletionStage<AuthorizationResult>>
> >> > > > > > authorize(AuthorizableRequestContext requestContext,
> >> > List<Action>
> >> > > > > > actions);
> >> > > > > > List<CompletionStage<AclCreateResult>>
> >> > > > > > createAcls(AuthorizableRequestContext requestContext,
> >> > > > > List<AclBinding>
> >> > > > > > aclBindings);
> >> > > > > > List<CompletionStage<AclDeleteResult>>
> >> > > > > > deleteAcls(AuthorizableRequestContext requestContext,
> >> > > > > > List<AclBindingFilter> aclBindingFilters);
> >> > > > > > CompletionStage<Collection<AclBinding>>
> >> > acls(AclBindingFilter
> >> > > > > > filter);
> >> > > > > > }
> >> > > > > >
> >> > > > > >
> >> > > > > > Thank you,
> >> > > > > >
> >> > > > > > Rajini
> >> > > > > >
> >> > > > > > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <
> >> > > bosco@apache.org>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Rajini
> >> > > > > > >
> >> > > > > > > Thanks for clarifying. I am good for now.
> >> > > > > > >
> >> > > > > > > Regards
> >> > > > > > >
> >> > > > > > > Bosco
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <
> >> > > rajinisivaram@gmail.com
> >> > > > >
> >> > > > > > wrote:
> >> > > > > > >
> >> > > > > > > Hi Don,
> >> > > > > > >
> >> > > > > > > That should be fine. I guess Ranger loads
> policies
> >> from
> >> > the
> >> > > > > > database
> >> > > > > > > synchronously when the authorizer is configured,
> >> similar
> >> > to
> >> > > > > > > SimpleAclAuthorizer loading from ZooKeeper.
> Ranger can
> >> > > > continue
> >> > > > > > to load
> >> > > > > > > synchronously from `configure()` or `start()`
> and
> >> return
> >> > an
> >> > > > > > empty map
> >> > > > > > > from
> >> > > > > > > `start()`. That would retain the existing
> behaviour..
> >> > When
> >> > > > the
> >> > > > > > same
> >> > > > > > > database stores policies for all listeners and
> the
> >> > policies
> >> > > > are
> >> > > > > > not
> >> > > > > > > stored
> >> > > > > > > in Kafka, there is no value in making the load
> >> > > asynchronous.
> >> > > > > > >
> >> > > > > > > Regards,
> >> > > > > > >
> >> > > > > > > Rajini
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai
> <
> >> > > > > > bosco@apache.org>
> >> > > > > > > wrote:
> >> > > > > > >
> >> > > > > > > > Hi Rajini
> >> > > > > > > >
> >> > > > > > > > Assuming this doesn't affect custom plugins,
> I don't
> >> > have
> >> > > > any
> >> > > > > > > concerns
> >> > > > > > > > regarding this.
> >> > > > > > > >
> >> > > > > > > > I do have one question regarding:
> >> > > > > > > >
> >> > > > > > > > "For authorizers that don’t store metadata in
> >> > ZooKeeper,
> >> > > > > > ensure that
> >> > > > > > > > authorizer metadata for each listener is
> available
> >> > before
> >> > > > > > starting
> >> > > > > > > up the
> >> > > > > > > > listener. This enables different authorization
> >> metadata
> >> > > > > stores
> >> > > > > > for
> >> > > > > > > > different listeners."
> >> > > > > > > >
> >> > > > > > > > Since Ranger uses its own database for storing
> >> > policies,
> >> > > do
> >> > > > > you
> >> > > > > > > anticipate
> >> > > > > > > > any issues?
> >> > > > > > > >
> >> > > > > > > > Thanks
> >> > > > > > > >
> >> > > > > > > > Bosco
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
> >> > > > > > rajinisivaram@gmail.com>
> >> > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > Hi all,
> >> > > > > > > >
> >> > > > > > > > I made another change to the KIP. The KIP
> was
> >> > > > originally
> >> > > > > > > proposing to
> >> > > > > > > > extend SimpleAclAuthorizer to also
> implement the
> >> > new
> >> > > > API
> >> > > > > > (in
> >> > > > > > > addition
> >> > > > > > > > to
> >> > > > > > > > the existing API). But since we use the
> new API
> >> > when
> >> > > > > > available,
> >> > > > > > > this
> >> > > > > > > > can
> >> > > > > > > > break custom authorizers that extend this
> class
> >> and
> >> > > > > > override
> >> > > > > > > specific
> >> > > > > > > > methods of the old API. To avoid breaking
> any
> >> > > existing
> >> > > > > > custom
> >> > > > > > > > implementations that extend this class,
> >> > particularly
> >> > > > > > because it
> >> > > > > > > is in
> >> > > > > > > > the
> >> > > > > > > > public package kafka.security.auth, the
> KIP now
> >> > > > proposes
> >> > > > > to
> >> > > > > > > retain the
> >> > > > > > > > old
> >> > > > > > > > authorizer as-is. SimpleAclAuthorizer
> will
> >> > continue
> >> > > to
> >> > > > > > > implement the
> >> > > > > > > > old
> >> > > > > > > > API, but will be deprecated. A new
> authorizer
> >> > > > > > implementation
> >> > > > > > > > kafka.security.authorizer.AclAuthorizer
> will be
> >> > added
> >> > > > for
> >> > > > > > the
> >> > > > > > > new API
> >> > > > > > > > (this
> >> > > > > > > > will not be in the public package).
> >> > > > > > > >
> >> > > > > > > > Please let me know if you have any
> concerns.
> >> > > > > > > >
> >> > > > > > > > Regards,
> >> > > > > > > >
> >> > > > > > > > Rajini
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Fri, Aug 16, 2019 at 8:48 AM Rajini
> Sivaram <
> >> > > > > > > > rajinisivaram@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Thanks Colin.
> >> > > > > > > > >
> >> > > > > > > > > If there are no other concerns, I will
> start
> >> vote
> >> > > > later
> >> > > > > > today.
> >> > > > > > > Many
> >> > > > > > > > thanks
> >> > > > > > > > > to every one for the feedback.
> >> > > > > > > > >
> >> > > > > > > > > Regards,
> >> > > > > > > > >
> >> > > > > > > > > Rajini
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Thu, Aug 15, 2019 at 11:57 PM Colin
> McCabe
> >> <
> >> > > > > > > cmccabe@apache.org>
> >> > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > >> Thanks, Rajini. It looks good to me.
> >> > > > > > > > >>
> >> > > > > > > > >> best,
> >> > > > > > > > >> Colin
> >> > > > > > > > >>
> >> > > > > > > > >>
> >> > > > > > > > >> On Thu, Aug 15, 2019, at 11:37, Rajini
> >> Sivaram
> >> > > > wrote:
> >> > > > > > > > >> > Hi Colin,
> >> > > > > > > > >> >
> >> > > > > > > > >> > Thanks for the review. I have
> updated the
> >> KIP
> >> > to
> >> > > > > move
> >> > > > > > the
> >> > > > > > > > interfaces for
> >> > > > > > > > >> > request context and server info to
> the
> >> > > authorizer
> >> > > > > > package.
> >> > > > > > > These
> >> > > > > > > > are now
> >> > > > > > > > >> > called AuthorizableRequestContext and
> >> > > > > > AuthorizerServerInfo.
> >> > > > > > > > Endpoint is
> >> > > > > > > > >> now
> >> > > > > > > > >> > a class in org.apache.kafka.common
> to make
> >> it
> >> > > > > reusable
> >> > > > > > > since we
> >> > > > > > > > already
> >> > > > > > > > >> > have multiple implementations of it.
> I have
> >> > > > removed
> >> > > > > > > requestName
> >> > > > > > > > from the
> >> > > > > > > > >> > request context interface since
> authorizers
> >> > can
> >> > > > > > distinguish
> >> > > > > > > > follower
> >> > > > > > > > >> fetch
> >> > > > > > > > >> > and consumer fetch from the
> operation being
> >> > > > > > authorized. So
> >> > > > > > > 16-bit
> >> > > > > > > > >> request
> >> > > > > > > > >> > type should be sufficient for audit
> >> logging.
> >> > > Also
> >> > > > > > replaced
> >> > > > > > > > AuditFlag
> >> > > > > > > > >> with
> >> > > > > > > > >> > two booleans as you suggested.
> >> > > > > > > > >> >
> >> > > > > > > > >> > Can you take another look and see if
> the
> >> KIP
> >> > is
> >> > > > > ready
> >> > > > > > for
> >> > > > > > > voting?
> >> > > > > > > > >> >
> >> > > > > > > > >> > Thanks for all your help!
> >> > > > > > > > >> >
> >> > > > > > > > >> > Regards,
> >> > > > > > > > >> >
> >> > > > > > > > >> > Rajini
> >> > > > > > > > >> >
> >> > > > > > > > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin
> >> McCabe <
> >> > > > > > > cmccabe@apache.org>
> >> > > > > > > > >> wrote:
> >> > > > > > > > >> >
> >> > > > > > > > >> > > Hi Rajini,
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > I think it would be good to rename
> >> > > > > > KafkaRequestContext to
> >> > > > > > > > something
> >> > > > > > > > >> like
> >> > > > > > > > >> > > AuthorizableRequestContext, and
> put it in
> >> > the
> >> > > > > > > > >> > > org.apache.kafka.server.authorizer
> >> > namespace.
> >> > > > If
> >> > > > > > we put
> >> > > > > > > it in
> >> > > > > > > > the
> >> > > > > > > > >> > > org.apache.kafka.common namespace,
> then
> >> it's
> >> > > not
> >> > > > > > really
> >> > > > > > > clear
> >> > > > > > > > that
> >> > > > > > > > >> it's
> >> > > > > > > > >> > > part of the Authorizer API. Since
> this
> >> > class
> >> > > is
> >> > > > > > purely an
> >> > > > > > > > interface,
> >> > > > > > > > >> with
> >> > > > > > > > >> > > no concrete implementation of
> anything,
> >> > > there's
> >> > > > > > nothing
> >> > > > > > > common
> >> > > > > > > > to
> >> > > > > > > > >> really
> >> > > > > > > > >> > > reuse in any case. We definitely
> don't
> >> want
> >> > > > > > someone to
> >> > > > > > > > accidentally
> >> > > > > > > > >> add or
> >> > > > > > > > >> > > remove methods because they think
> this is
> >> > just
> >> > > > > > another
> >> > > > > > > internal
> >> > > > > > > > class
> >> > > > > > > > >> used
> >> > > > > > > > >> > > for requests.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > The BrokerInfo class is a nice
> >> improvement.
> >> > > It
> >> > > > > > looks
> >> > > > > > > like it
> >> > > > > > > > will be
> >> > > > > > > > >> > > useful for passing in information
> about
> >> the
> >> > > > > context
> >> > > > > > we're
> >> > > > > > > > running
> >> > > > > > > > >> in. It
> >> > > > > > > > >> > > would be nice to call this class
> >> ServerInfo
> >> > > > rather
> >> > > > > > than
> >> > > > > > > > BrokerInfo,
> >> > > > > > > > >> since
> >> > > > > > > > >> > > we will want to run the authorizer
> on
> >> > > > controllers
> >> > > > > > as well
> >> > > > > > > as on
> >> > > > > > > > >> brokers,
> >> > > > > > > > >> > > and the controller may run as a
> separate
> >> > > process
> >> > > > > > post
> >> > > > > > > KIP-500.
> >> > > > > > > > I also
> >> > > > > > > > >> > > think that this class should be in
> the
> >> > > > > > > > >> org.apache.kafka.server.authorizer
> >> > > > > > > > >> > > namespace. Again, it is an
> interface,
> >> not a
> >> > > > > > concrete
> >> > > > > > > > implementation,
> >> > > > > > > > >> and
> >> > > > > > > > >> > > it's an interface that is very
> >> specifically
> >> > > for
> >> > > > > the
> >> > > > > > > authorizer.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > I agree that we probably don't have
> >> enough
> >> > > > > > information
> >> > > > > > > > preserved for
> >> > > > > > > > >> > > requests currently to always know
> what
> >> > entity
> >> > > > made
> >> > > > > > them.
> >> > > > > > > So we
> >> > > > > > > > can
> >> > > > > > > > >> leave
> >> > > > > > > > >> > > that out for now (except in the
> special
> >> case
> >> > > of
> >> > > > > > Fetch).
> >> > > > > > > > Perhaps we
> >> > > > > > > > >> can add
> >> > > > > > > > >> > > this later if it's needed.
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > I understand the intention behind
> >> > > > > AuthorizationMode
> >> > > > > > > (which is
> >> > > > > > > > now
> >> > > > > > > > >> called
> >> > > > > > > > >> > > AuditFlag in the latest
> revision). But
> >> it
> >> > > still
> >> > > > > > feels
> >> > > > > > > > complex. What
> >> > > > > > > > >> if we
> >> > > > > > > > >> > > just had two booleans in Action:
> >> > logSuccesses
> >> > > > and
> >> > > > > > > logFailures?
> >> > > > > > > > That
> >> > > > > > > > >> seems
> >> > > > > > > > >> > > to cover all the cases here.
> >> > > > MANDATORY_AUTHORIZE
> >> > > > > =
> >> > > > > > true,
> >> > > > > > > true.
> >> > > > > > > > >> > > OPTIONAL_AUTHORIZE = true, false.
> >> FILTER =
> >> > > > true,
> >> > > > > > false.
> >> > > > > > > > >> LIST_AUTHORIZED =
> >> > > > > > > > >> > > false, false. Would there be
> anything
> >> lost
> >> > > > versus
> >> > > > > > having
> >> > > > > > > the
> >> > > > > > > > enum?
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > best,
> >> > > > > > > > >> > > Colin
> >> > > > > > > > >> > >
> >> > > > > > > > >> > >
> >> > > > > > > > >> > > On Wed, Aug 14, 2019, at 06:29,
> Mickael
> >> > Maison
> >> > > > > > wrote:
> >> > > > > > > > >> > > > Hi Rajini,
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > Thanks for the KIP!
> >> > > > > > > > >> > > > I really like that authorize()
> will be
> >> > able
> >> > > to
> >> > > > > > take a
> >> > > > > > > batch of
> >> > > > > > > > >> > > > requests, this will speed up many
> >> > > > > implementations!
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM
> Rajini
> >> > > > Sivaram <
> >> > > > > > > > >> rajinisivaram@gmail.com>
> >> > > > > > > > >> > > wrote:
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > Thanks David! I have fixed the
> typo.
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > Also made a couple of changes
> to make
> >> > the
> >> > > > > > context
> >> > > > > > > > interfaces more
> >> > > > > > > > >> > > generic.
> >> > > > > > > > >> > > > > KafkaRequestContext now
> returns the
> >> > 16-bit
> >> > > > API
> >> > > > > > key as
> >> > > > > > > Colin
> >> > > > > > > > >> suggested
> >> > > > > > > > >> > > as
> >> > > > > > > > >> > > > > well as the friendly name used
> in
> >> > metrics
> >> > > > > which
> >> > > > > > are
> >> > > > > > > useful
> >> > > > > > > > in
> >> > > > > > > > >> audit
> >> > > > > > > > >> > > logs.
> >> > > > > > > > >> > > > > `Authorizer#start` is now
> provided a
> >> > > generic
> >> > > > > > > `BrokerInfo`
> >> > > > > > > > >> interface
> >> > > > > > > > >> > > that
> >> > > > > > > > >> > > > > gives cluster id, broker id and
> >> endpoint
> >> > > > > > information.
> >> > > > > > > The
> >> > > > > > > > generic
> >> > > > > > > > >> > > interface
> >> > > > > > > > >> > > > > can potentially be used in
> other
> >> broker
> >> > > > > plugins
> >> > > > > > in
> >> > > > > > > future
> >> > > > > > > > and
> >> > > > > > > > >> provides
> >> > > > > > > > >> > > > > dynamically generated configs
> like
> >> > broker
> >> > > id
> >> > > > > > and ports
> >> > > > > > > > which are
> >> > > > > > > > >> > > currently
> >> > > > > > > > >> > > > > not available to plugins
> unless these
> >> > > > configs
> >> > > > > > are
> >> > > > > > > statically
> >> > > > > > > > >> > > configured.
> >> > > > > > > > >> > > > > Please let me know if there
> are any
> >> > > > concerns.
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > Regards,
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > Rajini
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > On Tue, Aug 13, 2019 at 4:30
> PM David
> >> > > Jacot
> >> > > > <
> >> > > > > > > > djacot@confluent.io>
> >> > > > > > > > >> > > wrote:
> >> > > > > > > > >> > > > >
> >> > > > > > > > >> > > > > > Hi Rajini,
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > Thank you for the update! It
> looks
> >> > good
> >> > > to
> >> > > > > me.
> >> > > > > > > There is a
> >> > > > > > > > typo
> >> > > > > > > > >> in the
> >> > > > > > > > >> > > > > > `AuditFlag` enum:
> >> > `MANDATORY_AUTHOEIZE`
> >> > > ->
> >> > > > > > > > >> `MANDATORY_AUTHORIZE`.
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > Regards,
> >> > > > > > > > >> > > > > > David
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > On Mon, Aug 12, 2019 at 2:54
> PM
> >> Rajini
> >> > > > > > Sivaram <
> >> > > > > > > > >> > > rajinisivaram@gmail.com>
> >> > > > > > > > >> > > > > > wrote:
> >> > > > > > > > >> > > > > >
> >> > > > > > > > >> > > > > > > Hi David,
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > Thanks for reviewing the
> KIP!
> >> Since
> >> > > > > > questions
> >> > > > > > > about
> >> > > > > > > > >> `authorization
> >> > > > > > > > >> > > mode`
> >> > > > > > > > >> > > > > > > and `count` have come up
> multiple
> >> > > > times, I
> >> > > > > > have
> >> > > > > > > renamed
> >> > > > > > > > both.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > 1) Renamed `count` to
> >> > > > > > `resourceReferenceCount`.
> >> > > > > > > It is
> >> > > > > > > > the
> >> > > > > > > > >> number
> >> > > > > > > > >> > > of times
> >> > > > > > > > >> > > > > > > the resource being
> authorized is
> >> > > > > referenced
> >> > > > > > > within the
> >> > > > > > > > >> request.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > 2) Renamed
> `AuthorizationMode` to
> >> > > > > > `AuditFlag`. It
> >> > > > > > > is
> >> > > > > > > > provided
> >> > > > > > > > >> to
> >> > > > > > > > >> > > improve
> >> > > > > > > > >> > > > > > > audit logging in the
> authorizer.
> >> The
> >> > > > enum
> >> > > > > > values
> >> > > > > > > have
> >> > > > > > > > javadoc
> >> > > > > > > > >> which
> >> > > > > > > > >> > > > > > > indicate how the
> authorization
> >> > result
> >> > > is
> >> > > > > > used in
> >> > > > > > > each
> >> > > > > > > > of the
> >> > > > > > > > >> modes
> >> > > > > > > > >> > > to
> >> > > > > > > > >> > > > > > > enable authorizers to log
> audit
> >> > > messages
> >> > > > > at
> >> > > > > > the
> >> > > > > > > right
> >> > > > > > > > severity
> >> > > > > > > > >> > > level.
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > Regards,
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > Rajini
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > On Mon, Aug 12, 2019 at
> 5:54 PM
> >> > David
> >> > > > > Jacot
> >> > > > > > <
> >> > > > > > > > >> djacot@confluent.io>
> >> > > > > > > > >> > > wrote:
> >> > > > > > > > >> > > > > > >
> >> > > > > > > > >> > > > > > > > Hi Rajini,
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > Thank you for the KIP.
> >> Overall, it
> >> > > > looks
> >> > > > > > good
> >> > > > > > > to me.
> >> > > > > > > > I have
> >> > > > > > > > >> few
> >> > > > > > > > >> > > > > > > > questions/suggestions:
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > 1. It is hard to grasp
> what
> >> > > > > > `Action#count` is
> >> > > > > > > for. I
> >> > > > > > > > guess I
> >> > > > > > > > >> > > understand
> >> > > > > > > > >> > > > > > > > where you want to go
> with it
> >> but
> >> > it
> >> > > > took
> >> > > > > > me a
> >> > > > > > > while to
> >> > > > > > > > >> figure it
> >> > > > > > > > >> > > out.
> >> > > > > > > > >> > > > > > > > Perhaps, we could come
> up with
> >> a
> >> > > > better
> >> > > > > > name
> >> > > > > > > than
> >> > > > > > > > `count`?
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > 2. I had a hard time
> trying to
> >> > > > > understand
> >> > > > > > the
> >> > > > > > > > >> `AuthorizationMode`
> >> > > > > > > > >> > > > > > > concept,
> >> > > > > > > > >> > > > > > > > especially wrt. the
> OPTIONAL
> >> one.
> >> > My
> >> > > > > > > understanding is
> >> > > > > > > > that
> >> > > > > > > > >> an
> >> > > > > > > > >> > > ACL is
> >> > > > > > > > >> > > > > > > either
> >> > > > > > > > >> > > > > > > > defined or not. Could you
> >> > elaborate
> >> > > a
> >> > > > > bit
> >> > > > > > more
> >> > > > > > > on
> >> > > > > > > > that?
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > Thanks,
> >> > > > > > > > >> > > > > > > > David
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > On Fri, Aug 9, 2019 at
> 10:26 PM
> >> > Don
> >> > > > > Bosco
> >> > > > > > Durai
> >> > > > > > > <
> >> > > > > > > > >> > > bosco@apache.org>
> >> > > > > > > > >> > > > > > > wrote:
> >> > > > > > > > >> > > > > > > >
> >> > > > > > > > >> > > > > > > > > Hi Rajini
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > 3.2 - This makes sense.
> >> Thanks
> >> > for
> >> > > > > > clarifying.
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > Rest looks fine. Once
> the
> >> > > > > > implementations are
> >> > > > > > > done,
> >> > > > > > > > it
> >> > > > > > > > >> will be
> >> > > > > > > > >> > > more
> >> > > > > > > > >> > > > > > > clear
> >> > > > > > > > >> > > > > > > > > on the different values
> >> > > RequestType
> >> > > > > and
> >> > > > > > Mode.
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > Thanks
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > Bosco
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > On 8/9/19, 5:08 AM,
> "Rajini
> >> > > > Sivaram"
> >> > > > > <
> >> > > > > > > > >> rajinisivaram@gmail.com
> >> > > > > > > > >> > > >
> >> > > > > > > > >> > > > > > wrote:
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > Hi Don,
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > Thanks for the
> >> suggestions.
> >> > A
> >> > > > few
> >> > > > > > > responses
> >> > > > > > > > below:
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > 3.1 Can rename and
> >> improve
> >> > > docs
> >> > > > if
> >> > > > > > we keep
> >> > > > > > > > this. Let's
> >> > > > > > > > >> > > finish the
> >> > > > > > > > >> > > > > > > > > discussion on
> Colin's
> >> > > > suggestions
> >> > > > > > > regarding this
> >> > > > > > > > >> first.
> >> > > > > > > > >> > > > > > > > > 3.2 No, I was
> thinking of
> >> > some
> >> > > > > > requests
> >> > > > > > > that
> >> > > > > > > > have an
> >> > > > > > > > >> old
> >> > > > > > > > >> > > way of
> >> > > > > > > > >> > > > > > > > > authorizing
> >> > > > > > > > >> > > > > > > > > and a new way
> where we
> >> have
> >> > > > > > retained the
> >> > > > > > > old
> >> > > > > > > > way for
> >> > > > > > > > >> > > backward
> >> > > > > > > > >> > > > > > > > > compatibility. One
> >> example
> >> > is
> >> > > > > > > Cluster:Create
> >> > > > > > > > >> permission to
> >> > > > > > > > >> > > create
> >> > > > > > > > >> > > > > > > > > topics.
> >> > > > > > > > >> > > > > > > > > We have replaced
> this
> >> with
> >> > > > > > fine-grained
> >> > > > > > > topic
> >> > > > > > > > create
> >> > > > > > > > >> > > access using
> >> > > > > > > > >> > > > > > > > > Topic:Create
> >> > > > > > > > >> > > > > > > > > for topic
> patterns. But
> >> we
> >> > > still
> >> > > > > > check if
> >> > > > > > > user
> >> > > > > > > > has
> >> > > > > > > > >> > > Cluster:Create
> >> > > > > > > > >> > > > > > > > > first. If
> >> > > > > > > > >> > > > > > > > > Denied, the deny is
> >> ignored
> >> > > and
> >> > > > we
> >> > > > > > check
> >> > > > > > > > >> Topic:Create. We
> >> > > > > > > > >> > > dont
> >> > > > > > > > >> > > > > > want
> >> > > > > > > > >> > > > > > > > to
> >> > > > > > > > >> > > > > > > > > log
> >> > > > > > > > >> > > > > > > > > DENY for
> Cluster:Create
> >> at
> >> > > INFO
> >> > > > > > level for
> >> > > > > > > this,
> >> > > > > > > > since
> >> > > > > > > > >> this
> >> > > > > > > > >> > > is
> >> > > > > > > > >> > > > > > not a
> >> > > > > > > > >> > > > > > > > > mandatory ACL for
> >> creating
> >> > > > topics.
> >> > > > > > We
> >> > > > > > > will get
> >> > > > > > > > >> appropriate
> >> > > > > > > > >> > > logs
> >> > > > > > > > >> > > > > > > from
> >> > > > > > > > >> > > > > > > > > the
> >> > > > > > > > >> > > > > > > > > subsequent
> Topic:Create
> >> in
> >> > > this
> >> > > > > > case.
> >> > > > > > > > >> > > > > > > > > 3.3 They are not
> quite
> >> the
> >> > > same.
> >> > > > > > FILTER
> >> > > > > > > implies
> >> > > > > > > > that
> >> > > > > > > > >> user
> >> > > > > > > > >> > > > > > actually
> >> > > > > > > > >> > > > > > > > > requested access
> to and
> >> > > > performed
> >> > > > > > some
> >> > > > > > > > operation on
> >> > > > > > > > >> the
> >> > > > > > > > >> > > filtered
> >> > > > > > > > >> > > > > > > > > resources.
> >> > > > > > > > >> > > > > > > > > LIST_AUTHORZED did
> not
> >> > result
> >> > > in
> >> > > > > any
> >> > > > > > > actual
> >> > > > > > > > access.
> >> > > > > > > > >> User
> >> > > > > > > > >> > > simply
> >> > > > > > > > >> > > > > > > > wanted
> >> > > > > > > > >> > > > > > > > > to
> >> > > > > > > > >> > > > > > > > > know what they are
> >> allowed
> >> > to
> >> > > > > > access.
> >> > > > > > > > >> > > > > > > > > 3.4 Request types
> are
> >> > Produce,
> >> > > > > > JoinGroup,
> >> > > > > > > > OffsetCommit
> >> > > > > > > > >> > > etc. So
> >> > > > > > > > >> > > > > > that
> >> > > > > > > > >> > > > > > > > is
> >> > > > > > > > >> > > > > > > > > different from
> >> authorization
> >> > > > mode,
> >> > > > > > > operation
> >> > > > > > > > etc.
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > On Thu, Aug 8,
> 2019 at
> >> 11:36
> >> > > PM
> >> > > > > Don
> >> > > > > > Bosco
> >> > > > > > > Durai
> >> > > > > > > > <
> >> > > > > > > > >> > > > > > bosco@apache.org>
> >> > > > > > > > >> > > > > > > > > wrote:
> >> > > > > > > > >> > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > Hi Rajini
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > Thanks for
> clarifying.
> >> > This
> >> > > is
> >> > > > > > very
> >> > > > > > > helpful...
> >> > > > > > > > >> > > > > > > > > >
> >> > > > > > > > >> > > > > > > > > > #1 - On the
> Ranger
> >> side,
> >> > we
> >> > > > > > should be
> >> > > > > > > able to
> >> > > > > > > > handle
> >> > > > > > > > >> > > multiple
> >> > > > > > > > >> > > > > > > > > requests at
> >> > > > > > > > >> > > > > > > > > > the same time. I
> was
> >> just
> >> > > not
> >> > > > > > sure how
> >> > > > > > > much
> >> > > > > > > > >> processing
> >> > > > > > > > >> > > overhead
> >> > > > > > > > >> > > > > > > > will
> >> > > > > > > > >> > > > > > > > > be
> >> > > > > > > > >> > > > > > > > > > there on the
> Broker
> >> side
> >> > to
> >> > > > > split
> >> > > > > > and
> >> > > > > > > then
> >> > > > > > > > >> consolidate
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > > results.
> >> > > > > > > > >> > > > > > > > > If it
> >> > > > > > > > >> > > > > > > > > > is negligible,
> then
> >> this
> >> > is
> >> > > > the
> >> > > > > > better
> >> > > > > > > way.
> >> > > > > > > > It will
> >> > > > > > > > >> > > make it
> >> > > > > > > > >> > > > > > > future
> >> > > > > > > > >> > > > > > > > > proof.
> >> > > > > > > > >> > > > > > > > > > #2 - I agree,
> having a
> >> > > single
> >> > > > > > "start"
> >> > > > > > > call
> >> > > > > > > > makes it
> >> > > > > > > > >> > > cleaner.
> >> > > > > > > > >> > > > > > The
> >> > > > > > > > >> > > > > > > > > > Authorization
> >> > implementation
> >> > > > > will
> >> > > > > > only
> >> > > > > > > have
> >> > > > > > > > to do
> >> > > > > > > > >> the
> >> > > > > > > > >> > > > > > > > initialization
> >> > > > > > > > >> > > > > > > > > only
> >> > > > > > > > >> > > > > > > > > > once.
> >> > > > > > > > >> > > > > > > > > > #3.1 - Thanks
> for the
> >> > > > > > clarification. I
> >> > > > > > > think
> >> > > > > > > > it
> >> > > > > > > > >> makes
> >> > > > > > > > >> > > sense to
> >> > > > > > > > >> > > > > > > have
> >> > > > > > > > >> > > > > > > > > this.
> >> > > > > > > > >> > > > > > > > > > The term "Mode"
> might
> >> not
> >> > be
> >> > > > > > explicit
> >> > > > > > > enough.
> >> > > > > > > > >> > > Essentially it
> >> > > > > > > > >> > > > > > > seems
> >> > > > > > > > >> > > > > > > > > you want
> >> > > > > > > > >> > > > > > > > > > the Authorizer
> to know
> >> the
> >> > > > > > > Intent/Purpose of
> >> > > > > > > > the
> >> > > > > > > > >> > > authorize call
> >> > > > > > > > >> > > > > > > and
> >> > > > > > > > >> > > > > > > > > let the
> >> > > > > > > > >> > > > > > > > > > Authorizer
> decide what
> >> to
> >> > > log
> >> > > > as
> >> > > > > > Audit
> >> > > > > > > event.
> >> > > > > > > > >> Changing
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > > > > class/field name
> >> > > > > > > > >> > > > > > > > > > or giving more
> >> > documentation
> >> > > > > will
> >> > > > > > do.
> >> > > > > > > > >> > > > > > > > > > #3.2 - Regarding
> the
> >> > option
> >> > > > > > "OPTIONAL".
> >> > > > > > > Are
> >> > > > > > > > you
> >> > > > > > > > >> thinking
> >> > > > > > > > >> > > from
> >> > > > > > > > >> > > > > > > > > chaining
> >> > > > > > > > >> > > > > > > > > > multiple
> Authorizer? If
> >> > > so, I
> >> > > > > am
> >> > > > > > not
> >> > > > > > > sure
> >> > > > > > > > whether
> >> > > > > > > > >> the
> >> > > > > > > > >> > > Broker
> >> > > > > > > > >> > > > > > > would
> >> > > > > > > > >> > > > > > > > > have
> >> > > > > > > > >> > > > > > > > > > enough
> information to
> >> make
> >> > > > that
> >> > > > > > > decision. I
> >> > > > > > > > feel the
> >> > > > > > > > >> > > Authorizer
> >> > > > > > > > >> > > > > > > > will
> >> > > > > > > > >> > > > > > > > > be the
> >> > > > > > > > >> > > > > > > > > > one who would
> have that
> >> > > > > > knowledge. E.g.
> >> > > > > > > in
> >> > > > > > > > Ranger
> >> > > > > > > > >> we have
> >> > > > > > > > >> > > > > > > explicit
> >> > > > > > > > >> > > > > > > > > deny,
> >> > > > > > > > >> > > > > > > > > > which means no
> matter
> >> > what,
> >> > > > the
> >> > > > > > request
> >> > > > > > > > should be
> >> > > > > > > > >> denied
> >> > > > > > > > >> > > for
> >> > > > > > > > >> > > > > > the
> >> > > > > > > > >> > > > > > > > > user/group
> >> > > > > > > > >> > > > > > > > > > or condition. So
> if you
> >> > are
> >> > > > > > thinking of
> >> > > > > > > > chaining
> >> > > > > > > > >> > > Authorizers,
> >> > > > > > > > >> > > > > > > then
> >> > > > > > > > >> > > > > > > > > > responses should
> have
> >> the
> >> > > > third
> >> > > > > > state,
> >> > > > > > > e.g.
> >> > > > > > > > >> > > "DENIED_FINAL", in
> >> > > > > > > > >> > > > > > > > which
> >> > > > > > > > >> > > > > > > > > case
> >> > > > > > > > >> > > > > > > > > > if there is an
> >> > Authorization
> >> > > > > > chain, it
> >> > > > > > > will
> >> > > > > > > > be stop
> >> > > > > > > > >> and
> >> > > > > > > > >> > > the
> >> > > > > > > > >> > > > > > > request
> >> > > > > > > > >> > > > > > > > > will be
> >> > > > > > > > >> > > > > > > > > > denied and if it
> is
> >> just
> >> > > > denied,
> >> > > > > > then
> >> > > > > > > you
> >> > > > > > > > might fall
> >> > > > > > > > >> > > back to
> >> > > > > > > > >> > > > > > next
> >> > > > > > > > >> > > > > > > > > > authorizer. If
> we don't
> >> > have
> >> > > > > > chaining of
> >> > > > > > > > >> Authorizing in
> >> > > > > > > > >> > > mind,
> >> > > > > > > > >> > > > > > > then
> >> > > > > > > > >> > > > > > > > we
> >> > > > > > > > >> > > > > > > > > > should reconsider
> >> OPTIONAL
> >> > > for
> >> > > > > > now. Or
> >> > > > > > > > clarify under
> >> > > > > > > > >> > > which
> >> > > > > > > > >> > > > > > > scenario
> >> > > > > > > > >> > > > > > > > > > OPTIONAL will be
> >> called.
> >> > > > > > > > >> > > > > > > > > > #3.3 Regarding,
> FILTER
> >> v/s
> >> > > > > > > LIST_AUTHORIZED,
> >> > > > > > > > isn't
> >> > > > > > > > >> > > > > > > LIST_AUTHORIZED a
> >> > > > > > > > >> > > > > > > > > > special case for
> >> "FILTER
> >
> >
>
>
>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Don Bosco Durai <bo...@apache.org>.
+1
I might be wrong here, but here are few of my assumptions..
1. Unlike transactional services, in Kafka most of the users are application users, so once the connection is established, then they are going to be long running sessions. So the caching is more predictable and heavy lifting can be done during initial session setup and cache can be frequently updated using background thread. KSQL might have a different flow, but I don't have enough datapoints for it.
2. For publish/consume authorization calls, providing async option will be misleading for Kafka like load
3. For admin calls, hopefully no one does hundreds of call in multiple thread within few seconds. This would be a bad design.
So I feel, we should be okay with sync calls for now. And consider async in the future if needed. At least in Apache Ranger implementation, we would be okay using sync operation. I am not sure if any other plugin implementation would benefit from async implementation.
Thanks
Bosco
On 9/6/19, 8:45 AM, "Ismael Juma" <is...@juma.me.uk> wrote:
One more thing: if people have strong opinions that authorize has to be
sync, we could leave it like that for now. If needed, we can later add an
authorizeAsync with another method returning a Boolean indicating that it's
supported. That is, there is a path to evolve the interface (even if a bit
ugly).
What about the other methods, is there consensus that they should be
updated to return CompletionStage?
Ismael
On Fri, Sep 6, 2019, 8:32 AM Ismael Juma <is...@juma.me.uk> wrote:
> I'm on the move, but what Rajini said seems reasonable. I don't think
> using SSDs solves the issue. They can still hang for seconds when they
> fail. Also, many people may not have local SSDs available (remote SSDs like
> EBS hang for tens of seconds when there are issues).
>
> We are currently vulnerable to all of these in the normal read/write path,
> but it seems suboptimal to continue assuming non blocking behavior for
> things that actually do block.
>
> Ismael
>
> On Fri, Sep 6, 2019, 8:24 AM Rajini Sivaram <ra...@gmail.com>
> wrote:
>
>> Hi Jun,
>>
>> For ACLs, the size is also dependent on the number of users. So in
>> deployments with large number of users where some users are not active, an
>> LRU cache may be useful. Agree that there may be other solutions that help
>> avoid the requirement for an async API even for this case.
>>
>> I wasn't expecting our threading model to change very much. Having said
>> that, I haven't figured out how the purgatory can be adapted to work
>> efficiently for this. I was thinking:
>> 1) If authorization future is complete when returning from authorize()
>> call, we invoke the method immediately on the request thread
>> 2) If future is not complete, the request goes into a purgatory
>> 3) When future completes, we mark the request ready to be run. This can be
>> done on ForkJoinPool.commonPool or an executor. But we don't handle the
>> request on that thread
>> 4) We handle the request on the request thread after 3).
>>
>> I totally agree that this adds complexity to the code. Will wait for
>> Ismael
>> to comment on whether he had a different model in mind.
>>
>> Thanks,
>>
>> Rajini
>>
>>
>> On Thu, Sep 5, 2019 at 10:29 PM Jun Rao <ju...@confluent.io> wrote:
>>
>> > Hi, Ismael,
>> >
>> > Yes, I agree that there are 2 separate discussion points, one on the API
>> > and the other on the implementation.
>> >
>> > First on the API, in general, my feeling is that the authorize() call is
>> > expected to be fast to reduce request latency. Adding an async API
>> could be
>> > mis-leading since it might give users the impression that it would be
>> > arbitrarily long. In the short term, currently, we cache all partition
>> > level metadata on every broker. The authorization metadata is
>> proportional
>> > to the number of topics, which typically is less in size. So it we can
>> > cache all partition level metadata, we should be able to cache all
>> > authorization metadata. Longer term, if we want to support more metadata
>> > beyond memory, another option is to put all metadata in an SSD backed
>> > key/value store. The lookup time could still be under 1ms, which we
>> could
>> > potentially absorb in the request threads. So, I am not sure async
>> > authorize() api is strictly needed even in the long term.
>> >
>> > Second on the implementation. I was more concerned about the impact on
>> our
>> > threading model. If we make authorize() async, some threads have to
>> > complete the future when it's ready. It seems to me that those threads
>> have
>> > to be inside the Authorizer implementation since only it knows when to
>> > complete a future? If so, it feels weird that all the logic of request
>> > handling post authorize() is now handled by external threads inside a
>> > plugin. It's not clear how we can configure and monitor them, and how
>> > throttling would work. Perhaps we can provide a bit more detail on the
>> new
>> > threading model.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Wed, Sep 4, 2019 at 9:56 PM Ismael Juma <is...@juma.me.uk> wrote:
>> >
>> > > Hi Jun,
>> > >
>> > > I think it's important to discuss the pluggable API versus the calling
>> > > implementation as two separate points.
>> > >
>> > > From an API perspective, are you suggesting that we should tell users
>> > that
>> > > they cannot block in authorize? Or are you saying that it's OK to
>> block
>> > on
>> > > authorize on occasion and the recommendation would be for people to
>> > > increase the number of threads in the request thread pool?
>> > Architecturally,
>> > > this feels wrong in my opinion.
>> > >
>> > > From the calling implementation perspective, you don't need a
>> callback.
>> > You
>> > > can basically say:
>> > >
>> > > authorize(...).map { result =>
>> > > ...
>> > > }
>> > >
>> > > And then have a common method take that Future and handle it
>> > synchronously
>> > > if it's already complete or submit it to the purgatory if not. This is
>> > > similar to how many modern async web libraries work.
>> > >
>> > > As Rajini said, this could be done later. The initial implementation
>> > could
>> > > simply do `toCompletableFuture().get()` to do it synchronously. But it
>> > > would mean that we could add this functionality without changing the
>> > > pluggable interface.
>> > >
>> > > Ismael
>> > >
>> > > On Wed, Sep 4, 2019 at 5:29 AM Jun Rao <ju...@confluent.io> wrote:
>> > >
>> > > > Hi, Rajini,
>> > > >
>> > > > Thanks for the KIP. I was concerned about #4 too. If we change the
>> > > handling
>> > > > of all requests to use an async authorize() api, will that cause the
>> > code
>> > > > much harder to understand? There are quite a few callbacks already.
>> I
>> > am
>> > > > not sure that we want to introduce more of those. The benefit from
>> > async
>> > > > authorize() api seems limited.
>> > > >
>> > > > Jun
>> > > >
>> > > > On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <
>> rajinisivaram@gmail.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > Hi Don,
>> > > > >
>> > > > > Thanks for your note.
>> > > > >
>> > > > > 1) The intention is to avoid blocking in the calling thread. We
>> > already
>> > > > > have several requests that are put into a purgatory when waiting
>> for
>> > > > remote
>> > > > > communication, for example produce request waiting for
>> replication.
>> > > Since
>> > > > > we have a limited number of request threads in the broker, we
>> want to
>> > > > make
>> > > > > progress with other requests, while one is waiting on any form of
>> > > remote
>> > > > > communication.
>> > > > >
>> > > > > 2) Async management calls will be useful with the default
>> authorizer
>> > > when
>> > > > > KIP-500 removes ZK and we rely on Kafka instead. Our current
>> ZK-based
>> > > > > implementation as well as any custom implementations that don't
>> want
>> > to
>> > > > be
>> > > > > async will just need to return a sync'ed value. So instead of
>> > > returning `
>> > > > > value`, the code would just return
>> > > > > `CompletableFuture.completedFuture(value)
>> > > > > `. So it would be just a single line change in the implementation
>> > with
>> > > > the
>> > > > > new API. The caller would treat completedFuture exactly as it does
>> > > today,
>> > > > > processing the request synchronously without using a purgatory.
>> > > > >
>> > > > > 3) For implementations that return a completedFuture as described
>> in
>> > > 2),
>> > > > > the behaviour would remain exactly the same. No additional
>> threads or
>> > > > > purgatory will be used for this case. So there would be no
>> > performance
>> > > > > penalty. For implementations that return a future that is not
>> > complete,
>> > > > we
>> > > > > prioritise running more requests concurrently. So in a deployment
>> > with
>> > > a
>> > > > > large number of clients, we would improve performance by allowing
>> > other
>> > > > > requests to be processed on the request threads while some are
>> > waiting
>> > > > for
>> > > > > authorization metadata.
>> > > > >
>> > > > > 4) I was concerned about this too. The goal is to make the API
>> > flexible
>> > > > > enough to handle large scale deployments in future when caching
>> all
>> > > > > authorization metadata in each broker is not viable. Using an
>> async
>> > API
>> > > > > that returns CompletionStage, the caller has the option to handle
>> the
>> > > > > result synchronously or asynchronously, so we don't necessarily
>> need
>> > to
>> > > > > update the calling code right away. Custom authorizers using the
>> > async
>> > > > API
>> > > > > have full control over whether authorization is performed in-line
>> > since
>> > > > > completedFuture will always be handled synchronously. We do need
>> to
>> > > > update
>> > > > > KafkaApis to take advantage of the asynchronous API to improve
>> scale.
>> > > > Even
>> > > > > though this is a big change, since we will be doing the same for
>> all
>> > > > > requests, it shouldn't be too hard to maintain since the same
>> pattern
>> > > > will
>> > > > > be used for all requests.
>> > > > >
>> > > > > Regards,
>> > > > >
>> > > > > Rajini
>> > > > >
>> > > > > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <bosco@apache.org
>> >
>> > > > wrote:
>> > > > >
>> > > > > > Hi Rajini
>> > > > > >
>> > > > > > Help me understand this a bit more.
>> > > > > >
>> > > > > > 1. For all practical purpose, without authorization you can't
>> go to
>> > > the
>> > > > > > next step. The calling code needs to block anyway. So should we
>> > just
>> > > > let
>> > > > > > the implementation code do the async part?
>> > > > > > 2. If you feel management calls need to be async, then we should
>> > > > consider
>> > > > > > another set of async APIs. I don't feel we should complicate it
>> > > > further (
>> > > > > > 3. Another concern I have is wrt performance. Kafka has been
>> built
>> > to
>> > > > > > handle 1000s per second requests. Not sure whether making it
>> async
>> > > will
>> > > > > add
>> > > > > > any unnecessary overhead.
>> > > > > > 4. How much complication would this add on the calling side?
>> And is
>> > > it
>> > > > > > worth it?
>> > > > > >
>> > > > > > Thanks
>> > > > > >
>> > > > > > Bosco
>> > > > > >
>> > > > > >
>> > > > > > On 9/3/19, 8:50 AM, "Rajini Sivaram" <ra...@gmail.com>
>> > > wrote:
>> > > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > Ismael brought up a point that it will be good to make the
>> > > > Authorizer
>> > > > > > interface asynchronous to avoid blocking request threads
>> during
>> > > > > remote
>> > > > > > operations.
>> > > > > >
>> > > > > > 1) Since we want to support different backends for
>> > authorization
>> > > > > > metadata,
>> > > > > > making createAcls() and deleteAcls() asynchronous makes
>> sense
>> > > since
>> > > > > > these
>> > > > > > always involve remote operations. When KIP-500 removes
>> > ZooKeeper,
>> > > > we
>> > > > > > would
>> > > > > > want to move ACLs to Kafka and async updates will avoid
>> > > unnecessary
>> > > > > > blocking.
>> > > > > > 2) For authorize() method, we currently use cached ACLs in
>> the
>> > > > > built-in
>> > > > > > authorizers, so synchronous authorise operations work well
>> now.
>> > > But
>> > > > > > async
>> > > > > > authorize() would support this scenario as well as
>> authorizers
>> > in
>> > > > > large
>> > > > > > organisations where an LRU cache would enable a smaller
>> cache
>> > > even
>> > > > > > when the
>> > > > > > backend holds a large amount of ACLs for infrequently used
>> > > > resources
>> > > > > or
>> > > > > > users who don't use the system frequently.
>> > > > > >
>> > > > > > For both cases, the built-in authorizer will continue to be
>> > > > > > synchronous,
>> > > > > > using CompletableFuture.completedFuture() to return the
>> actual
>> > > > > result.
>> > > > > > But
>> > > > > > async API will make custom authorizer implementations more
>> > > > flexible.
>> > > > > I
>> > > > > > would like to know if there are any concerns with these
>> changes
>> > > > > before
>> > > > > > updating the KIP.
>> > > > > >
>> > > > > > *Proposed API:*
>> > > > > > public interface Authorizer extends Configurable, Closeable
>> {
>> > > > > >
>> > > > > > Map<Endpoint, CompletionStage<Void>>
>> > > start(AuthorizerServerInfo
>> > > > > > serverInfo);
>> > > > > > List<CompletionStage<AuthorizationResult>>
>> > > > > > authorize(AuthorizableRequestContext requestContext,
>> > List<Action>
>> > > > > > actions);
>> > > > > > List<CompletionStage<AclCreateResult>>
>> > > > > > createAcls(AuthorizableRequestContext requestContext,
>> > > > > List<AclBinding>
>> > > > > > aclBindings);
>> > > > > > List<CompletionStage<AclDeleteResult>>
>> > > > > > deleteAcls(AuthorizableRequestContext requestContext,
>> > > > > > List<AclBindingFilter> aclBindingFilters);
>> > > > > > CompletionStage<Collection<AclBinding>>
>> > acls(AclBindingFilter
>> > > > > > filter);
>> > > > > > }
>> > > > > >
>> > > > > >
>> > > > > > Thank you,
>> > > > > >
>> > > > > > Rajini
>> > > > > >
>> > > > > > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <
>> > > bosco@apache.org>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Rajini
>> > > > > > >
>> > > > > > > Thanks for clarifying. I am good for now.
>> > > > > > >
>> > > > > > > Regards
>> > > > > > >
>> > > > > > > Bosco
>> > > > > > >
>> > > > > > >
>> > > > > > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <
>> > > rajinisivaram@gmail.com
>> > > > >
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > Hi Don,
>> > > > > > >
>> > > > > > > That should be fine. I guess Ranger loads policies
>> from
>> > the
>> > > > > > database
>> > > > > > > synchronously when the authorizer is configured,
>> similar
>> > to
>> > > > > > > SimpleAclAuthorizer loading from ZooKeeper. Ranger can
>> > > > continue
>> > > > > > to load
>> > > > > > > synchronously from `configure()` or `start()` and
>> return
>> > an
>> > > > > > empty map
>> > > > > > > from
>> > > > > > > `start()`. That would retain the existing behaviour..
>> > When
>> > > > the
>> > > > > > same
>> > > > > > > database stores policies for all listeners and the
>> > policies
>> > > > are
>> > > > > > not
>> > > > > > > stored
>> > > > > > > in Kafka, there is no value in making the load
>> > > asynchronous.
>> > > > > > >
>> > > > > > > Regards,
>> > > > > > >
>> > > > > > > Rajini
>> > > > > > >
>> > > > > > >
>> > > > > > > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai <
>> > > > > > bosco@apache.org>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Rajini
>> > > > > > > >
>> > > > > > > > Assuming this doesn't affect custom plugins, I don't
>> > have
>> > > > any
>> > > > > > > concerns
>> > > > > > > > regarding this.
>> > > > > > > >
>> > > > > > > > I do have one question regarding:
>> > > > > > > >
>> > > > > > > > "For authorizers that don’t store metadata in
>> > ZooKeeper,
>> > > > > > ensure that
>> > > > > > > > authorizer metadata for each listener is available
>> > before
>> > > > > > starting
>> > > > > > > up the
>> > > > > > > > listener. This enables different authorization
>> metadata
>> > > > > stores
>> > > > > > for
>> > > > > > > > different listeners."
>> > > > > > > >
>> > > > > > > > Since Ranger uses its own database for storing
>> > policies,
>> > > do
>> > > > > you
>> > > > > > > anticipate
>> > > > > > > > any issues?
>> > > > > > > >
>> > > > > > > > Thanks
>> > > > > > > >
>> > > > > > > > Bosco
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
>> > > > > > rajinisivaram@gmail.com>
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > Hi all,
>> > > > > > > >
>> > > > > > > > I made another change to the KIP. The KIP was
>> > > > originally
>> > > > > > > proposing to
>> > > > > > > > extend SimpleAclAuthorizer to also implement the
>> > new
>> > > > API
>> > > > > > (in
>> > > > > > > addition
>> > > > > > > > to
>> > > > > > > > the existing API). But since we use the new API
>> > when
>> > > > > > available,
>> > > > > > > this
>> > > > > > > > can
>> > > > > > > > break custom authorizers that extend this class
>> and
>> > > > > > override
>> > > > > > > specific
>> > > > > > > > methods of the old API. To avoid breaking any
>> > > existing
>> > > > > > custom
>> > > > > > > > implementations that extend this class,
>> > particularly
>> > > > > > because it
>> > > > > > > is in
>> > > > > > > > the
>> > > > > > > > public package kafka.security.auth, the KIP now
>> > > > proposes
>> > > > > to
>> > > > > > > retain the
>> > > > > > > > old
>> > > > > > > > authorizer as-is. SimpleAclAuthorizer will
>> > continue
>> > > to
>> > > > > > > implement the
>> > > > > > > > old
>> > > > > > > > API, but will be deprecated. A new authorizer
>> > > > > > implementation
>> > > > > > > > kafka.security.authorizer.AclAuthorizer will be
>> > added
>> > > > for
>> > > > > > the
>> > > > > > > new API
>> > > > > > > > (this
>> > > > > > > > will not be in the public package).
>> > > > > > > >
>> > > > > > > > Please let me know if you have any concerns.
>> > > > > > > >
>> > > > > > > > Regards,
>> > > > > > > >
>> > > > > > > > Rajini
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
>> > > > > > > > rajinisivaram@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Thanks Colin.
>> > > > > > > > >
>> > > > > > > > > If there are no other concerns, I will start
>> vote
>> > > > later
>> > > > > > today.
>> > > > > > > Many
>> > > > > > > > thanks
>> > > > > > > > > to every one for the feedback.
>> > > > > > > > >
>> > > > > > > > > Regards,
>> > > > > > > > >
>> > > > > > > > > Rajini
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe
>> <
>> > > > > > > cmccabe@apache.org>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > >> Thanks, Rajini. It looks good to me.
>> > > > > > > > >>
>> > > > > > > > >> best,
>> > > > > > > > >> Colin
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > > >> On Thu, Aug 15, 2019, at 11:37, Rajini
>> Sivaram
>> > > > wrote:
>> > > > > > > > >> > Hi Colin,
>> > > > > > > > >> >
>> > > > > > > > >> > Thanks for the review. I have updated the
>> KIP
>> > to
>> > > > > move
>> > > > > > the
>> > > > > > > > interfaces for
>> > > > > > > > >> > request context and server info to the
>> > > authorizer
>> > > > > > package.
>> > > > > > > These
>> > > > > > > > are now
>> > > > > > > > >> > called AuthorizableRequestContext and
>> > > > > > AuthorizerServerInfo.
>> > > > > > > > Endpoint is
>> > > > > > > > >> now
>> > > > > > > > >> > a class in org.apache.kafka.common to make
>> it
>> > > > > reusable
>> > > > > > > since we
>> > > > > > > > already
>> > > > > > > > >> > have multiple implementations of it. I have
>> > > > removed
>> > > > > > > requestName
>> > > > > > > > from the
>> > > > > > > > >> > request context interface since authorizers
>> > can
>> > > > > > distinguish
>> > > > > > > > follower
>> > > > > > > > >> fetch
>> > > > > > > > >> > and consumer fetch from the operation being
>> > > > > > authorized. So
>> > > > > > > 16-bit
>> > > > > > > > >> request
>> > > > > > > > >> > type should be sufficient for audit
>> logging.
>> > > Also
>> > > > > > replaced
>> > > > > > > > AuditFlag
>> > > > > > > > >> with
>> > > > > > > > >> > two booleans as you suggested.
>> > > > > > > > >> >
>> > > > > > > > >> > Can you take another look and see if the
>> KIP
>> > is
>> > > > > ready
>> > > > > > for
>> > > > > > > voting?
>> > > > > > > > >> >
>> > > > > > > > >> > Thanks for all your help!
>> > > > > > > > >> >
>> > > > > > > > >> > Regards,
>> > > > > > > > >> >
>> > > > > > > > >> > Rajini
>> > > > > > > > >> >
>> > > > > > > > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin
>> McCabe <
>> > > > > > > cmccabe@apache.org>
>> > > > > > > > >> wrote:
>> > > > > > > > >> >
>> > > > > > > > >> > > Hi Rajini,
>> > > > > > > > >> > >
>> > > > > > > > >> > > I think it would be good to rename
>> > > > > > KafkaRequestContext to
>> > > > > > > > something
>> > > > > > > > >> like
>> > > > > > > > >> > > AuthorizableRequestContext, and put it in
>> > the
>> > > > > > > > >> > > org.apache.kafka.server.authorizer
>> > namespace.
>> > > > If
>> > > > > > we put
>> > > > > > > it in
>> > > > > > > > the
>> > > > > > > > >> > > org.apache.kafka.common namespace, then
>> it's
>> > > not
>> > > > > > really
>> > > > > > > clear
>> > > > > > > > that
>> > > > > > > > >> it's
>> > > > > > > > >> > > part of the Authorizer API. Since this
>> > class
>> > > is
>> > > > > > purely an
>> > > > > > > > interface,
>> > > > > > > > >> with
>> > > > > > > > >> > > no concrete implementation of anything,
>> > > there's
>> > > > > > nothing
>> > > > > > > common
>> > > > > > > > to
>> > > > > > > > >> really
>> > > > > > > > >> > > reuse in any case. We definitely don't
>> want
>> > > > > > someone to
>> > > > > > > > accidentally
>> > > > > > > > >> add or
>> > > > > > > > >> > > remove methods because they think this is
>> > just
>> > > > > > another
>> > > > > > > internal
>> > > > > > > > class
>> > > > > > > > >> used
>> > > > > > > > >> > > for requests.
>> > > > > > > > >> > >
>> > > > > > > > >> > > The BrokerInfo class is a nice
>> improvement.
>> > > It
>> > > > > > looks
>> > > > > > > like it
>> > > > > > > > will be
>> > > > > > > > >> > > useful for passing in information about
>> the
>> > > > > context
>> > > > > > we're
>> > > > > > > > running
>> > > > > > > > >> in. It
>> > > > > > > > >> > > would be nice to call this class
>> ServerInfo
>> > > > rather
>> > > > > > than
>> > > > > > > > BrokerInfo,
>> > > > > > > > >> since
>> > > > > > > > >> > > we will want to run the authorizer on
>> > > > controllers
>> > > > > > as well
>> > > > > > > as on
>> > > > > > > > >> brokers,
>> > > > > > > > >> > > and the controller may run as a separate
>> > > process
>> > > > > > post
>> > > > > > > KIP-500.
>> > > > > > > > I also
>> > > > > > > > >> > > think that this class should be in the
>> > > > > > > > >> org.apache.kafka.server.authorizer
>> > > > > > > > >> > > namespace. Again, it is an interface,
>> not a
>> > > > > > concrete
>> > > > > > > > implementation,
>> > > > > > > > >> and
>> > > > > > > > >> > > it's an interface that is very
>> specifically
>> > > for
>> > > > > the
>> > > > > > > authorizer.
>> > > > > > > > >> > >
>> > > > > > > > >> > > I agree that we probably don't have
>> enough
>> > > > > > information
>> > > > > > > > preserved for
>> > > > > > > > >> > > requests currently to always know what
>> > entity
>> > > > made
>> > > > > > them.
>> > > > > > > So we
>> > > > > > > > can
>> > > > > > > > >> leave
>> > > > > > > > >> > > that out for now (except in the special
>> case
>> > > of
>> > > > > > Fetch).
>> > > > > > > > Perhaps we
>> > > > > > > > >> can add
>> > > > > > > > >> > > this later if it's needed.
>> > > > > > > > >> > >
>> > > > > > > > >> > > I understand the intention behind
>> > > > > AuthorizationMode
>> > > > > > > (which is
>> > > > > > > > now
>> > > > > > > > >> called
>> > > > > > > > >> > > AuditFlag in the latest revision). But
>> it
>> > > still
>> > > > > > feels
>> > > > > > > > complex. What
>> > > > > > > > >> if we
>> > > > > > > > >> > > just had two booleans in Action:
>> > logSuccesses
>> > > > and
>> > > > > > > logFailures?
>> > > > > > > > That
>> > > > > > > > >> seems
>> > > > > > > > >> > > to cover all the cases here.
>> > > > MANDATORY_AUTHORIZE
>> > > > > =
>> > > > > > true,
>> > > > > > > true.
>> > > > > > > > >> > > OPTIONAL_AUTHORIZE = true, false.
>> FILTER =
>> > > > true,
>> > > > > > false.
>> > > > > > > > >> LIST_AUTHORIZED =
>> > > > > > > > >> > > false, false. Would there be anything
>> lost
>> > > > versus
>> > > > > > having
>> > > > > > > the
>> > > > > > > > enum?
>> > > > > > > > >> > >
>> > > > > > > > >> > > best,
>> > > > > > > > >> > > Colin
>> > > > > > > > >> > >
>> > > > > > > > >> > >
>> > > > > > > > >> > > On Wed, Aug 14, 2019, at 06:29, Mickael
>> > Maison
>> > > > > > wrote:
>> > > > > > > > >> > > > Hi Rajini,
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > Thanks for the KIP!
>> > > > > > > > >> > > > I really like that authorize() will be
>> > able
>> > > to
>> > > > > > take a
>> > > > > > > batch of
>> > > > > > > > >> > > > requests, this will speed up many
>> > > > > implementations!
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini
>> > > > Sivaram <
>> > > > > > > > >> rajinisivaram@gmail.com>
>> > > > > > > > >> > > wrote:
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > Thanks David! I have fixed the typo.
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > Also made a couple of changes to make
>> > the
>> > > > > > context
>> > > > > > > > interfaces more
>> > > > > > > > >> > > generic.
>> > > > > > > > >> > > > > KafkaRequestContext now returns the
>> > 16-bit
>> > > > API
>> > > > > > key as
>> > > > > > > Colin
>> > > > > > > > >> suggested
>> > > > > > > > >> > > as
>> > > > > > > > >> > > > > well as the friendly name used in
>> > metrics
>> > > > > which
>> > > > > > are
>> > > > > > > useful
>> > > > > > > > in
>> > > > > > > > >> audit
>> > > > > > > > >> > > logs.
>> > > > > > > > >> > > > > `Authorizer#start` is now provided a
>> > > generic
>> > > > > > > `BrokerInfo`
>> > > > > > > > >> interface
>> > > > > > > > >> > > that
>> > > > > > > > >> > > > > gives cluster id, broker id and
>> endpoint
>> > > > > > information.
>> > > > > > > The
>> > > > > > > > generic
>> > > > > > > > >> > > interface
>> > > > > > > > >> > > > > can potentially be used in other
>> broker
>> > > > > plugins
>> > > > > > in
>> > > > > > > future
>> > > > > > > > and
>> > > > > > > > >> provides
>> > > > > > > > >> > > > > dynamically generated configs like
>> > broker
>> > > id
>> > > > > > and ports
>> > > > > > > > which are
>> > > > > > > > >> > > currently
>> > > > > > > > >> > > > > not available to plugins unless these
>> > > > configs
>> > > > > > are
>> > > > > > > statically
>> > > > > > > > >> > > configured.
>> > > > > > > > >> > > > > Please let me know if there are any
>> > > > concerns.
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > Regards,
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > Rajini
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > On Tue, Aug 13, 2019 at 4:30 PM David
>> > > Jacot
>> > > > <
>> > > > > > > > djacot@confluent.io>
>> > > > > > > > >> > > wrote:
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > > Hi Rajini,
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > Thank you for the update! It looks
>> > good
>> > > to
>> > > > > me.
>> > > > > > > There is a
>> > > > > > > > typo
>> > > > > > > > >> in the
>> > > > > > > > >> > > > > > `AuditFlag` enum:
>> > `MANDATORY_AUTHOEIZE`
>> > > ->
>> > > > > > > > >> `MANDATORY_AUTHORIZE`.
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > Regards,
>> > > > > > > > >> > > > > > David
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > On Mon, Aug 12, 2019 at 2:54 PM
>> Rajini
>> > > > > > Sivaram <
>> > > > > > > > >> > > rajinisivaram@gmail.com>
>> > > > > > > > >> > > > > > wrote:
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > > Hi David,
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Thanks for reviewing the KIP!
>> Since
>> > > > > > questions
>> > > > > > > about
>> > > > > > > > >> `authorization
>> > > > > > > > >> > > mode`
>> > > > > > > > >> > > > > > > and `count` have come up multiple
>> > > > times, I
>> > > > > > have
>> > > > > > > renamed
>> > > > > > > > both.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > 1) Renamed `count` to
>> > > > > > `resourceReferenceCount`.
>> > > > > > > It is
>> > > > > > > > the
>> > > > > > > > >> number
>> > > > > > > > >> > > of times
>> > > > > > > > >> > > > > > > the resource being authorized is
>> > > > > referenced
>> > > > > > > within the
>> > > > > > > > >> request.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > 2) Renamed `AuthorizationMode` to
>> > > > > > `AuditFlag`. It
>> > > > > > > is
>> > > > > > > > provided
>> > > > > > > > >> to
>> > > > > > > > >> > > improve
>> > > > > > > > >> > > > > > > audit logging in the authorizer.
>> The
>> > > > enum
>> > > > > > values
>> > > > > > > have
>> > > > > > > > javadoc
>> > > > > > > > >> which
>> > > > > > > > >> > > > > > > indicate how the authorization
>> > result
>> > > is
>> > > > > > used in
>> > > > > > > each
>> > > > > > > > of the
>> > > > > > > > >> modes
>> > > > > > > > >> > > to
>> > > > > > > > >> > > > > > > enable authorizers to log audit
>> > > messages
>> > > > > at
>> > > > > > the
>> > > > > > > right
>> > > > > > > > severity
>> > > > > > > > >> > > level.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Regards,
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Rajini
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM
>> > David
>> > > > > Jacot
>> > > > > > <
>> > > > > > > > >> djacot@confluent.io>
>> > > > > > > > >> > > wrote:
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > > Hi Rajini,
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > Thank you for the KIP.
>> Overall, it
>> > > > looks
>> > > > > > good
>> > > > > > > to me.
>> > > > > > > > I have
>> > > > > > > > >> few
>> > > > > > > > >> > > > > > > > questions/suggestions:
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > 1. It is hard to grasp what
>> > > > > > `Action#count` is
>> > > > > > > for. I
>> > > > > > > > guess I
>> > > > > > > > >> > > understand
>> > > > > > > > >> > > > > > > > where you want to go with it
>> but
>> > it
>> > > > took
>> > > > > > me a
>> > > > > > > while to
>> > > > > > > > >> figure it
>> > > > > > > > >> > > out.
>> > > > > > > > >> > > > > > > > Perhaps, we could come up with
>> a
>> > > > better
>> > > > > > name
>> > > > > > > than
>> > > > > > > > `count`?
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > 2. I had a hard time trying to
>> > > > > understand
>> > > > > > the
>> > > > > > > > >> `AuthorizationMode`
>> > > > > > > > >> > > > > > > concept,
>> > > > > > > > >> > > > > > > > especially wrt. the OPTIONAL
>> one.
>> > My
>> > > > > > > understanding is
>> > > > > > > > that
>> > > > > > > > >> an
>> > > > > > > > >> > > ACL is
>> > > > > > > > >> > > > > > > either
>> > > > > > > > >> > > > > > > > defined or not. Could you
>> > elaborate
>> > > a
>> > > > > bit
>> > > > > > more
>> > > > > > > on
>> > > > > > > > that?
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > Thanks,
>> > > > > > > > >> > > > > > > > David
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM
>> > Don
>> > > > > Bosco
>> > > > > > Durai
>> > > > > > > <
>> > > > > > > > >> > > bosco@apache.org>
>> > > > > > > > >> > > > > > > wrote:
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > > Hi Rajini
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > 3.2 - This makes sense.
>> Thanks
>> > for
>> > > > > > clarifying.
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > Rest looks fine. Once the
>> > > > > > implementations are
>> > > > > > > done,
>> > > > > > > > it
>> > > > > > > > >> will be
>> > > > > > > > >> > > more
>> > > > > > > > >> > > > > > > clear
>> > > > > > > > >> > > > > > > > > on the different values
>> > > RequestType
>> > > > > and
>> > > > > > Mode.
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > Thanks
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > Bosco
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini
>> > > > Sivaram"
>> > > > > <
>> > > > > > > > >> rajinisivaram@gmail.com
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > > > wrote:
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > Hi Don,
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > Thanks for the
>> suggestions.
>> > A
>> > > > few
>> > > > > > > responses
>> > > > > > > > below:
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > 3.1 Can rename and
>> improve
>> > > docs
>> > > > if
>> > > > > > we keep
>> > > > > > > > this. Let's
>> > > > > > > > >> > > finish the
>> > > > > > > > >> > > > > > > > > discussion on Colin's
>> > > > suggestions
>> > > > > > > regarding this
>> > > > > > > > >> first.
>> > > > > > > > >> > > > > > > > > 3.2 No, I was thinking of
>> > some
>> > > > > > requests
>> > > > > > > that
>> > > > > > > > have an
>> > > > > > > > >> old
>> > > > > > > > >> > > way of
>> > > > > > > > >> > > > > > > > > authorizing
>> > > > > > > > >> > > > > > > > > and a new way where we
>> have
>> > > > > > retained the
>> > > > > > > old
>> > > > > > > > way for
>> > > > > > > > >> > > backward
>> > > > > > > > >> > > > > > > > > compatibility. One
>> example
>> > is
>> > > > > > > Cluster:Create
>> > > > > > > > >> permission to
>> > > > > > > > >> > > create
>> > > > > > > > >> > > > > > > > > topics.
>> > > > > > > > >> > > > > > > > > We have replaced this
>> with
>> > > > > > fine-grained
>> > > > > > > topic
>> > > > > > > > create
>> > > > > > > > >> > > access using
>> > > > > > > > >> > > > > > > > > Topic:Create
>> > > > > > > > >> > > > > > > > > for topic patterns. But
>> we
>> > > still
>> > > > > > check if
>> > > > > > > user
>> > > > > > > > has
>> > > > > > > > >> > > Cluster:Create
>> > > > > > > > >> > > > > > > > > first. If
>> > > > > > > > >> > > > > > > > > Denied, the deny is
>> ignored
>> > > and
>> > > > we
>> > > > > > check
>> > > > > > > > >> Topic:Create. We
>> > > > > > > > >> > > dont
>> > > > > > > > >> > > > > > want
>> > > > > > > > >> > > > > > > > to
>> > > > > > > > >> > > > > > > > > log
>> > > > > > > > >> > > > > > > > > DENY for Cluster:Create
>> at
>> > > INFO
>> > > > > > level for
>> > > > > > > this,
>> > > > > > > > since
>> > > > > > > > >> this
>> > > > > > > > >> > > is
>> > > > > > > > >> > > > > > not a
>> > > > > > > > >> > > > > > > > > mandatory ACL for
>> creating
>> > > > topics.
>> > > > > > We
>> > > > > > > will get
>> > > > > > > > >> appropriate
>> > > > > > > > >> > > logs
>> > > > > > > > >> > > > > > > from
>> > > > > > > > >> > > > > > > > > the
>> > > > > > > > >> > > > > > > > > subsequent Topic:Create
>> in
>> > > this
>> > > > > > case.
>> > > > > > > > >> > > > > > > > > 3.3 They are not quite
>> the
>> > > same.
>> > > > > > FILTER
>> > > > > > > implies
>> > > > > > > > that
>> > > > > > > > >> user
>> > > > > > > > >> > > > > > actually
>> > > > > > > > >> > > > > > > > > requested access to and
>> > > > performed
>> > > > > > some
>> > > > > > > > operation on
>> > > > > > > > >> the
>> > > > > > > > >> > > filtered
>> > > > > > > > >> > > > > > > > > resources.
>> > > > > > > > >> > > > > > > > > LIST_AUTHORZED did not
>> > result
>> > > in
>> > > > > any
>> > > > > > > actual
>> > > > > > > > access.
>> > > > > > > > >> User
>> > > > > > > > >> > > simply
>> > > > > > > > >> > > > > > > > wanted
>> > > > > > > > >> > > > > > > > > to
>> > > > > > > > >> > > > > > > > > know what they are
>> allowed
>> > to
>> > > > > > access.
>> > > > > > > > >> > > > > > > > > 3.4 Request types are
>> > Produce,
>> > > > > > JoinGroup,
>> > > > > > > > OffsetCommit
>> > > > > > > > >> > > etc. So
>> > > > > > > > >> > > > > > that
>> > > > > > > > >> > > > > > > > is
>> > > > > > > > >> > > > > > > > > different from
>> authorization
>> > > > mode,
>> > > > > > > operation
>> > > > > > > > etc.
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > On Thu, Aug 8, 2019 at
>> 11:36
>> > > PM
>> > > > > Don
>> > > > > > Bosco
>> > > > > > > Durai
>> > > > > > > > <
>> > > > > > > > >> > > > > > bosco@apache.org>
>> > > > > > > > >> > > > > > > > > wrote:
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > > Hi Rajini
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > Thanks for clarifying.
>> > This
>> > > is
>> > > > > > very
>> > > > > > > helpful...
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > #1 - On the Ranger
>> side,
>> > we
>> > > > > > should be
>> > > > > > > able to
>> > > > > > > > handle
>> > > > > > > > >> > > multiple
>> > > > > > > > >> > > > > > > > > requests at
>> > > > > > > > >> > > > > > > > > > the same time. I was
>> just
>> > > not
>> > > > > > sure how
>> > > > > > > much
>> > > > > > > > >> processing
>> > > > > > > > >> > > overhead
>> > > > > > > > >> > > > > > > > will
>> > > > > > > > >> > > > > > > > > be
>> > > > > > > > >> > > > > > > > > > there on the Broker
>> side
>> > to
>> > > > > split
>> > > > > > and
>> > > > > > > then
>> > > > > > > > >> consolidate
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > > results.
>> > > > > > > > >> > > > > > > > > If it
>> > > > > > > > >> > > > > > > > > > is negligible, then
>> this
>> > is
>> > > > the
>> > > > > > better
>> > > > > > > way.
>> > > > > > > > It will
>> > > > > > > > >> > > make it
>> > > > > > > > >> > > > > > > future
>> > > > > > > > >> > > > > > > > > proof.
>> > > > > > > > >> > > > > > > > > > #2 - I agree, having a
>> > > single
>> > > > > > "start"
>> > > > > > > call
>> > > > > > > > makes it
>> > > > > > > > >> > > cleaner.
>> > > > > > > > >> > > > > > The
>> > > > > > > > >> > > > > > > > > > Authorization
>> > implementation
>> > > > > will
>> > > > > > only
>> > > > > > > have
>> > > > > > > > to do
>> > > > > > > > >> the
>> > > > > > > > >> > > > > > > > initialization
>> > > > > > > > >> > > > > > > > > only
>> > > > > > > > >> > > > > > > > > > once.
>> > > > > > > > >> > > > > > > > > > #3.1 - Thanks for the
>> > > > > > clarification. I
>> > > > > > > think
>> > > > > > > > it
>> > > > > > > > >> makes
>> > > > > > > > >> > > sense to
>> > > > > > > > >> > > > > > > have
>> > > > > > > > >> > > > > > > > > this.
>> > > > > > > > >> > > > > > > > > > The term "Mode" might
>> not
>> > be
>> > > > > > explicit
>> > > > > > > enough.
>> > > > > > > > >> > > Essentially it
>> > > > > > > > >> > > > > > > seems
>> > > > > > > > >> > > > > > > > > you want
>> > > > > > > > >> > > > > > > > > > the Authorizer to know
>> the
>> > > > > > > Intent/Purpose of
>> > > > > > > > the
>> > > > > > > > >> > > authorize call
>> > > > > > > > >> > > > > > > and
>> > > > > > > > >> > > > > > > > > let the
>> > > > > > > > >> > > > > > > > > > Authorizer decide what
>> to
>> > > log
>> > > > as
>> > > > > > Audit
>> > > > > > > event.
>> > > > > > > > >> Changing
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > > > > class/field name
>> > > > > > > > >> > > > > > > > > > or giving more
>> > documentation
>> > > > > will
>> > > > > > do.
>> > > > > > > > >> > > > > > > > > > #3.2 - Regarding the
>> > option
>> > > > > > "OPTIONAL".
>> > > > > > > Are
>> > > > > > > > you
>> > > > > > > > >> thinking
>> > > > > > > > >> > > from
>> > > > > > > > >> > > > > > > > > chaining
>> > > > > > > > >> > > > > > > > > > multiple Authorizer? If
>> > > so, I
>> > > > > am
>> > > > > > not
>> > > > > > > sure
>> > > > > > > > whether
>> > > > > > > > >> the
>> > > > > > > > >> > > Broker
>> > > > > > > > >> > > > > > > would
>> > > > > > > > >> > > > > > > > > have
>> > > > > > > > >> > > > > > > > > > enough information to
>> make
>> > > > that
>> > > > > > > decision. I
>> > > > > > > > feel the
>> > > > > > > > >> > > Authorizer
>> > > > > > > > >> > > > > > > > will
>> > > > > > > > >> > > > > > > > > be the
>> > > > > > > > >> > > > > > > > > > one who would have that
>> > > > > > knowledge. E.g.
>> > > > > > > in
>> > > > > > > > Ranger
>> > > > > > > > >> we have
>> > > > > > > > >> > > > > > > explicit
>> > > > > > > > >> > > > > > > > > deny,
>> > > > > > > > >> > > > > > > > > > which means no matter
>> > what,
>> > > > the
>> > > > > > request
>> > > > > > > > should be
>> > > > > > > > >> denied
>> > > > > > > > >> > > for
>> > > > > > > > >> > > > > > the
>> > > > > > > > >> > > > > > > > > user/group
>> > > > > > > > >> > > > > > > > > > or condition. So if you
>> > are
>> > > > > > thinking of
>> > > > > > > > chaining
>> > > > > > > > >> > > Authorizers,
>> > > > > > > > >> > > > > > > then
>> > > > > > > > >> > > > > > > > > > responses should have
>> the
>> > > > third
>> > > > > > state,
>> > > > > > > e.g.
>> > > > > > > > >> > > "DENIED_FINAL", in
>> > > > > > > > >> > > > > > > > which
>> > > > > > > > >> > > > > > > > > case
>> > > > > > > > >> > > > > > > > > > if there is an
>> > Authorization
>> > > > > > chain, it
>> > > > > > > will
>> > > > > > > > be stop
>> > > > > > > > >> and
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > > request
>> > > > > > > > >> > > > > > > > > will be
>> > > > > > > > >> > > > > > > > > > denied and if it is
>> just
>> > > > denied,
>> > > > > > then
>> > > > > > > you
>> > > > > > > > might fall
>> > > > > > > > >> > > back to
>> > > > > > > > >> > > > > > next
>> > > > > > > > >> > > > > > > > > > authorizer. If we don't
>> > have
>> > > > > > chaining of
>> > > > > > > > >> Authorizing in
>> > > > > > > > >> > > mind,
>> > > > > > > > >> > > > > > > then
>> > > > > > > > >> > > > > > > > we
>> > > > > > > > >> > > > > > > > > > should reconsider
>> OPTIONAL
>> > > for
>> > > > > > now. Or
>> > > > > > > > clarify under
>> > > > > > > > >> > > which
>> > > > > > > > >> > > > > > > scenario
>> > > > > > > > >> > > > > > > > > > OPTIONAL will be
>> called.
>> > > > > > > > >> > > > > > > > > > #3.3 Regarding, FILTER
>> v/s
>> > > > > > > LIST_AUTHORIZED,
>> > > > > > > > isn't
>> > > > > > > > >> > > > > > > LIST_AUTHORIZED a
>> > > > > > > > >> > > > > > > > > > special case for
>> "FILTER
>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Ismael Juma <is...@juma.me.uk>.
One more thing: if people have strong opinions that authorize has to be
sync, we could leave it like that for now. If needed, we can later add an
authorizeAsync with another method returning a Boolean indicating that it's
supported. That is, there is a path to evolve the interface (even if a bit
ugly).
What about the other methods, is there consensus that they should be
updated to return CompletionStage?
Ismael
On Fri, Sep 6, 2019, 8:32 AM Ismael Juma <is...@juma.me.uk> wrote:
> I'm on the move, but what Rajini said seems reasonable. I don't think
> using SSDs solves the issue. They can still hang for seconds when they
> fail. Also, many people may not have local SSDs available (remote SSDs like
> EBS hang for tens of seconds when there are issues).
>
> We are currently vulnerable to all of these in the normal read/write path,
> but it seems suboptimal to continue assuming non blocking behavior for
> things that actually do block.
>
> Ismael
>
> On Fri, Sep 6, 2019, 8:24 AM Rajini Sivaram <ra...@gmail.com>
> wrote:
>
>> Hi Jun,
>>
>> For ACLs, the size is also dependent on the number of users. So in
>> deployments with large number of users where some users are not active, an
>> LRU cache may be useful. Agree that there may be other solutions that help
>> avoid the requirement for an async API even for this case.
>>
>> I wasn't expecting our threading model to change very much. Having said
>> that, I haven't figured out how the purgatory can be adapted to work
>> efficiently for this. I was thinking:
>> 1) If authorization future is complete when returning from authorize()
>> call, we invoke the method immediately on the request thread
>> 2) If future is not complete, the request goes into a purgatory
>> 3) When future completes, we mark the request ready to be run. This can be
>> done on ForkJoinPool.commonPool or an executor. But we don't handle the
>> request on that thread
>> 4) We handle the request on the request thread after 3).
>>
>> I totally agree that this adds complexity to the code. Will wait for
>> Ismael
>> to comment on whether he had a different model in mind.
>>
>> Thanks,
>>
>> Rajini
>>
>>
>> On Thu, Sep 5, 2019 at 10:29 PM Jun Rao <ju...@confluent.io> wrote:
>>
>> > Hi, Ismael,
>> >
>> > Yes, I agree that there are 2 separate discussion points, one on the API
>> > and the other on the implementation.
>> >
>> > First on the API, in general, my feeling is that the authorize() call is
>> > expected to be fast to reduce request latency. Adding an async API
>> could be
>> > mis-leading since it might give users the impression that it would be
>> > arbitrarily long. In the short term, currently, we cache all partition
>> > level metadata on every broker. The authorization metadata is
>> proportional
>> > to the number of topics, which typically is less in size. So it we can
>> > cache all partition level metadata, we should be able to cache all
>> > authorization metadata. Longer term, if we want to support more metadata
>> > beyond memory, another option is to put all metadata in an SSD backed
>> > key/value store. The lookup time could still be under 1ms, which we
>> could
>> > potentially absorb in the request threads. So, I am not sure async
>> > authorize() api is strictly needed even in the long term.
>> >
>> > Second on the implementation. I was more concerned about the impact on
>> our
>> > threading model. If we make authorize() async, some threads have to
>> > complete the future when it's ready. It seems to me that those threads
>> have
>> > to be inside the Authorizer implementation since only it knows when to
>> > complete a future? If so, it feels weird that all the logic of request
>> > handling post authorize() is now handled by external threads inside a
>> > plugin. It's not clear how we can configure and monitor them, and how
>> > throttling would work. Perhaps we can provide a bit more detail on the
>> new
>> > threading model.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Wed, Sep 4, 2019 at 9:56 PM Ismael Juma <is...@juma.me.uk> wrote:
>> >
>> > > Hi Jun,
>> > >
>> > > I think it's important to discuss the pluggable API versus the calling
>> > > implementation as two separate points.
>> > >
>> > > From an API perspective, are you suggesting that we should tell users
>> > that
>> > > they cannot block in authorize? Or are you saying that it's OK to
>> block
>> > on
>> > > authorize on occasion and the recommendation would be for people to
>> > > increase the number of threads in the request thread pool?
>> > Architecturally,
>> > > this feels wrong in my opinion.
>> > >
>> > > From the calling implementation perspective, you don't need a
>> callback.
>> > You
>> > > can basically say:
>> > >
>> > > authorize(...).map { result =>
>> > > ...
>> > > }
>> > >
>> > > And then have a common method take that Future and handle it
>> > synchronously
>> > > if it's already complete or submit it to the purgatory if not. This is
>> > > similar to how many modern async web libraries work.
>> > >
>> > > As Rajini said, this could be done later. The initial implementation
>> > could
>> > > simply do `toCompletableFuture().get()` to do it synchronously. But it
>> > > would mean that we could add this functionality without changing the
>> > > pluggable interface.
>> > >
>> > > Ismael
>> > >
>> > > On Wed, Sep 4, 2019 at 5:29 AM Jun Rao <ju...@confluent.io> wrote:
>> > >
>> > > > Hi, Rajini,
>> > > >
>> > > > Thanks for the KIP. I was concerned about #4 too. If we change the
>> > > handling
>> > > > of all requests to use an async authorize() api, will that cause the
>> > code
>> > > > much harder to understand? There are quite a few callbacks already.
>> I
>> > am
>> > > > not sure that we want to introduce more of those. The benefit from
>> > async
>> > > > authorize() api seems limited.
>> > > >
>> > > > Jun
>> > > >
>> > > > On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <
>> rajinisivaram@gmail.com
>> > >
>> > > > wrote:
>> > > >
>> > > > > Hi Don,
>> > > > >
>> > > > > Thanks for your note.
>> > > > >
>> > > > > 1) The intention is to avoid blocking in the calling thread. We
>> > already
>> > > > > have several requests that are put into a purgatory when waiting
>> for
>> > > > remote
>> > > > > communication, for example produce request waiting for
>> replication.
>> > > Since
>> > > > > we have a limited number of request threads in the broker, we
>> want to
>> > > > make
>> > > > > progress with other requests, while one is waiting on any form of
>> > > remote
>> > > > > communication.
>> > > > >
>> > > > > 2) Async management calls will be useful with the default
>> authorizer
>> > > when
>> > > > > KIP-500 removes ZK and we rely on Kafka instead. Our current
>> ZK-based
>> > > > > implementation as well as any custom implementations that don't
>> want
>> > to
>> > > > be
>> > > > > async will just need to return a sync'ed value. So instead of
>> > > returning `
>> > > > > value`, the code would just return
>> > > > > `CompletableFuture.completedFuture(value)
>> > > > > `. So it would be just a single line change in the implementation
>> > with
>> > > > the
>> > > > > new API. The caller would treat completedFuture exactly as it does
>> > > today,
>> > > > > processing the request synchronously without using a purgatory.
>> > > > >
>> > > > > 3) For implementations that return a completedFuture as described
>> in
>> > > 2),
>> > > > > the behaviour would remain exactly the same. No additional
>> threads or
>> > > > > purgatory will be used for this case. So there would be no
>> > performance
>> > > > > penalty. For implementations that return a future that is not
>> > complete,
>> > > > we
>> > > > > prioritise running more requests concurrently. So in a deployment
>> > with
>> > > a
>> > > > > large number of clients, we would improve performance by allowing
>> > other
>> > > > > requests to be processed on the request threads while some are
>> > waiting
>> > > > for
>> > > > > authorization metadata.
>> > > > >
>> > > > > 4) I was concerned about this too. The goal is to make the API
>> > flexible
>> > > > > enough to handle large scale deployments in future when caching
>> all
>> > > > > authorization metadata in each broker is not viable. Using an
>> async
>> > API
>> > > > > that returns CompletionStage, the caller has the option to handle
>> the
>> > > > > result synchronously or asynchronously, so we don't necessarily
>> need
>> > to
>> > > > > update the calling code right away. Custom authorizers using the
>> > async
>> > > > API
>> > > > > have full control over whether authorization is performed in-line
>> > since
>> > > > > completedFuture will always be handled synchronously. We do need
>> to
>> > > > update
>> > > > > KafkaApis to take advantage of the asynchronous API to improve
>> scale.
>> > > > Even
>> > > > > though this is a big change, since we will be doing the same for
>> all
>> > > > > requests, it shouldn't be too hard to maintain since the same
>> pattern
>> > > > will
>> > > > > be used for all requests.
>> > > > >
>> > > > > Regards,
>> > > > >
>> > > > > Rajini
>> > > > >
>> > > > > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <bosco@apache.org
>> >
>> > > > wrote:
>> > > > >
>> > > > > > Hi Rajini
>> > > > > >
>> > > > > > Help me understand this a bit more.
>> > > > > >
>> > > > > > 1. For all practical purpose, without authorization you can't
>> go to
>> > > the
>> > > > > > next step. The calling code needs to block anyway. So should we
>> > just
>> > > > let
>> > > > > > the implementation code do the async part?
>> > > > > > 2. If you feel management calls need to be async, then we should
>> > > > consider
>> > > > > > another set of async APIs. I don't feel we should complicate it
>> > > > further (
>> > > > > > 3. Another concern I have is wrt performance. Kafka has been
>> built
>> > to
>> > > > > > handle 1000s per second requests. Not sure whether making it
>> async
>> > > will
>> > > > > add
>> > > > > > any unnecessary overhead.
>> > > > > > 4. How much complication would this add on the calling side?
>> And is
>> > > it
>> > > > > > worth it?
>> > > > > >
>> > > > > > Thanks
>> > > > > >
>> > > > > > Bosco
>> > > > > >
>> > > > > >
>> > > > > > On 9/3/19, 8:50 AM, "Rajini Sivaram" <ra...@gmail.com>
>> > > wrote:
>> > > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > Ismael brought up a point that it will be good to make the
>> > > > Authorizer
>> > > > > > interface asynchronous to avoid blocking request threads
>> during
>> > > > > remote
>> > > > > > operations.
>> > > > > >
>> > > > > > 1) Since we want to support different backends for
>> > authorization
>> > > > > > metadata,
>> > > > > > making createAcls() and deleteAcls() asynchronous makes
>> sense
>> > > since
>> > > > > > these
>> > > > > > always involve remote operations. When KIP-500 removes
>> > ZooKeeper,
>> > > > we
>> > > > > > would
>> > > > > > want to move ACLs to Kafka and async updates will avoid
>> > > unnecessary
>> > > > > > blocking.
>> > > > > > 2) For authorize() method, we currently use cached ACLs in
>> the
>> > > > > built-in
>> > > > > > authorizers, so synchronous authorise operations work well
>> now.
>> > > But
>> > > > > > async
>> > > > > > authorize() would support this scenario as well as
>> authorizers
>> > in
>> > > > > large
>> > > > > > organisations where an LRU cache would enable a smaller
>> cache
>> > > even
>> > > > > > when the
>> > > > > > backend holds a large amount of ACLs for infrequently used
>> > > > resources
>> > > > > or
>> > > > > > users who don't use the system frequently.
>> > > > > >
>> > > > > > For both cases, the built-in authorizer will continue to be
>> > > > > > synchronous,
>> > > > > > using CompletableFuture.completedFuture() to return the
>> actual
>> > > > > result.
>> > > > > > But
>> > > > > > async API will make custom authorizer implementations more
>> > > > flexible.
>> > > > > I
>> > > > > > would like to know if there are any concerns with these
>> changes
>> > > > > before
>> > > > > > updating the KIP.
>> > > > > >
>> > > > > > *Proposed API:*
>> > > > > > public interface Authorizer extends Configurable, Closeable
>> {
>> > > > > >
>> > > > > > Map<Endpoint, CompletionStage<Void>>
>> > > start(AuthorizerServerInfo
>> > > > > > serverInfo);
>> > > > > > List<CompletionStage<AuthorizationResult>>
>> > > > > > authorize(AuthorizableRequestContext requestContext,
>> > List<Action>
>> > > > > > actions);
>> > > > > > List<CompletionStage<AclCreateResult>>
>> > > > > > createAcls(AuthorizableRequestContext requestContext,
>> > > > > List<AclBinding>
>> > > > > > aclBindings);
>> > > > > > List<CompletionStage<AclDeleteResult>>
>> > > > > > deleteAcls(AuthorizableRequestContext requestContext,
>> > > > > > List<AclBindingFilter> aclBindingFilters);
>> > > > > > CompletionStage<Collection<AclBinding>>
>> > acls(AclBindingFilter
>> > > > > > filter);
>> > > > > > }
>> > > > > >
>> > > > > >
>> > > > > > Thank you,
>> > > > > >
>> > > > > > Rajini
>> > > > > >
>> > > > > > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <
>> > > bosco@apache.org>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Rajini
>> > > > > > >
>> > > > > > > Thanks for clarifying. I am good for now.
>> > > > > > >
>> > > > > > > Regards
>> > > > > > >
>> > > > > > > Bosco
>> > > > > > >
>> > > > > > >
>> > > > > > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <
>> > > rajinisivaram@gmail.com
>> > > > >
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > Hi Don,
>> > > > > > >
>> > > > > > > That should be fine. I guess Ranger loads policies
>> from
>> > the
>> > > > > > database
>> > > > > > > synchronously when the authorizer is configured,
>> similar
>> > to
>> > > > > > > SimpleAclAuthorizer loading from ZooKeeper. Ranger can
>> > > > continue
>> > > > > > to load
>> > > > > > > synchronously from `configure()` or `start()` and
>> return
>> > an
>> > > > > > empty map
>> > > > > > > from
>> > > > > > > `start()`. That would retain the existing behaviour..
>> > When
>> > > > the
>> > > > > > same
>> > > > > > > database stores policies for all listeners and the
>> > policies
>> > > > are
>> > > > > > not
>> > > > > > > stored
>> > > > > > > in Kafka, there is no value in making the load
>> > > asynchronous.
>> > > > > > >
>> > > > > > > Regards,
>> > > > > > >
>> > > > > > > Rajini
>> > > > > > >
>> > > > > > >
>> > > > > > > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai <
>> > > > > > bosco@apache.org>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Rajini
>> > > > > > > >
>> > > > > > > > Assuming this doesn't affect custom plugins, I don't
>> > have
>> > > > any
>> > > > > > > concerns
>> > > > > > > > regarding this.
>> > > > > > > >
>> > > > > > > > I do have one question regarding:
>> > > > > > > >
>> > > > > > > > "For authorizers that don’t store metadata in
>> > ZooKeeper,
>> > > > > > ensure that
>> > > > > > > > authorizer metadata for each listener is available
>> > before
>> > > > > > starting
>> > > > > > > up the
>> > > > > > > > listener. This enables different authorization
>> metadata
>> > > > > stores
>> > > > > > for
>> > > > > > > > different listeners."
>> > > > > > > >
>> > > > > > > > Since Ranger uses its own database for storing
>> > policies,
>> > > do
>> > > > > you
>> > > > > > > anticipate
>> > > > > > > > any issues?
>> > > > > > > >
>> > > > > > > > Thanks
>> > > > > > > >
>> > > > > > > > Bosco
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
>> > > > > > rajinisivaram@gmail.com>
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > Hi all,
>> > > > > > > >
>> > > > > > > > I made another change to the KIP. The KIP was
>> > > > originally
>> > > > > > > proposing to
>> > > > > > > > extend SimpleAclAuthorizer to also implement the
>> > new
>> > > > API
>> > > > > > (in
>> > > > > > > addition
>> > > > > > > > to
>> > > > > > > > the existing API). But since we use the new API
>> > when
>> > > > > > available,
>> > > > > > > this
>> > > > > > > > can
>> > > > > > > > break custom authorizers that extend this class
>> and
>> > > > > > override
>> > > > > > > specific
>> > > > > > > > methods of the old API. To avoid breaking any
>> > > existing
>> > > > > > custom
>> > > > > > > > implementations that extend this class,
>> > particularly
>> > > > > > because it
>> > > > > > > is in
>> > > > > > > > the
>> > > > > > > > public package kafka.security.auth, the KIP now
>> > > > proposes
>> > > > > to
>> > > > > > > retain the
>> > > > > > > > old
>> > > > > > > > authorizer as-is. SimpleAclAuthorizer will
>> > continue
>> > > to
>> > > > > > > implement the
>> > > > > > > > old
>> > > > > > > > API, but will be deprecated. A new authorizer
>> > > > > > implementation
>> > > > > > > > kafka.security.authorizer.AclAuthorizer will be
>> > added
>> > > > for
>> > > > > > the
>> > > > > > > new API
>> > > > > > > > (this
>> > > > > > > > will not be in the public package).
>> > > > > > > >
>> > > > > > > > Please let me know if you have any concerns.
>> > > > > > > >
>> > > > > > > > Regards,
>> > > > > > > >
>> > > > > > > > Rajini
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
>> > > > > > > > rajinisivaram@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Thanks Colin.
>> > > > > > > > >
>> > > > > > > > > If there are no other concerns, I will start
>> vote
>> > > > later
>> > > > > > today.
>> > > > > > > Many
>> > > > > > > > thanks
>> > > > > > > > > to every one for the feedback.
>> > > > > > > > >
>> > > > > > > > > Regards,
>> > > > > > > > >
>> > > > > > > > > Rajini
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe
>> <
>> > > > > > > cmccabe@apache.org>
>> > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > >> Thanks, Rajini. It looks good to me.
>> > > > > > > > >>
>> > > > > > > > >> best,
>> > > > > > > > >> Colin
>> > > > > > > > >>
>> > > > > > > > >>
>> > > > > > > > >> On Thu, Aug 15, 2019, at 11:37, Rajini
>> Sivaram
>> > > > wrote:
>> > > > > > > > >> > Hi Colin,
>> > > > > > > > >> >
>> > > > > > > > >> > Thanks for the review. I have updated the
>> KIP
>> > to
>> > > > > move
>> > > > > > the
>> > > > > > > > interfaces for
>> > > > > > > > >> > request context and server info to the
>> > > authorizer
>> > > > > > package.
>> > > > > > > These
>> > > > > > > > are now
>> > > > > > > > >> > called AuthorizableRequestContext and
>> > > > > > AuthorizerServerInfo.
>> > > > > > > > Endpoint is
>> > > > > > > > >> now
>> > > > > > > > >> > a class in org.apache.kafka.common to make
>> it
>> > > > > reusable
>> > > > > > > since we
>> > > > > > > > already
>> > > > > > > > >> > have multiple implementations of it. I have
>> > > > removed
>> > > > > > > requestName
>> > > > > > > > from the
>> > > > > > > > >> > request context interface since authorizers
>> > can
>> > > > > > distinguish
>> > > > > > > > follower
>> > > > > > > > >> fetch
>> > > > > > > > >> > and consumer fetch from the operation being
>> > > > > > authorized. So
>> > > > > > > 16-bit
>> > > > > > > > >> request
>> > > > > > > > >> > type should be sufficient for audit
>> logging.
>> > > Also
>> > > > > > replaced
>> > > > > > > > AuditFlag
>> > > > > > > > >> with
>> > > > > > > > >> > two booleans as you suggested.
>> > > > > > > > >> >
>> > > > > > > > >> > Can you take another look and see if the
>> KIP
>> > is
>> > > > > ready
>> > > > > > for
>> > > > > > > voting?
>> > > > > > > > >> >
>> > > > > > > > >> > Thanks for all your help!
>> > > > > > > > >> >
>> > > > > > > > >> > Regards,
>> > > > > > > > >> >
>> > > > > > > > >> > Rajini
>> > > > > > > > >> >
>> > > > > > > > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin
>> McCabe <
>> > > > > > > cmccabe@apache.org>
>> > > > > > > > >> wrote:
>> > > > > > > > >> >
>> > > > > > > > >> > > Hi Rajini,
>> > > > > > > > >> > >
>> > > > > > > > >> > > I think it would be good to rename
>> > > > > > KafkaRequestContext to
>> > > > > > > > something
>> > > > > > > > >> like
>> > > > > > > > >> > > AuthorizableRequestContext, and put it in
>> > the
>> > > > > > > > >> > > org.apache.kafka.server.authorizer
>> > namespace.
>> > > > If
>> > > > > > we put
>> > > > > > > it in
>> > > > > > > > the
>> > > > > > > > >> > > org.apache.kafka.common namespace, then
>> it's
>> > > not
>> > > > > > really
>> > > > > > > clear
>> > > > > > > > that
>> > > > > > > > >> it's
>> > > > > > > > >> > > part of the Authorizer API. Since this
>> > class
>> > > is
>> > > > > > purely an
>> > > > > > > > interface,
>> > > > > > > > >> with
>> > > > > > > > >> > > no concrete implementation of anything,
>> > > there's
>> > > > > > nothing
>> > > > > > > common
>> > > > > > > > to
>> > > > > > > > >> really
>> > > > > > > > >> > > reuse in any case. We definitely don't
>> want
>> > > > > > someone to
>> > > > > > > > accidentally
>> > > > > > > > >> add or
>> > > > > > > > >> > > remove methods because they think this is
>> > just
>> > > > > > another
>> > > > > > > internal
>> > > > > > > > class
>> > > > > > > > >> used
>> > > > > > > > >> > > for requests.
>> > > > > > > > >> > >
>> > > > > > > > >> > > The BrokerInfo class is a nice
>> improvement.
>> > > It
>> > > > > > looks
>> > > > > > > like it
>> > > > > > > > will be
>> > > > > > > > >> > > useful for passing in information about
>> the
>> > > > > context
>> > > > > > we're
>> > > > > > > > running
>> > > > > > > > >> in. It
>> > > > > > > > >> > > would be nice to call this class
>> ServerInfo
>> > > > rather
>> > > > > > than
>> > > > > > > > BrokerInfo,
>> > > > > > > > >> since
>> > > > > > > > >> > > we will want to run the authorizer on
>> > > > controllers
>> > > > > > as well
>> > > > > > > as on
>> > > > > > > > >> brokers,
>> > > > > > > > >> > > and the controller may run as a separate
>> > > process
>> > > > > > post
>> > > > > > > KIP-500.
>> > > > > > > > I also
>> > > > > > > > >> > > think that this class should be in the
>> > > > > > > > >> org.apache.kafka.server.authorizer
>> > > > > > > > >> > > namespace. Again, it is an interface,
>> not a
>> > > > > > concrete
>> > > > > > > > implementation,
>> > > > > > > > >> and
>> > > > > > > > >> > > it's an interface that is very
>> specifically
>> > > for
>> > > > > the
>> > > > > > > authorizer.
>> > > > > > > > >> > >
>> > > > > > > > >> > > I agree that we probably don't have
>> enough
>> > > > > > information
>> > > > > > > > preserved for
>> > > > > > > > >> > > requests currently to always know what
>> > entity
>> > > > made
>> > > > > > them.
>> > > > > > > So we
>> > > > > > > > can
>> > > > > > > > >> leave
>> > > > > > > > >> > > that out for now (except in the special
>> case
>> > > of
>> > > > > > Fetch).
>> > > > > > > > Perhaps we
>> > > > > > > > >> can add
>> > > > > > > > >> > > this later if it's needed.
>> > > > > > > > >> > >
>> > > > > > > > >> > > I understand the intention behind
>> > > > > AuthorizationMode
>> > > > > > > (which is
>> > > > > > > > now
>> > > > > > > > >> called
>> > > > > > > > >> > > AuditFlag in the latest revision). But
>> it
>> > > still
>> > > > > > feels
>> > > > > > > > complex. What
>> > > > > > > > >> if we
>> > > > > > > > >> > > just had two booleans in Action:
>> > logSuccesses
>> > > > and
>> > > > > > > logFailures?
>> > > > > > > > That
>> > > > > > > > >> seems
>> > > > > > > > >> > > to cover all the cases here.
>> > > > MANDATORY_AUTHORIZE
>> > > > > =
>> > > > > > true,
>> > > > > > > true.
>> > > > > > > > >> > > OPTIONAL_AUTHORIZE = true, false.
>> FILTER =
>> > > > true,
>> > > > > > false.
>> > > > > > > > >> LIST_AUTHORIZED =
>> > > > > > > > >> > > false, false. Would there be anything
>> lost
>> > > > versus
>> > > > > > having
>> > > > > > > the
>> > > > > > > > enum?
>> > > > > > > > >> > >
>> > > > > > > > >> > > best,
>> > > > > > > > >> > > Colin
>> > > > > > > > >> > >
>> > > > > > > > >> > >
>> > > > > > > > >> > > On Wed, Aug 14, 2019, at 06:29, Mickael
>> > Maison
>> > > > > > wrote:
>> > > > > > > > >> > > > Hi Rajini,
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > Thanks for the KIP!
>> > > > > > > > >> > > > I really like that authorize() will be
>> > able
>> > > to
>> > > > > > take a
>> > > > > > > batch of
>> > > > > > > > >> > > > requests, this will speed up many
>> > > > > implementations!
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini
>> > > > Sivaram <
>> > > > > > > > >> rajinisivaram@gmail.com>
>> > > > > > > > >> > > wrote:
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > Thanks David! I have fixed the typo.
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > Also made a couple of changes to make
>> > the
>> > > > > > context
>> > > > > > > > interfaces more
>> > > > > > > > >> > > generic.
>> > > > > > > > >> > > > > KafkaRequestContext now returns the
>> > 16-bit
>> > > > API
>> > > > > > key as
>> > > > > > > Colin
>> > > > > > > > >> suggested
>> > > > > > > > >> > > as
>> > > > > > > > >> > > > > well as the friendly name used in
>> > metrics
>> > > > > which
>> > > > > > are
>> > > > > > > useful
>> > > > > > > > in
>> > > > > > > > >> audit
>> > > > > > > > >> > > logs.
>> > > > > > > > >> > > > > `Authorizer#start` is now provided a
>> > > generic
>> > > > > > > `BrokerInfo`
>> > > > > > > > >> interface
>> > > > > > > > >> > > that
>> > > > > > > > >> > > > > gives cluster id, broker id and
>> endpoint
>> > > > > > information.
>> > > > > > > The
>> > > > > > > > generic
>> > > > > > > > >> > > interface
>> > > > > > > > >> > > > > can potentially be used in other
>> broker
>> > > > > plugins
>> > > > > > in
>> > > > > > > future
>> > > > > > > > and
>> > > > > > > > >> provides
>> > > > > > > > >> > > > > dynamically generated configs like
>> > broker
>> > > id
>> > > > > > and ports
>> > > > > > > > which are
>> > > > > > > > >> > > currently
>> > > > > > > > >> > > > > not available to plugins unless these
>> > > > configs
>> > > > > > are
>> > > > > > > statically
>> > > > > > > > >> > > configured.
>> > > > > > > > >> > > > > Please let me know if there are any
>> > > > concerns.
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > Regards,
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > Rajini
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > On Tue, Aug 13, 2019 at 4:30 PM David
>> > > Jacot
>> > > > <
>> > > > > > > > djacot@confluent.io>
>> > > > > > > > >> > > wrote:
>> > > > > > > > >> > > > >
>> > > > > > > > >> > > > > > Hi Rajini,
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > Thank you for the update! It looks
>> > good
>> > > to
>> > > > > me.
>> > > > > > > There is a
>> > > > > > > > typo
>> > > > > > > > >> in the
>> > > > > > > > >> > > > > > `AuditFlag` enum:
>> > `MANDATORY_AUTHOEIZE`
>> > > ->
>> > > > > > > > >> `MANDATORY_AUTHORIZE`.
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > Regards,
>> > > > > > > > >> > > > > > David
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > On Mon, Aug 12, 2019 at 2:54 PM
>> Rajini
>> > > > > > Sivaram <
>> > > > > > > > >> > > rajinisivaram@gmail.com>
>> > > > > > > > >> > > > > > wrote:
>> > > > > > > > >> > > > > >
>> > > > > > > > >> > > > > > > Hi David,
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Thanks for reviewing the KIP!
>> Since
>> > > > > > questions
>> > > > > > > about
>> > > > > > > > >> `authorization
>> > > > > > > > >> > > mode`
>> > > > > > > > >> > > > > > > and `count` have come up multiple
>> > > > times, I
>> > > > > > have
>> > > > > > > renamed
>> > > > > > > > both.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > 1) Renamed `count` to
>> > > > > > `resourceReferenceCount`.
>> > > > > > > It is
>> > > > > > > > the
>> > > > > > > > >> number
>> > > > > > > > >> > > of times
>> > > > > > > > >> > > > > > > the resource being authorized is
>> > > > > referenced
>> > > > > > > within the
>> > > > > > > > >> request.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > 2) Renamed `AuthorizationMode` to
>> > > > > > `AuditFlag`. It
>> > > > > > > is
>> > > > > > > > provided
>> > > > > > > > >> to
>> > > > > > > > >> > > improve
>> > > > > > > > >> > > > > > > audit logging in the authorizer.
>> The
>> > > > enum
>> > > > > > values
>> > > > > > > have
>> > > > > > > > javadoc
>> > > > > > > > >> which
>> > > > > > > > >> > > > > > > indicate how the authorization
>> > result
>> > > is
>> > > > > > used in
>> > > > > > > each
>> > > > > > > > of the
>> > > > > > > > >> modes
>> > > > > > > > >> > > to
>> > > > > > > > >> > > > > > > enable authorizers to log audit
>> > > messages
>> > > > > at
>> > > > > > the
>> > > > > > > right
>> > > > > > > > severity
>> > > > > > > > >> > > level.
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Regards,
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > Rajini
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM
>> > David
>> > > > > Jacot
>> > > > > > <
>> > > > > > > > >> djacot@confluent.io>
>> > > > > > > > >> > > wrote:
>> > > > > > > > >> > > > > > >
>> > > > > > > > >> > > > > > > > Hi Rajini,
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > Thank you for the KIP.
>> Overall, it
>> > > > looks
>> > > > > > good
>> > > > > > > to me.
>> > > > > > > > I have
>> > > > > > > > >> few
>> > > > > > > > >> > > > > > > > questions/suggestions:
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > 1. It is hard to grasp what
>> > > > > > `Action#count` is
>> > > > > > > for. I
>> > > > > > > > guess I
>> > > > > > > > >> > > understand
>> > > > > > > > >> > > > > > > > where you want to go with it
>> but
>> > it
>> > > > took
>> > > > > > me a
>> > > > > > > while to
>> > > > > > > > >> figure it
>> > > > > > > > >> > > out.
>> > > > > > > > >> > > > > > > > Perhaps, we could come up with
>> a
>> > > > better
>> > > > > > name
>> > > > > > > than
>> > > > > > > > `count`?
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > 2. I had a hard time trying to
>> > > > > understand
>> > > > > > the
>> > > > > > > > >> `AuthorizationMode`
>> > > > > > > > >> > > > > > > concept,
>> > > > > > > > >> > > > > > > > especially wrt. the OPTIONAL
>> one.
>> > My
>> > > > > > > understanding is
>> > > > > > > > that
>> > > > > > > > >> an
>> > > > > > > > >> > > ACL is
>> > > > > > > > >> > > > > > > either
>> > > > > > > > >> > > > > > > > defined or not. Could you
>> > elaborate
>> > > a
>> > > > > bit
>> > > > > > more
>> > > > > > > on
>> > > > > > > > that?
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > Thanks,
>> > > > > > > > >> > > > > > > > David
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM
>> > Don
>> > > > > Bosco
>> > > > > > Durai
>> > > > > > > <
>> > > > > > > > >> > > bosco@apache.org>
>> > > > > > > > >> > > > > > > wrote:
>> > > > > > > > >> > > > > > > >
>> > > > > > > > >> > > > > > > > > Hi Rajini
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > 3.2 - This makes sense.
>> Thanks
>> > for
>> > > > > > clarifying.
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > Rest looks fine. Once the
>> > > > > > implementations are
>> > > > > > > done,
>> > > > > > > > it
>> > > > > > > > >> will be
>> > > > > > > > >> > > more
>> > > > > > > > >> > > > > > > clear
>> > > > > > > > >> > > > > > > > > on the different values
>> > > RequestType
>> > > > > and
>> > > > > > Mode.
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > Thanks
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > Bosco
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini
>> > > > Sivaram"
>> > > > > <
>> > > > > > > > >> rajinisivaram@gmail.com
>> > > > > > > > >> > > >
>> > > > > > > > >> > > > > > wrote:
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > Hi Don,
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > Thanks for the
>> suggestions.
>> > A
>> > > > few
>> > > > > > > responses
>> > > > > > > > below:
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > 3.1 Can rename and
>> improve
>> > > docs
>> > > > if
>> > > > > > we keep
>> > > > > > > > this. Let's
>> > > > > > > > >> > > finish the
>> > > > > > > > >> > > > > > > > > discussion on Colin's
>> > > > suggestions
>> > > > > > > regarding this
>> > > > > > > > >> first.
>> > > > > > > > >> > > > > > > > > 3.2 No, I was thinking of
>> > some
>> > > > > > requests
>> > > > > > > that
>> > > > > > > > have an
>> > > > > > > > >> old
>> > > > > > > > >> > > way of
>> > > > > > > > >> > > > > > > > > authorizing
>> > > > > > > > >> > > > > > > > > and a new way where we
>> have
>> > > > > > retained the
>> > > > > > > old
>> > > > > > > > way for
>> > > > > > > > >> > > backward
>> > > > > > > > >> > > > > > > > > compatibility. One
>> example
>> > is
>> > > > > > > Cluster:Create
>> > > > > > > > >> permission to
>> > > > > > > > >> > > create
>> > > > > > > > >> > > > > > > > > topics.
>> > > > > > > > >> > > > > > > > > We have replaced this
>> with
>> > > > > > fine-grained
>> > > > > > > topic
>> > > > > > > > create
>> > > > > > > > >> > > access using
>> > > > > > > > >> > > > > > > > > Topic:Create
>> > > > > > > > >> > > > > > > > > for topic patterns. But
>> we
>> > > still
>> > > > > > check if
>> > > > > > > user
>> > > > > > > > has
>> > > > > > > > >> > > Cluster:Create
>> > > > > > > > >> > > > > > > > > first. If
>> > > > > > > > >> > > > > > > > > Denied, the deny is
>> ignored
>> > > and
>> > > > we
>> > > > > > check
>> > > > > > > > >> Topic:Create. We
>> > > > > > > > >> > > dont
>> > > > > > > > >> > > > > > want
>> > > > > > > > >> > > > > > > > to
>> > > > > > > > >> > > > > > > > > log
>> > > > > > > > >> > > > > > > > > DENY for Cluster:Create
>> at
>> > > INFO
>> > > > > > level for
>> > > > > > > this,
>> > > > > > > > since
>> > > > > > > > >> this
>> > > > > > > > >> > > is
>> > > > > > > > >> > > > > > not a
>> > > > > > > > >> > > > > > > > > mandatory ACL for
>> creating
>> > > > topics.
>> > > > > > We
>> > > > > > > will get
>> > > > > > > > >> appropriate
>> > > > > > > > >> > > logs
>> > > > > > > > >> > > > > > > from
>> > > > > > > > >> > > > > > > > > the
>> > > > > > > > >> > > > > > > > > subsequent Topic:Create
>> in
>> > > this
>> > > > > > case.
>> > > > > > > > >> > > > > > > > > 3.3 They are not quite
>> the
>> > > same.
>> > > > > > FILTER
>> > > > > > > implies
>> > > > > > > > that
>> > > > > > > > >> user
>> > > > > > > > >> > > > > > actually
>> > > > > > > > >> > > > > > > > > requested access to and
>> > > > performed
>> > > > > > some
>> > > > > > > > operation on
>> > > > > > > > >> the
>> > > > > > > > >> > > filtered
>> > > > > > > > >> > > > > > > > > resources.
>> > > > > > > > >> > > > > > > > > LIST_AUTHORZED did not
>> > result
>> > > in
>> > > > > any
>> > > > > > > actual
>> > > > > > > > access.
>> > > > > > > > >> User
>> > > > > > > > >> > > simply
>> > > > > > > > >> > > > > > > > wanted
>> > > > > > > > >> > > > > > > > > to
>> > > > > > > > >> > > > > > > > > know what they are
>> allowed
>> > to
>> > > > > > access.
>> > > > > > > > >> > > > > > > > > 3.4 Request types are
>> > Produce,
>> > > > > > JoinGroup,
>> > > > > > > > OffsetCommit
>> > > > > > > > >> > > etc. So
>> > > > > > > > >> > > > > > that
>> > > > > > > > >> > > > > > > > is
>> > > > > > > > >> > > > > > > > > different from
>> authorization
>> > > > mode,
>> > > > > > > operation
>> > > > > > > > etc.
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > On Thu, Aug 8, 2019 at
>> 11:36
>> > > PM
>> > > > > Don
>> > > > > > Bosco
>> > > > > > > Durai
>> > > > > > > > <
>> > > > > > > > >> > > > > > bosco@apache.org>
>> > > > > > > > >> > > > > > > > > wrote:
>> > > > > > > > >> > > > > > > > >
>> > > > > > > > >> > > > > > > > > > Hi Rajini
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > Thanks for clarifying.
>> > This
>> > > is
>> > > > > > very
>> > > > > > > helpful...
>> > > > > > > > >> > > > > > > > > >
>> > > > > > > > >> > > > > > > > > > #1 - On the Ranger
>> side,
>> > we
>> > > > > > should be
>> > > > > > > able to
>> > > > > > > > handle
>> > > > > > > > >> > > multiple
>> > > > > > > > >> > > > > > > > > requests at
>> > > > > > > > >> > > > > > > > > > the same time. I was
>> just
>> > > not
>> > > > > > sure how
>> > > > > > > much
>> > > > > > > > >> processing
>> > > > > > > > >> > > overhead
>> > > > > > > > >> > > > > > > > will
>> > > > > > > > >> > > > > > > > > be
>> > > > > > > > >> > > > > > > > > > there on the Broker
>> side
>> > to
>> > > > > split
>> > > > > > and
>> > > > > > > then
>> > > > > > > > >> consolidate
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > > results.
>> > > > > > > > >> > > > > > > > > If it
>> > > > > > > > >> > > > > > > > > > is negligible, then
>> this
>> > is
>> > > > the
>> > > > > > better
>> > > > > > > way.
>> > > > > > > > It will
>> > > > > > > > >> > > make it
>> > > > > > > > >> > > > > > > future
>> > > > > > > > >> > > > > > > > > proof.
>> > > > > > > > >> > > > > > > > > > #2 - I agree, having a
>> > > single
>> > > > > > "start"
>> > > > > > > call
>> > > > > > > > makes it
>> > > > > > > > >> > > cleaner.
>> > > > > > > > >> > > > > > The
>> > > > > > > > >> > > > > > > > > > Authorization
>> > implementation
>> > > > > will
>> > > > > > only
>> > > > > > > have
>> > > > > > > > to do
>> > > > > > > > >> the
>> > > > > > > > >> > > > > > > > initialization
>> > > > > > > > >> > > > > > > > > only
>> > > > > > > > >> > > > > > > > > > once.
>> > > > > > > > >> > > > > > > > > > #3.1 - Thanks for the
>> > > > > > clarification. I
>> > > > > > > think
>> > > > > > > > it
>> > > > > > > > >> makes
>> > > > > > > > >> > > sense to
>> > > > > > > > >> > > > > > > have
>> > > > > > > > >> > > > > > > > > this.
>> > > > > > > > >> > > > > > > > > > The term "Mode" might
>> not
>> > be
>> > > > > > explicit
>> > > > > > > enough.
>> > > > > > > > >> > > Essentially it
>> > > > > > > > >> > > > > > > seems
>> > > > > > > > >> > > > > > > > > you want
>> > > > > > > > >> > > > > > > > > > the Authorizer to know
>> the
>> > > > > > > Intent/Purpose of
>> > > > > > > > the
>> > > > > > > > >> > > authorize call
>> > > > > > > > >> > > > > > > and
>> > > > > > > > >> > > > > > > > > let the
>> > > > > > > > >> > > > > > > > > > Authorizer decide what
>> to
>> > > log
>> > > > as
>> > > > > > Audit
>> > > > > > > event.
>> > > > > > > > >> Changing
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > > > > class/field name
>> > > > > > > > >> > > > > > > > > > or giving more
>> > documentation
>> > > > > will
>> > > > > > do.
>> > > > > > > > >> > > > > > > > > > #3.2 - Regarding the
>> > option
>> > > > > > "OPTIONAL".
>> > > > > > > Are
>> > > > > > > > you
>> > > > > > > > >> thinking
>> > > > > > > > >> > > from
>> > > > > > > > >> > > > > > > > > chaining
>> > > > > > > > >> > > > > > > > > > multiple Authorizer? If
>> > > so, I
>> > > > > am
>> > > > > > not
>> > > > > > > sure
>> > > > > > > > whether
>> > > > > > > > >> the
>> > > > > > > > >> > > Broker
>> > > > > > > > >> > > > > > > would
>> > > > > > > > >> > > > > > > > > have
>> > > > > > > > >> > > > > > > > > > enough information to
>> make
>> > > > that
>> > > > > > > decision. I
>> > > > > > > > feel the
>> > > > > > > > >> > > Authorizer
>> > > > > > > > >> > > > > > > > will
>> > > > > > > > >> > > > > > > > > be the
>> > > > > > > > >> > > > > > > > > > one who would have that
>> > > > > > knowledge. E.g.
>> > > > > > > in
>> > > > > > > > Ranger
>> > > > > > > > >> we have
>> > > > > > > > >> > > > > > > explicit
>> > > > > > > > >> > > > > > > > > deny,
>> > > > > > > > >> > > > > > > > > > which means no matter
>> > what,
>> > > > the
>> > > > > > request
>> > > > > > > > should be
>> > > > > > > > >> denied
>> > > > > > > > >> > > for
>> > > > > > > > >> > > > > > the
>> > > > > > > > >> > > > > > > > > user/group
>> > > > > > > > >> > > > > > > > > > or condition. So if you
>> > are
>> > > > > > thinking of
>> > > > > > > > chaining
>> > > > > > > > >> > > Authorizers,
>> > > > > > > > >> > > > > > > then
>> > > > > > > > >> > > > > > > > > > responses should have
>> the
>> > > > third
>> > > > > > state,
>> > > > > > > e.g.
>> > > > > > > > >> > > "DENIED_FINAL", in
>> > > > > > > > >> > > > > > > > which
>> > > > > > > > >> > > > > > > > > case
>> > > > > > > > >> > > > > > > > > > if there is an
>> > Authorization
>> > > > > > chain, it
>> > > > > > > will
>> > > > > > > > be stop
>> > > > > > > > >> and
>> > > > > > > > >> > > the
>> > > > > > > > >> > > > > > > request
>> > > > > > > > >> > > > > > > > > will be
>> > > > > > > > >> > > > > > > > > > denied and if it is
>> just
>> > > > denied,
>> > > > > > then
>> > > > > > > you
>> > > > > > > > might fall
>> > > > > > > > >> > > back to
>> > > > > > > > >> > > > > > next
>> > > > > > > > >> > > > > > > > > > authorizer. If we don't
>> > have
>> > > > > > chaining of
>> > > > > > > > >> Authorizing in
>> > > > > > > > >> > > mind,
>> > > > > > > > >> > > > > > > then
>> > > > > > > > >> > > > > > > > we
>> > > > > > > > >> > > > > > > > > > should reconsider
>> OPTIONAL
>> > > for
>> > > > > > now. Or
>> > > > > > > > clarify under
>> > > > > > > > >> > > which
>> > > > > > > > >> > > > > > > scenario
>> > > > > > > > >> > > > > > > > > > OPTIONAL will be
>> called.
>> > > > > > > > >> > > > > > > > > > #3.3 Regarding, FILTER
>> v/s
>> > > > > > > LIST_AUTHORIZED,
>> > > > > > > > isn't
>> > > > > > > > >> > > > > > > LIST_AUTHORIZED a
>> > > > > > > > >> > > > > > > > > > special case for
>> "FILTER
>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Ismael Juma <is...@juma.me.uk>.
I'm on the move, but what Rajini said seems reasonable. I don't think using
SSDs solves the issue. They can still hang for seconds when they fail.
Also, many people may not have local SSDs available (remote SSDs like EBS
hang for tens of seconds when there are issues).
We are currently vulnerable to all of these in the normal read/write path,
but it seems suboptimal to continue assuming non blocking behavior for
things that actually do block.
Ismael
On Fri, Sep 6, 2019, 8:24 AM Rajini Sivaram <ra...@gmail.com> wrote:
> Hi Jun,
>
> For ACLs, the size is also dependent on the number of users. So in
> deployments with large number of users where some users are not active, an
> LRU cache may be useful. Agree that there may be other solutions that help
> avoid the requirement for an async API even for this case.
>
> I wasn't expecting our threading model to change very much. Having said
> that, I haven't figured out how the purgatory can be adapted to work
> efficiently for this. I was thinking:
> 1) If authorization future is complete when returning from authorize()
> call, we invoke the method immediately on the request thread
> 2) If future is not complete, the request goes into a purgatory
> 3) When future completes, we mark the request ready to be run. This can be
> done on ForkJoinPool.commonPool or an executor. But we don't handle the
> request on that thread
> 4) We handle the request on the request thread after 3).
>
> I totally agree that this adds complexity to the code. Will wait for Ismael
> to comment on whether he had a different model in mind.
>
> Thanks,
>
> Rajini
>
>
> On Thu, Sep 5, 2019 at 10:29 PM Jun Rao <ju...@confluent.io> wrote:
>
> > Hi, Ismael,
> >
> > Yes, I agree that there are 2 separate discussion points, one on the API
> > and the other on the implementation.
> >
> > First on the API, in general, my feeling is that the authorize() call is
> > expected to be fast to reduce request latency. Adding an async API could
> be
> > mis-leading since it might give users the impression that it would be
> > arbitrarily long. In the short term, currently, we cache all partition
> > level metadata on every broker. The authorization metadata is
> proportional
> > to the number of topics, which typically is less in size. So it we can
> > cache all partition level metadata, we should be able to cache all
> > authorization metadata. Longer term, if we want to support more metadata
> > beyond memory, another option is to put all metadata in an SSD backed
> > key/value store. The lookup time could still be under 1ms, which we could
> > potentially absorb in the request threads. So, I am not sure async
> > authorize() api is strictly needed even in the long term.
> >
> > Second on the implementation. I was more concerned about the impact on
> our
> > threading model. If we make authorize() async, some threads have to
> > complete the future when it's ready. It seems to me that those threads
> have
> > to be inside the Authorizer implementation since only it knows when to
> > complete a future? If so, it feels weird that all the logic of request
> > handling post authorize() is now handled by external threads inside a
> > plugin. It's not clear how we can configure and monitor them, and how
> > throttling would work. Perhaps we can provide a bit more detail on the
> new
> > threading model.
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Sep 4, 2019 at 9:56 PM Ismael Juma <is...@juma.me.uk> wrote:
> >
> > > Hi Jun,
> > >
> > > I think it's important to discuss the pluggable API versus the calling
> > > implementation as two separate points.
> > >
> > > From an API perspective, are you suggesting that we should tell users
> > that
> > > they cannot block in authorize? Or are you saying that it's OK to block
> > on
> > > authorize on occasion and the recommendation would be for people to
> > > increase the number of threads in the request thread pool?
> > Architecturally,
> > > this feels wrong in my opinion.
> > >
> > > From the calling implementation perspective, you don't need a callback.
> > You
> > > can basically say:
> > >
> > > authorize(...).map { result =>
> > > ...
> > > }
> > >
> > > And then have a common method take that Future and handle it
> > synchronously
> > > if it's already complete or submit it to the purgatory if not. This is
> > > similar to how many modern async web libraries work.
> > >
> > > As Rajini said, this could be done later. The initial implementation
> > could
> > > simply do `toCompletableFuture().get()` to do it synchronously. But it
> > > would mean that we could add this functionality without changing the
> > > pluggable interface.
> > >
> > > Ismael
> > >
> > > On Wed, Sep 4, 2019 at 5:29 AM Jun Rao <ju...@confluent.io> wrote:
> > >
> > > > Hi, Rajini,
> > > >
> > > > Thanks for the KIP. I was concerned about #4 too. If we change the
> > > handling
> > > > of all requests to use an async authorize() api, will that cause the
> > code
> > > > much harder to understand? There are quite a few callbacks already. I
> > am
> > > > not sure that we want to introduce more of those. The benefit from
> > async
> > > > authorize() api seems limited.
> > > >
> > > > Jun
> > > >
> > > > On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <
> rajinisivaram@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Hi Don,
> > > > >
> > > > > Thanks for your note.
> > > > >
> > > > > 1) The intention is to avoid blocking in the calling thread. We
> > already
> > > > > have several requests that are put into a purgatory when waiting
> for
> > > > remote
> > > > > communication, for example produce request waiting for replication.
> > > Since
> > > > > we have a limited number of request threads in the broker, we want
> to
> > > > make
> > > > > progress with other requests, while one is waiting on any form of
> > > remote
> > > > > communication.
> > > > >
> > > > > 2) Async management calls will be useful with the default
> authorizer
> > > when
> > > > > KIP-500 removes ZK and we rely on Kafka instead. Our current
> ZK-based
> > > > > implementation as well as any custom implementations that don't
> want
> > to
> > > > be
> > > > > async will just need to return a sync'ed value. So instead of
> > > returning `
> > > > > value`, the code would just return
> > > > > `CompletableFuture.completedFuture(value)
> > > > > `. So it would be just a single line change in the implementation
> > with
> > > > the
> > > > > new API. The caller would treat completedFuture exactly as it does
> > > today,
> > > > > processing the request synchronously without using a purgatory.
> > > > >
> > > > > 3) For implementations that return a completedFuture as described
> in
> > > 2),
> > > > > the behaviour would remain exactly the same. No additional threads
> or
> > > > > purgatory will be used for this case. So there would be no
> > performance
> > > > > penalty. For implementations that return a future that is not
> > complete,
> > > > we
> > > > > prioritise running more requests concurrently. So in a deployment
> > with
> > > a
> > > > > large number of clients, we would improve performance by allowing
> > other
> > > > > requests to be processed on the request threads while some are
> > waiting
> > > > for
> > > > > authorization metadata.
> > > > >
> > > > > 4) I was concerned about this too. The goal is to make the API
> > flexible
> > > > > enough to handle large scale deployments in future when caching all
> > > > > authorization metadata in each broker is not viable. Using an async
> > API
> > > > > that returns CompletionStage, the caller has the option to handle
> the
> > > > > result synchronously or asynchronously, so we don't necessarily
> need
> > to
> > > > > update the calling code right away. Custom authorizers using the
> > async
> > > > API
> > > > > have full control over whether authorization is performed in-line
> > since
> > > > > completedFuture will always be handled synchronously. We do need to
> > > > update
> > > > > KafkaApis to take advantage of the asynchronous API to improve
> scale.
> > > > Even
> > > > > though this is a big change, since we will be doing the same for
> all
> > > > > requests, it shouldn't be too hard to maintain since the same
> pattern
> > > > will
> > > > > be used for all requests.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <bo...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Hi Rajini
> > > > > >
> > > > > > Help me understand this a bit more.
> > > > > >
> > > > > > 1. For all practical purpose, without authorization you can't go
> to
> > > the
> > > > > > next step. The calling code needs to block anyway. So should we
> > just
> > > > let
> > > > > > the implementation code do the async part?
> > > > > > 2. If you feel management calls need to be async, then we should
> > > > consider
> > > > > > another set of async APIs. I don't feel we should complicate it
> > > > further (
> > > > > > 3. Another concern I have is wrt performance. Kafka has been
> built
> > to
> > > > > > handle 1000s per second requests. Not sure whether making it
> async
> > > will
> > > > > add
> > > > > > any unnecessary overhead.
> > > > > > 4. How much complication would this add on the calling side? And
> is
> > > it
> > > > > > worth it?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Bosco
> > > > > >
> > > > > >
> > > > > > On 9/3/19, 8:50 AM, "Rajini Sivaram" <ra...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Ismael brought up a point that it will be good to make the
> > > > Authorizer
> > > > > > interface asynchronous to avoid blocking request threads
> during
> > > > > remote
> > > > > > operations.
> > > > > >
> > > > > > 1) Since we want to support different backends for
> > authorization
> > > > > > metadata,
> > > > > > making createAcls() and deleteAcls() asynchronous makes sense
> > > since
> > > > > > these
> > > > > > always involve remote operations. When KIP-500 removes
> > ZooKeeper,
> > > > we
> > > > > > would
> > > > > > want to move ACLs to Kafka and async updates will avoid
> > > unnecessary
> > > > > > blocking.
> > > > > > 2) For authorize() method, we currently use cached ACLs in
> the
> > > > > built-in
> > > > > > authorizers, so synchronous authorise operations work well
> now.
> > > But
> > > > > > async
> > > > > > authorize() would support this scenario as well as
> authorizers
> > in
> > > > > large
> > > > > > organisations where an LRU cache would enable a smaller cache
> > > even
> > > > > > when the
> > > > > > backend holds a large amount of ACLs for infrequently used
> > > > resources
> > > > > or
> > > > > > users who don't use the system frequently.
> > > > > >
> > > > > > For both cases, the built-in authorizer will continue to be
> > > > > > synchronous,
> > > > > > using CompletableFuture.completedFuture() to return the
> actual
> > > > > result.
> > > > > > But
> > > > > > async API will make custom authorizer implementations more
> > > > flexible.
> > > > > I
> > > > > > would like to know if there are any concerns with these
> changes
> > > > > before
> > > > > > updating the KIP.
> > > > > >
> > > > > > *Proposed API:*
> > > > > > public interface Authorizer extends Configurable, Closeable {
> > > > > >
> > > > > > Map<Endpoint, CompletionStage<Void>>
> > > start(AuthorizerServerInfo
> > > > > > serverInfo);
> > > > > > List<CompletionStage<AuthorizationResult>>
> > > > > > authorize(AuthorizableRequestContext requestContext,
> > List<Action>
> > > > > > actions);
> > > > > > List<CompletionStage<AclCreateResult>>
> > > > > > createAcls(AuthorizableRequestContext requestContext,
> > > > > List<AclBinding>
> > > > > > aclBindings);
> > > > > > List<CompletionStage<AclDeleteResult>>
> > > > > > deleteAcls(AuthorizableRequestContext requestContext,
> > > > > > List<AclBindingFilter> aclBindingFilters);
> > > > > > CompletionStage<Collection<AclBinding>>
> > acls(AclBindingFilter
> > > > > > filter);
> > > > > > }
> > > > > >
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <
> > > bosco@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Rajini
> > > > > > >
> > > > > > > Thanks for clarifying. I am good for now.
> > > > > > >
> > > > > > > Regards
> > > > > > >
> > > > > > > Bosco
> > > > > > >
> > > > > > >
> > > > > > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <
> > > rajinisivaram@gmail.com
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Don,
> > > > > > >
> > > > > > > That should be fine. I guess Ranger loads policies from
> > the
> > > > > > database
> > > > > > > synchronously when the authorizer is configured,
> similar
> > to
> > > > > > > SimpleAclAuthorizer loading from ZooKeeper. Ranger can
> > > > continue
> > > > > > to load
> > > > > > > synchronously from `configure()` or `start()` and
> return
> > an
> > > > > > empty map
> > > > > > > from
> > > > > > > `start()`. That would retain the existing behaviour..
> > When
> > > > the
> > > > > > same
> > > > > > > database stores policies for all listeners and the
> > policies
> > > > are
> > > > > > not
> > > > > > > stored
> > > > > > > in Kafka, there is no value in making the load
> > > asynchronous.
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai <
> > > > > > bosco@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Rajini
> > > > > > > >
> > > > > > > > Assuming this doesn't affect custom plugins, I don't
> > have
> > > > any
> > > > > > > concerns
> > > > > > > > regarding this.
> > > > > > > >
> > > > > > > > I do have one question regarding:
> > > > > > > >
> > > > > > > > "For authorizers that don’t store metadata in
> > ZooKeeper,
> > > > > > ensure that
> > > > > > > > authorizer metadata for each listener is available
> > before
> > > > > > starting
> > > > > > > up the
> > > > > > > > listener. This enables different authorization
> metadata
> > > > > stores
> > > > > > for
> > > > > > > > different listeners."
> > > > > > > >
> > > > > > > > Since Ranger uses its own database for storing
> > policies,
> > > do
> > > > > you
> > > > > > > anticipate
> > > > > > > > any issues?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > Bosco
> > > > > > > >
> > > > > > > >
> > > > > > > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
> > > > > > rajinisivaram@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I made another change to the KIP. The KIP was
> > > > originally
> > > > > > > proposing to
> > > > > > > > extend SimpleAclAuthorizer to also implement the
> > new
> > > > API
> > > > > > (in
> > > > > > > addition
> > > > > > > > to
> > > > > > > > the existing API). But since we use the new API
> > when
> > > > > > available,
> > > > > > > this
> > > > > > > > can
> > > > > > > > break custom authorizers that extend this class
> and
> > > > > > override
> > > > > > > specific
> > > > > > > > methods of the old API. To avoid breaking any
> > > existing
> > > > > > custom
> > > > > > > > implementations that extend this class,
> > particularly
> > > > > > because it
> > > > > > > is in
> > > > > > > > the
> > > > > > > > public package kafka.security.auth, the KIP now
> > > > proposes
> > > > > to
> > > > > > > retain the
> > > > > > > > old
> > > > > > > > authorizer as-is. SimpleAclAuthorizer will
> > continue
> > > to
> > > > > > > implement the
> > > > > > > > old
> > > > > > > > API, but will be deprecated. A new authorizer
> > > > > > implementation
> > > > > > > > kafka.security.authorizer.AclAuthorizer will be
> > added
> > > > for
> > > > > > the
> > > > > > > new API
> > > > > > > > (this
> > > > > > > > will not be in the public package).
> > > > > > > >
> > > > > > > > Please let me know if you have any concerns.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Rajini
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
> > > > > > > > rajinisivaram@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Thanks Colin.
> > > > > > > > >
> > > > > > > > > If there are no other concerns, I will start
> vote
> > > > later
> > > > > > today.
> > > > > > > Many
> > > > > > > > thanks
> > > > > > > > > to every one for the feedback.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > >
> > > > > > > > > Rajini
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <
> > > > > > > cmccabe@apache.org>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Thanks, Rajini. It looks good to me.
> > > > > > > > >>
> > > > > > > > >> best,
> > > > > > > > >> Colin
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram
> > > > wrote:
> > > > > > > > >> > Hi Colin,
> > > > > > > > >> >
> > > > > > > > >> > Thanks for the review. I have updated the
> KIP
> > to
> > > > > move
> > > > > > the
> > > > > > > > interfaces for
> > > > > > > > >> > request context and server info to the
> > > authorizer
> > > > > > package.
> > > > > > > These
> > > > > > > > are now
> > > > > > > > >> > called AuthorizableRequestContext and
> > > > > > AuthorizerServerInfo.
> > > > > > > > Endpoint is
> > > > > > > > >> now
> > > > > > > > >> > a class in org.apache.kafka.common to make
> it
> > > > > reusable
> > > > > > > since we
> > > > > > > > already
> > > > > > > > >> > have multiple implementations of it. I have
> > > > removed
> > > > > > > requestName
> > > > > > > > from the
> > > > > > > > >> > request context interface since authorizers
> > can
> > > > > > distinguish
> > > > > > > > follower
> > > > > > > > >> fetch
> > > > > > > > >> > and consumer fetch from the operation being
> > > > > > authorized. So
> > > > > > > 16-bit
> > > > > > > > >> request
> > > > > > > > >> > type should be sufficient for audit logging.
> > > Also
> > > > > > replaced
> > > > > > > > AuditFlag
> > > > > > > > >> with
> > > > > > > > >> > two booleans as you suggested.
> > > > > > > > >> >
> > > > > > > > >> > Can you take another look and see if the KIP
> > is
> > > > > ready
> > > > > > for
> > > > > > > voting?
> > > > > > > > >> >
> > > > > > > > >> > Thanks for all your help!
> > > > > > > > >> >
> > > > > > > > >> > Regards,
> > > > > > > > >> >
> > > > > > > > >> > Rajini
> > > > > > > > >> >
> > > > > > > > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin
> McCabe <
> > > > > > > cmccabe@apache.org>
> > > > > > > > >> wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Hi Rajini,
> > > > > > > > >> > >
> > > > > > > > >> > > I think it would be good to rename
> > > > > > KafkaRequestContext to
> > > > > > > > something
> > > > > > > > >> like
> > > > > > > > >> > > AuthorizableRequestContext, and put it in
> > the
> > > > > > > > >> > > org.apache.kafka.server.authorizer
> > namespace.
> > > > If
> > > > > > we put
> > > > > > > it in
> > > > > > > > the
> > > > > > > > >> > > org.apache.kafka.common namespace, then
> it's
> > > not
> > > > > > really
> > > > > > > clear
> > > > > > > > that
> > > > > > > > >> it's
> > > > > > > > >> > > part of the Authorizer API. Since this
> > class
> > > is
> > > > > > purely an
> > > > > > > > interface,
> > > > > > > > >> with
> > > > > > > > >> > > no concrete implementation of anything,
> > > there's
> > > > > > nothing
> > > > > > > common
> > > > > > > > to
> > > > > > > > >> really
> > > > > > > > >> > > reuse in any case. We definitely don't
> want
> > > > > > someone to
> > > > > > > > accidentally
> > > > > > > > >> add or
> > > > > > > > >> > > remove methods because they think this is
> > just
> > > > > > another
> > > > > > > internal
> > > > > > > > class
> > > > > > > > >> used
> > > > > > > > >> > > for requests.
> > > > > > > > >> > >
> > > > > > > > >> > > The BrokerInfo class is a nice
> improvement.
> > > It
> > > > > > looks
> > > > > > > like it
> > > > > > > > will be
> > > > > > > > >> > > useful for passing in information about
> the
> > > > > context
> > > > > > we're
> > > > > > > > running
> > > > > > > > >> in. It
> > > > > > > > >> > > would be nice to call this class
> ServerInfo
> > > > rather
> > > > > > than
> > > > > > > > BrokerInfo,
> > > > > > > > >> since
> > > > > > > > >> > > we will want to run the authorizer on
> > > > controllers
> > > > > > as well
> > > > > > > as on
> > > > > > > > >> brokers,
> > > > > > > > >> > > and the controller may run as a separate
> > > process
> > > > > > post
> > > > > > > KIP-500.
> > > > > > > > I also
> > > > > > > > >> > > think that this class should be in the
> > > > > > > > >> org.apache.kafka.server.authorizer
> > > > > > > > >> > > namespace. Again, it is an interface,
> not a
> > > > > > concrete
> > > > > > > > implementation,
> > > > > > > > >> and
> > > > > > > > >> > > it's an interface that is very
> specifically
> > > for
> > > > > the
> > > > > > > authorizer.
> > > > > > > > >> > >
> > > > > > > > >> > > I agree that we probably don't have enough
> > > > > > information
> > > > > > > > preserved for
> > > > > > > > >> > > requests currently to always know what
> > entity
> > > > made
> > > > > > them.
> > > > > > > So we
> > > > > > > > can
> > > > > > > > >> leave
> > > > > > > > >> > > that out for now (except in the special
> case
> > > of
> > > > > > Fetch).
> > > > > > > > Perhaps we
> > > > > > > > >> can add
> > > > > > > > >> > > this later if it's needed.
> > > > > > > > >> > >
> > > > > > > > >> > > I understand the intention behind
> > > > > AuthorizationMode
> > > > > > > (which is
> > > > > > > > now
> > > > > > > > >> called
> > > > > > > > >> > > AuditFlag in the latest revision). But it
> > > still
> > > > > > feels
> > > > > > > > complex. What
> > > > > > > > >> if we
> > > > > > > > >> > > just had two booleans in Action:
> > logSuccesses
> > > > and
> > > > > > > logFailures?
> > > > > > > > That
> > > > > > > > >> seems
> > > > > > > > >> > > to cover all the cases here.
> > > > MANDATORY_AUTHORIZE
> > > > > =
> > > > > > true,
> > > > > > > true.
> > > > > > > > >> > > OPTIONAL_AUTHORIZE = true, false. FILTER
> =
> > > > true,
> > > > > > false.
> > > > > > > > >> LIST_AUTHORIZED =
> > > > > > > > >> > > false, false. Would there be anything
> lost
> > > > versus
> > > > > > having
> > > > > > > the
> > > > > > > > enum?
> > > > > > > > >> > >
> > > > > > > > >> > > best,
> > > > > > > > >> > > Colin
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > On Wed, Aug 14, 2019, at 06:29, Mickael
> > Maison
> > > > > > wrote:
> > > > > > > > >> > > > Hi Rajini,
> > > > > > > > >> > > >
> > > > > > > > >> > > > Thanks for the KIP!
> > > > > > > > >> > > > I really like that authorize() will be
> > able
> > > to
> > > > > > take a
> > > > > > > batch of
> > > > > > > > >> > > > requests, this will speed up many
> > > > > implementations!
> > > > > > > > >> > > >
> > > > > > > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini
> > > > Sivaram <
> > > > > > > > >> rajinisivaram@gmail.com>
> > > > > > > > >> > > wrote:
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Thanks David! I have fixed the typo.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Also made a couple of changes to make
> > the
> > > > > > context
> > > > > > > > interfaces more
> > > > > > > > >> > > generic.
> > > > > > > > >> > > > > KafkaRequestContext now returns the
> > 16-bit
> > > > API
> > > > > > key as
> > > > > > > Colin
> > > > > > > > >> suggested
> > > > > > > > >> > > as
> > > > > > > > >> > > > > well as the friendly name used in
> > metrics
> > > > > which
> > > > > > are
> > > > > > > useful
> > > > > > > > in
> > > > > > > > >> audit
> > > > > > > > >> > > logs.
> > > > > > > > >> > > > > `Authorizer#start` is now provided a
> > > generic
> > > > > > > `BrokerInfo`
> > > > > > > > >> interface
> > > > > > > > >> > > that
> > > > > > > > >> > > > > gives cluster id, broker id and
> endpoint
> > > > > > information.
> > > > > > > The
> > > > > > > > generic
> > > > > > > > >> > > interface
> > > > > > > > >> > > > > can potentially be used in other
> broker
> > > > > plugins
> > > > > > in
> > > > > > > future
> > > > > > > > and
> > > > > > > > >> provides
> > > > > > > > >> > > > > dynamically generated configs like
> > broker
> > > id
> > > > > > and ports
> > > > > > > > which are
> > > > > > > > >> > > currently
> > > > > > > > >> > > > > not available to plugins unless these
> > > > configs
> > > > > > are
> > > > > > > statically
> > > > > > > > >> > > configured.
> > > > > > > > >> > > > > Please let me know if there are any
> > > > concerns.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Regards,
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Rajini
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > On Tue, Aug 13, 2019 at 4:30 PM David
> > > Jacot
> > > > <
> > > > > > > > djacot@confluent.io>
> > > > > > > > >> > > wrote:
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > Hi Rajini,
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Thank you for the update! It looks
> > good
> > > to
> > > > > me.
> > > > > > > There is a
> > > > > > > > typo
> > > > > > > > >> in the
> > > > > > > > >> > > > > > `AuditFlag` enum:
> > `MANDATORY_AUTHOEIZE`
> > > ->
> > > > > > > > >> `MANDATORY_AUTHORIZE`.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Regards,
> > > > > > > > >> > > > > > David
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > On Mon, Aug 12, 2019 at 2:54 PM
> Rajini
> > > > > > Sivaram <
> > > > > > > > >> > > rajinisivaram@gmail.com>
> > > > > > > > >> > > > > > wrote:
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > Hi David,
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Thanks for reviewing the KIP!
> Since
> > > > > > questions
> > > > > > > about
> > > > > > > > >> `authorization
> > > > > > > > >> > > mode`
> > > > > > > > >> > > > > > > and `count` have come up multiple
> > > > times, I
> > > > > > have
> > > > > > > renamed
> > > > > > > > both.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > 1) Renamed `count` to
> > > > > > `resourceReferenceCount`.
> > > > > > > It is
> > > > > > > > the
> > > > > > > > >> number
> > > > > > > > >> > > of times
> > > > > > > > >> > > > > > > the resource being authorized is
> > > > > referenced
> > > > > > > within the
> > > > > > > > >> request.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > 2) Renamed `AuthorizationMode` to
> > > > > > `AuditFlag`. It
> > > > > > > is
> > > > > > > > provided
> > > > > > > > >> to
> > > > > > > > >> > > improve
> > > > > > > > >> > > > > > > audit logging in the authorizer.
> The
> > > > enum
> > > > > > values
> > > > > > > have
> > > > > > > > javadoc
> > > > > > > > >> which
> > > > > > > > >> > > > > > > indicate how the authorization
> > result
> > > is
> > > > > > used in
> > > > > > > each
> > > > > > > > of the
> > > > > > > > >> modes
> > > > > > > > >> > > to
> > > > > > > > >> > > > > > > enable authorizers to log audit
> > > messages
> > > > > at
> > > > > > the
> > > > > > > right
> > > > > > > > severity
> > > > > > > > >> > > level.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Regards,
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Rajini
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM
> > David
> > > > > Jacot
> > > > > > <
> > > > > > > > >> djacot@confluent.io>
> > > > > > > > >> > > wrote:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > > Hi Rajini,
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > Thank you for the KIP. Overall,
> it
> > > > looks
> > > > > > good
> > > > > > > to me.
> > > > > > > > I have
> > > > > > > > >> few
> > > > > > > > >> > > > > > > > questions/suggestions:
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > 1. It is hard to grasp what
> > > > > > `Action#count` is
> > > > > > > for. I
> > > > > > > > guess I
> > > > > > > > >> > > understand
> > > > > > > > >> > > > > > > > where you want to go with it but
> > it
> > > > took
> > > > > > me a
> > > > > > > while to
> > > > > > > > >> figure it
> > > > > > > > >> > > out.
> > > > > > > > >> > > > > > > > Perhaps, we could come up with a
> > > > better
> > > > > > name
> > > > > > > than
> > > > > > > > `count`?
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > 2. I had a hard time trying to
> > > > > understand
> > > > > > the
> > > > > > > > >> `AuthorizationMode`
> > > > > > > > >> > > > > > > concept,
> > > > > > > > >> > > > > > > > especially wrt. the OPTIONAL
> one.
> > My
> > > > > > > understanding is
> > > > > > > > that
> > > > > > > > >> an
> > > > > > > > >> > > ACL is
> > > > > > > > >> > > > > > > either
> > > > > > > > >> > > > > > > > defined or not. Could you
> > elaborate
> > > a
> > > > > bit
> > > > > > more
> > > > > > > on
> > > > > > > > that?
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > Thanks,
> > > > > > > > >> > > > > > > > David
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM
> > Don
> > > > > Bosco
> > > > > > Durai
> > > > > > > <
> > > > > > > > >> > > bosco@apache.org>
> > > > > > > > >> > > > > > > wrote:
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > > Hi Rajini
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > 3.2 - This makes sense. Thanks
> > for
> > > > > > clarifying.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Rest looks fine. Once the
> > > > > > implementations are
> > > > > > > done,
> > > > > > > > it
> > > > > > > > >> will be
> > > > > > > > >> > > more
> > > > > > > > >> > > > > > > clear
> > > > > > > > >> > > > > > > > > on the different values
> > > RequestType
> > > > > and
> > > > > > Mode.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Thanks
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Bosco
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini
> > > > Sivaram"
> > > > > <
> > > > > > > > >> rajinisivaram@gmail.com
> > > > > > > > >> > > >
> > > > > > > > >> > > > > > wrote:
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Hi Don,
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Thanks for the
> suggestions.
> > A
> > > > few
> > > > > > > responses
> > > > > > > > below:
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > 3.1 Can rename and improve
> > > docs
> > > > if
> > > > > > we keep
> > > > > > > > this. Let's
> > > > > > > > >> > > finish the
> > > > > > > > >> > > > > > > > > discussion on Colin's
> > > > suggestions
> > > > > > > regarding this
> > > > > > > > >> first.
> > > > > > > > >> > > > > > > > > 3.2 No, I was thinking of
> > some
> > > > > > requests
> > > > > > > that
> > > > > > > > have an
> > > > > > > > >> old
> > > > > > > > >> > > way of
> > > > > > > > >> > > > > > > > > authorizing
> > > > > > > > >> > > > > > > > > and a new way where we
> have
> > > > > > retained the
> > > > > > > old
> > > > > > > > way for
> > > > > > > > >> > > backward
> > > > > > > > >> > > > > > > > > compatibility. One example
> > is
> > > > > > > Cluster:Create
> > > > > > > > >> permission to
> > > > > > > > >> > > create
> > > > > > > > >> > > > > > > > > topics.
> > > > > > > > >> > > > > > > > > We have replaced this with
> > > > > > fine-grained
> > > > > > > topic
> > > > > > > > create
> > > > > > > > >> > > access using
> > > > > > > > >> > > > > > > > > Topic:Create
> > > > > > > > >> > > > > > > > > for topic patterns. But we
> > > still
> > > > > > check if
> > > > > > > user
> > > > > > > > has
> > > > > > > > >> > > Cluster:Create
> > > > > > > > >> > > > > > > > > first. If
> > > > > > > > >> > > > > > > > > Denied, the deny is
> ignored
> > > and
> > > > we
> > > > > > check
> > > > > > > > >> Topic:Create. We
> > > > > > > > >> > > dont
> > > > > > > > >> > > > > > want
> > > > > > > > >> > > > > > > > to
> > > > > > > > >> > > > > > > > > log
> > > > > > > > >> > > > > > > > > DENY for Cluster:Create at
> > > INFO
> > > > > > level for
> > > > > > > this,
> > > > > > > > since
> > > > > > > > >> this
> > > > > > > > >> > > is
> > > > > > > > >> > > > > > not a
> > > > > > > > >> > > > > > > > > mandatory ACL for creating
> > > > topics.
> > > > > > We
> > > > > > > will get
> > > > > > > > >> appropriate
> > > > > > > > >> > > logs
> > > > > > > > >> > > > > > > from
> > > > > > > > >> > > > > > > > > the
> > > > > > > > >> > > > > > > > > subsequent Topic:Create in
> > > this
> > > > > > case.
> > > > > > > > >> > > > > > > > > 3.3 They are not quite the
> > > same.
> > > > > > FILTER
> > > > > > > implies
> > > > > > > > that
> > > > > > > > >> user
> > > > > > > > >> > > > > > actually
> > > > > > > > >> > > > > > > > > requested access to and
> > > > performed
> > > > > > some
> > > > > > > > operation on
> > > > > > > > >> the
> > > > > > > > >> > > filtered
> > > > > > > > >> > > > > > > > > resources.
> > > > > > > > >> > > > > > > > > LIST_AUTHORZED did not
> > result
> > > in
> > > > > any
> > > > > > > actual
> > > > > > > > access.
> > > > > > > > >> User
> > > > > > > > >> > > simply
> > > > > > > > >> > > > > > > > wanted
> > > > > > > > >> > > > > > > > > to
> > > > > > > > >> > > > > > > > > know what they are allowed
> > to
> > > > > > access.
> > > > > > > > >> > > > > > > > > 3.4 Request types are
> > Produce,
> > > > > > JoinGroup,
> > > > > > > > OffsetCommit
> > > > > > > > >> > > etc. So
> > > > > > > > >> > > > > > that
> > > > > > > > >> > > > > > > > is
> > > > > > > > >> > > > > > > > > different from
> authorization
> > > > mode,
> > > > > > > operation
> > > > > > > > etc.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > On Thu, Aug 8, 2019 at
> 11:36
> > > PM
> > > > > Don
> > > > > > Bosco
> > > > > > > Durai
> > > > > > > > <
> > > > > > > > >> > > > > > bosco@apache.org>
> > > > > > > > >> > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > Hi Rajini
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Thanks for clarifying.
> > This
> > > is
> > > > > > very
> > > > > > > helpful...
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > #1 - On the Ranger side,
> > we
> > > > > > should be
> > > > > > > able to
> > > > > > > > handle
> > > > > > > > >> > > multiple
> > > > > > > > >> > > > > > > > > requests at
> > > > > > > > >> > > > > > > > > > the same time. I was
> just
> > > not
> > > > > > sure how
> > > > > > > much
> > > > > > > > >> processing
> > > > > > > > >> > > overhead
> > > > > > > > >> > > > > > > > will
> > > > > > > > >> > > > > > > > > be
> > > > > > > > >> > > > > > > > > > there on the Broker side
> > to
> > > > > split
> > > > > > and
> > > > > > > then
> > > > > > > > >> consolidate
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > > results.
> > > > > > > > >> > > > > > > > > If it
> > > > > > > > >> > > > > > > > > > is negligible, then
> this
> > is
> > > > the
> > > > > > better
> > > > > > > way.
> > > > > > > > It will
> > > > > > > > >> > > make it
> > > > > > > > >> > > > > > > future
> > > > > > > > >> > > > > > > > > proof.
> > > > > > > > >> > > > > > > > > > #2 - I agree, having a
> > > single
> > > > > > "start"
> > > > > > > call
> > > > > > > > makes it
> > > > > > > > >> > > cleaner.
> > > > > > > > >> > > > > > The
> > > > > > > > >> > > > > > > > > > Authorization
> > implementation
> > > > > will
> > > > > > only
> > > > > > > have
> > > > > > > > to do
> > > > > > > > >> the
> > > > > > > > >> > > > > > > > initialization
> > > > > > > > >> > > > > > > > > only
> > > > > > > > >> > > > > > > > > > once.
> > > > > > > > >> > > > > > > > > > #3.1 - Thanks for the
> > > > > > clarification. I
> > > > > > > think
> > > > > > > > it
> > > > > > > > >> makes
> > > > > > > > >> > > sense to
> > > > > > > > >> > > > > > > have
> > > > > > > > >> > > > > > > > > this.
> > > > > > > > >> > > > > > > > > > The term "Mode" might
> not
> > be
> > > > > > explicit
> > > > > > > enough.
> > > > > > > > >> > > Essentially it
> > > > > > > > >> > > > > > > seems
> > > > > > > > >> > > > > > > > > you want
> > > > > > > > >> > > > > > > > > > the Authorizer to know
> the
> > > > > > > Intent/Purpose of
> > > > > > > > the
> > > > > > > > >> > > authorize call
> > > > > > > > >> > > > > > > and
> > > > > > > > >> > > > > > > > > let the
> > > > > > > > >> > > > > > > > > > Authorizer decide what
> to
> > > log
> > > > as
> > > > > > Audit
> > > > > > > event.
> > > > > > > > >> Changing
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > > > > class/field name
> > > > > > > > >> > > > > > > > > > or giving more
> > documentation
> > > > > will
> > > > > > do.
> > > > > > > > >> > > > > > > > > > #3.2 - Regarding the
> > option
> > > > > > "OPTIONAL".
> > > > > > > Are
> > > > > > > > you
> > > > > > > > >> thinking
> > > > > > > > >> > > from
> > > > > > > > >> > > > > > > > > chaining
> > > > > > > > >> > > > > > > > > > multiple Authorizer? If
> > > so, I
> > > > > am
> > > > > > not
> > > > > > > sure
> > > > > > > > whether
> > > > > > > > >> the
> > > > > > > > >> > > Broker
> > > > > > > > >> > > > > > > would
> > > > > > > > >> > > > > > > > > have
> > > > > > > > >> > > > > > > > > > enough information to
> make
> > > > that
> > > > > > > decision. I
> > > > > > > > feel the
> > > > > > > > >> > > Authorizer
> > > > > > > > >> > > > > > > > will
> > > > > > > > >> > > > > > > > > be the
> > > > > > > > >> > > > > > > > > > one who would have that
> > > > > > knowledge. E.g.
> > > > > > > in
> > > > > > > > Ranger
> > > > > > > > >> we have
> > > > > > > > >> > > > > > > explicit
> > > > > > > > >> > > > > > > > > deny,
> > > > > > > > >> > > > > > > > > > which means no matter
> > what,
> > > > the
> > > > > > request
> > > > > > > > should be
> > > > > > > > >> denied
> > > > > > > > >> > > for
> > > > > > > > >> > > > > > the
> > > > > > > > >> > > > > > > > > user/group
> > > > > > > > >> > > > > > > > > > or condition. So if you
> > are
> > > > > > thinking of
> > > > > > > > chaining
> > > > > > > > >> > > Authorizers,
> > > > > > > > >> > > > > > > then
> > > > > > > > >> > > > > > > > > > responses should have
> the
> > > > third
> > > > > > state,
> > > > > > > e.g.
> > > > > > > > >> > > "DENIED_FINAL", in
> > > > > > > > >> > > > > > > > which
> > > > > > > > >> > > > > > > > > case
> > > > > > > > >> > > > > > > > > > if there is an
> > Authorization
> > > > > > chain, it
> > > > > > > will
> > > > > > > > be stop
> > > > > > > > >> and
> > > > > > > > >> > > the
> > > > > > > > >> > > > > > > request
> > > > > > > > >> > > > > > > > > will be
> > > > > > > > >> > > > > > > > > > denied and if it is just
> > > > denied,
> > > > > > then
> > > > > > > you
> > > > > > > > might fall
> > > > > > > > >> > > back to
> > > > > > > > >> > > > > > next
> > > > > > > > >> > > > > > > > > > authorizer. If we don't
> > have
> > > > > > chaining of
> > > > > > > > >> Authorizing in
> > > > > > > > >> > > mind,
> > > > > > > > >> > > > > > > then
> > > > > > > > >> > > > > > > > we
> > > > > > > > >> > > > > > > > > > should reconsider
> OPTIONAL
> > > for
> > > > > > now. Or
> > > > > > > > clarify under
> > > > > > > > >> > > which
> > > > > > > > >> > > > > > > scenario
> > > > > > > > >> > > > > > > > > > OPTIONAL will be called.
> > > > > > > > >> > > > > > > > > > #3.3 Regarding, FILTER
> v/s
> > > > > > > LIST_AUTHORIZED,
> > > > > > > > isn't
> > > > > > > > >> > > > > > > LIST_AUTHORIZED a
> > > > > > > > >> > > > > > > > > > special case for "FILTER
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Jun,
For ACLs, the size is also dependent on the number of users. So in
deployments with large number of users where some users are not active, an
LRU cache may be useful. Agree that there may be other solutions that help
avoid the requirement for an async API even for this case.
I wasn't expecting our threading model to change very much. Having said
that, I haven't figured out how the purgatory can be adapted to work
efficiently for this. I was thinking:
1) If authorization future is complete when returning from authorize()
call, we invoke the method immediately on the request thread
2) If future is not complete, the request goes into a purgatory
3) When future completes, we mark the request ready to be run. This can be
done on ForkJoinPool.commonPool or an executor. But we don't handle the
request on that thread
4) We handle the request on the request thread after 3).
I totally agree that this adds complexity to the code. Will wait for Ismael
to comment on whether he had a different model in mind.
Thanks,
Rajini
On Thu, Sep 5, 2019 at 10:29 PM Jun Rao <ju...@confluent.io> wrote:
> Hi, Ismael,
>
> Yes, I agree that there are 2 separate discussion points, one on the API
> and the other on the implementation.
>
> First on the API, in general, my feeling is that the authorize() call is
> expected to be fast to reduce request latency. Adding an async API could be
> mis-leading since it might give users the impression that it would be
> arbitrarily long. In the short term, currently, we cache all partition
> level metadata on every broker. The authorization metadata is proportional
> to the number of topics, which typically is less in size. So it we can
> cache all partition level metadata, we should be able to cache all
> authorization metadata. Longer term, if we want to support more metadata
> beyond memory, another option is to put all metadata in an SSD backed
> key/value store. The lookup time could still be under 1ms, which we could
> potentially absorb in the request threads. So, I am not sure async
> authorize() api is strictly needed even in the long term.
>
> Second on the implementation. I was more concerned about the impact on our
> threading model. If we make authorize() async, some threads have to
> complete the future when it's ready. It seems to me that those threads have
> to be inside the Authorizer implementation since only it knows when to
> complete a future? If so, it feels weird that all the logic of request
> handling post authorize() is now handled by external threads inside a
> plugin. It's not clear how we can configure and monitor them, and how
> throttling would work. Perhaps we can provide a bit more detail on the new
> threading model.
>
> Thanks,
>
> Jun
>
> On Wed, Sep 4, 2019 at 9:56 PM Ismael Juma <is...@juma.me.uk> wrote:
>
> > Hi Jun,
> >
> > I think it's important to discuss the pluggable API versus the calling
> > implementation as two separate points.
> >
> > From an API perspective, are you suggesting that we should tell users
> that
> > they cannot block in authorize? Or are you saying that it's OK to block
> on
> > authorize on occasion and the recommendation would be for people to
> > increase the number of threads in the request thread pool?
> Architecturally,
> > this feels wrong in my opinion.
> >
> > From the calling implementation perspective, you don't need a callback.
> You
> > can basically say:
> >
> > authorize(...).map { result =>
> > ...
> > }
> >
> > And then have a common method take that Future and handle it
> synchronously
> > if it's already complete or submit it to the purgatory if not. This is
> > similar to how many modern async web libraries work.
> >
> > As Rajini said, this could be done later. The initial implementation
> could
> > simply do `toCompletableFuture().get()` to do it synchronously. But it
> > would mean that we could add this functionality without changing the
> > pluggable interface.
> >
> > Ismael
> >
> > On Wed, Sep 4, 2019 at 5:29 AM Jun Rao <ju...@confluent.io> wrote:
> >
> > > Hi, Rajini,
> > >
> > > Thanks for the KIP. I was concerned about #4 too. If we change the
> > handling
> > > of all requests to use an async authorize() api, will that cause the
> code
> > > much harder to understand? There are quite a few callbacks already. I
> am
> > > not sure that we want to introduce more of those. The benefit from
> async
> > > authorize() api seems limited.
> > >
> > > Jun
> > >
> > > On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <rajinisivaram@gmail.com
> >
> > > wrote:
> > >
> > > > Hi Don,
> > > >
> > > > Thanks for your note.
> > > >
> > > > 1) The intention is to avoid blocking in the calling thread. We
> already
> > > > have several requests that are put into a purgatory when waiting for
> > > remote
> > > > communication, for example produce request waiting for replication.
> > Since
> > > > we have a limited number of request threads in the broker, we want to
> > > make
> > > > progress with other requests, while one is waiting on any form of
> > remote
> > > > communication.
> > > >
> > > > 2) Async management calls will be useful with the default authorizer
> > when
> > > > KIP-500 removes ZK and we rely on Kafka instead. Our current ZK-based
> > > > implementation as well as any custom implementations that don't want
> to
> > > be
> > > > async will just need to return a sync'ed value. So instead of
> > returning `
> > > > value`, the code would just return
> > > > `CompletableFuture.completedFuture(value)
> > > > `. So it would be just a single line change in the implementation
> with
> > > the
> > > > new API. The caller would treat completedFuture exactly as it does
> > today,
> > > > processing the request synchronously without using a purgatory.
> > > >
> > > > 3) For implementations that return a completedFuture as described in
> > 2),
> > > > the behaviour would remain exactly the same. No additional threads or
> > > > purgatory will be used for this case. So there would be no
> performance
> > > > penalty. For implementations that return a future that is not
> complete,
> > > we
> > > > prioritise running more requests concurrently. So in a deployment
> with
> > a
> > > > large number of clients, we would improve performance by allowing
> other
> > > > requests to be processed on the request threads while some are
> waiting
> > > for
> > > > authorization metadata.
> > > >
> > > > 4) I was concerned about this too. The goal is to make the API
> flexible
> > > > enough to handle large scale deployments in future when caching all
> > > > authorization metadata in each broker is not viable. Using an async
> API
> > > > that returns CompletionStage, the caller has the option to handle the
> > > > result synchronously or asynchronously, so we don't necessarily need
> to
> > > > update the calling code right away. Custom authorizers using the
> async
> > > API
> > > > have full control over whether authorization is performed in-line
> since
> > > > completedFuture will always be handled synchronously. We do need to
> > > update
> > > > KafkaApis to take advantage of the asynchronous API to improve scale.
> > > Even
> > > > though this is a big change, since we will be doing the same for all
> > > > requests, it shouldn't be too hard to maintain since the same pattern
> > > will
> > > > be used for all requests.
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <bo...@apache.org>
> > > wrote:
> > > >
> > > > > Hi Rajini
> > > > >
> > > > > Help me understand this a bit more.
> > > > >
> > > > > 1. For all practical purpose, without authorization you can't go to
> > the
> > > > > next step. The calling code needs to block anyway. So should we
> just
> > > let
> > > > > the implementation code do the async part?
> > > > > 2. If you feel management calls need to be async, then we should
> > > consider
> > > > > another set of async APIs. I don't feel we should complicate it
> > > further (
> > > > > 3. Another concern I have is wrt performance. Kafka has been built
> to
> > > > > handle 1000s per second requests. Not sure whether making it async
> > will
> > > > add
> > > > > any unnecessary overhead.
> > > > > 4. How much complication would this add on the calling side? And is
> > it
> > > > > worth it?
> > > > >
> > > > > Thanks
> > > > >
> > > > > Bosco
> > > > >
> > > > >
> > > > > On 9/3/19, 8:50 AM, "Rajini Sivaram" <ra...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > Ismael brought up a point that it will be good to make the
> > > Authorizer
> > > > > interface asynchronous to avoid blocking request threads during
> > > > remote
> > > > > operations.
> > > > >
> > > > > 1) Since we want to support different backends for
> authorization
> > > > > metadata,
> > > > > making createAcls() and deleteAcls() asynchronous makes sense
> > since
> > > > > these
> > > > > always involve remote operations. When KIP-500 removes
> ZooKeeper,
> > > we
> > > > > would
> > > > > want to move ACLs to Kafka and async updates will avoid
> > unnecessary
> > > > > blocking.
> > > > > 2) For authorize() method, we currently use cached ACLs in the
> > > > built-in
> > > > > authorizers, so synchronous authorise operations work well now.
> > But
> > > > > async
> > > > > authorize() would support this scenario as well as authorizers
> in
> > > > large
> > > > > organisations where an LRU cache would enable a smaller cache
> > even
> > > > > when the
> > > > > backend holds a large amount of ACLs for infrequently used
> > > resources
> > > > or
> > > > > users who don't use the system frequently.
> > > > >
> > > > > For both cases, the built-in authorizer will continue to be
> > > > > synchronous,
> > > > > using CompletableFuture.completedFuture() to return the actual
> > > > result.
> > > > > But
> > > > > async API will make custom authorizer implementations more
> > > flexible.
> > > > I
> > > > > would like to know if there are any concerns with these changes
> > > > before
> > > > > updating the KIP.
> > > > >
> > > > > *Proposed API:*
> > > > > public interface Authorizer extends Configurable, Closeable {
> > > > >
> > > > > Map<Endpoint, CompletionStage<Void>>
> > start(AuthorizerServerInfo
> > > > > serverInfo);
> > > > > List<CompletionStage<AuthorizationResult>>
> > > > > authorize(AuthorizableRequestContext requestContext,
> List<Action>
> > > > > actions);
> > > > > List<CompletionStage<AclCreateResult>>
> > > > > createAcls(AuthorizableRequestContext requestContext,
> > > > List<AclBinding>
> > > > > aclBindings);
> > > > > List<CompletionStage<AclDeleteResult>>
> > > > > deleteAcls(AuthorizableRequestContext requestContext,
> > > > > List<AclBindingFilter> aclBindingFilters);
> > > > > CompletionStage<Collection<AclBinding>>
> acls(AclBindingFilter
> > > > > filter);
> > > > > }
> > > > >
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <
> > bosco@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Rajini
> > > > > >
> > > > > > Thanks for clarifying. I am good for now.
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > Bosco
> > > > > >
> > > > > >
> > > > > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <
> > rajinisivaram@gmail.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > > Hi Don,
> > > > > >
> > > > > > That should be fine. I guess Ranger loads policies from
> the
> > > > > database
> > > > > > synchronously when the authorizer is configured, similar
> to
> > > > > > SimpleAclAuthorizer loading from ZooKeeper. Ranger can
> > > continue
> > > > > to load
> > > > > > synchronously from `configure()` or `start()` and return
> an
> > > > > empty map
> > > > > > from
> > > > > > `start()`. That would retain the existing behaviour..
> When
> > > the
> > > > > same
> > > > > > database stores policies for all listeners and the
> policies
> > > are
> > > > > not
> > > > > > stored
> > > > > > in Kafka, there is no value in making the load
> > asynchronous.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > >
> > > > > > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai <
> > > > > bosco@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Rajini
> > > > > > >
> > > > > > > Assuming this doesn't affect custom plugins, I don't
> have
> > > any
> > > > > > concerns
> > > > > > > regarding this.
> > > > > > >
> > > > > > > I do have one question regarding:
> > > > > > >
> > > > > > > "For authorizers that don’t store metadata in
> ZooKeeper,
> > > > > ensure that
> > > > > > > authorizer metadata for each listener is available
> before
> > > > > starting
> > > > > > up the
> > > > > > > listener. This enables different authorization metadata
> > > > stores
> > > > > for
> > > > > > > different listeners."
> > > > > > >
> > > > > > > Since Ranger uses its own database for storing
> policies,
> > do
> > > > you
> > > > > > anticipate
> > > > > > > any issues?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > Bosco
> > > > > > >
> > > > > > >
> > > > > > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
> > > > > rajinisivaram@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I made another change to the KIP. The KIP was
> > > originally
> > > > > > proposing to
> > > > > > > extend SimpleAclAuthorizer to also implement the
> new
> > > API
> > > > > (in
> > > > > > addition
> > > > > > > to
> > > > > > > the existing API). But since we use the new API
> when
> > > > > available,
> > > > > > this
> > > > > > > can
> > > > > > > break custom authorizers that extend this class and
> > > > > override
> > > > > > specific
> > > > > > > methods of the old API. To avoid breaking any
> > existing
> > > > > custom
> > > > > > > implementations that extend this class,
> particularly
> > > > > because it
> > > > > > is in
> > > > > > > the
> > > > > > > public package kafka.security.auth, the KIP now
> > > proposes
> > > > to
> > > > > > retain the
> > > > > > > old
> > > > > > > authorizer as-is. SimpleAclAuthorizer will
> continue
> > to
> > > > > > implement the
> > > > > > > old
> > > > > > > API, but will be deprecated. A new authorizer
> > > > > implementation
> > > > > > > kafka.security.authorizer.AclAuthorizer will be
> added
> > > for
> > > > > the
> > > > > > new API
> > > > > > > (this
> > > > > > > will not be in the public package).
> > > > > > >
> > > > > > > Please let me know if you have any concerns.
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
> > > > > > > rajinisivaram@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks Colin.
> > > > > > > >
> > > > > > > > If there are no other concerns, I will start vote
> > > later
> > > > > today.
> > > > > > Many
> > > > > > > thanks
> > > > > > > > to every one for the feedback.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Rajini
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <
> > > > > > cmccabe@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Thanks, Rajini. It looks good to me.
> > > > > > > >>
> > > > > > > >> best,
> > > > > > > >> Colin
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram
> > > wrote:
> > > > > > > >> > Hi Colin,
> > > > > > > >> >
> > > > > > > >> > Thanks for the review. I have updated the KIP
> to
> > > > move
> > > > > the
> > > > > > > interfaces for
> > > > > > > >> > request context and server info to the
> > authorizer
> > > > > package.
> > > > > > These
> > > > > > > are now
> > > > > > > >> > called AuthorizableRequestContext and
> > > > > AuthorizerServerInfo.
> > > > > > > Endpoint is
> > > > > > > >> now
> > > > > > > >> > a class in org.apache.kafka.common to make it
> > > > reusable
> > > > > > since we
> > > > > > > already
> > > > > > > >> > have multiple implementations of it. I have
> > > removed
> > > > > > requestName
> > > > > > > from the
> > > > > > > >> > request context interface since authorizers
> can
> > > > > distinguish
> > > > > > > follower
> > > > > > > >> fetch
> > > > > > > >> > and consumer fetch from the operation being
> > > > > authorized. So
> > > > > > 16-bit
> > > > > > > >> request
> > > > > > > >> > type should be sufficient for audit logging.
> > Also
> > > > > replaced
> > > > > > > AuditFlag
> > > > > > > >> with
> > > > > > > >> > two booleans as you suggested.
> > > > > > > >> >
> > > > > > > >> > Can you take another look and see if the KIP
> is
> > > > ready
> > > > > for
> > > > > > voting?
> > > > > > > >> >
> > > > > > > >> > Thanks for all your help!
> > > > > > > >> >
> > > > > > > >> > Regards,
> > > > > > > >> >
> > > > > > > >> > Rajini
> > > > > > > >> >
> > > > > > > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <
> > > > > > cmccabe@apache.org>
> > > > > > > >> wrote:
> > > > > > > >> >
> > > > > > > >> > > Hi Rajini,
> > > > > > > >> > >
> > > > > > > >> > > I think it would be good to rename
> > > > > KafkaRequestContext to
> > > > > > > something
> > > > > > > >> like
> > > > > > > >> > > AuthorizableRequestContext, and put it in
> the
> > > > > > > >> > > org.apache.kafka.server.authorizer
> namespace.
> > > If
> > > > > we put
> > > > > > it in
> > > > > > > the
> > > > > > > >> > > org.apache.kafka.common namespace, then it's
> > not
> > > > > really
> > > > > > clear
> > > > > > > that
> > > > > > > >> it's
> > > > > > > >> > > part of the Authorizer API. Since this
> class
> > is
> > > > > purely an
> > > > > > > interface,
> > > > > > > >> with
> > > > > > > >> > > no concrete implementation of anything,
> > there's
> > > > > nothing
> > > > > > common
> > > > > > > to
> > > > > > > >> really
> > > > > > > >> > > reuse in any case. We definitely don't want
> > > > > someone to
> > > > > > > accidentally
> > > > > > > >> add or
> > > > > > > >> > > remove methods because they think this is
> just
> > > > > another
> > > > > > internal
> > > > > > > class
> > > > > > > >> used
> > > > > > > >> > > for requests.
> > > > > > > >> > >
> > > > > > > >> > > The BrokerInfo class is a nice improvement.
> > It
> > > > > looks
> > > > > > like it
> > > > > > > will be
> > > > > > > >> > > useful for passing in information about the
> > > > context
> > > > > we're
> > > > > > > running
> > > > > > > >> in. It
> > > > > > > >> > > would be nice to call this class ServerInfo
> > > rather
> > > > > than
> > > > > > > BrokerInfo,
> > > > > > > >> since
> > > > > > > >> > > we will want to run the authorizer on
> > > controllers
> > > > > as well
> > > > > > as on
> > > > > > > >> brokers,
> > > > > > > >> > > and the controller may run as a separate
> > process
> > > > > post
> > > > > > KIP-500.
> > > > > > > I also
> > > > > > > >> > > think that this class should be in the
> > > > > > > >> org.apache.kafka.server.authorizer
> > > > > > > >> > > namespace. Again, it is an interface, not a
> > > > > concrete
> > > > > > > implementation,
> > > > > > > >> and
> > > > > > > >> > > it's an interface that is very specifically
> > for
> > > > the
> > > > > > authorizer.
> > > > > > > >> > >
> > > > > > > >> > > I agree that we probably don't have enough
> > > > > information
> > > > > > > preserved for
> > > > > > > >> > > requests currently to always know what
> entity
> > > made
> > > > > them.
> > > > > > So we
> > > > > > > can
> > > > > > > >> leave
> > > > > > > >> > > that out for now (except in the special case
> > of
> > > > > Fetch).
> > > > > > > Perhaps we
> > > > > > > >> can add
> > > > > > > >> > > this later if it's needed.
> > > > > > > >> > >
> > > > > > > >> > > I understand the intention behind
> > > > AuthorizationMode
> > > > > > (which is
> > > > > > > now
> > > > > > > >> called
> > > > > > > >> > > AuditFlag in the latest revision). But it
> > still
> > > > > feels
> > > > > > > complex. What
> > > > > > > >> if we
> > > > > > > >> > > just had two booleans in Action:
> logSuccesses
> > > and
> > > > > > logFailures?
> > > > > > > That
> > > > > > > >> seems
> > > > > > > >> > > to cover all the cases here.
> > > MANDATORY_AUTHORIZE
> > > > =
> > > > > true,
> > > > > > true.
> > > > > > > >> > > OPTIONAL_AUTHORIZE = true, false. FILTER =
> > > true,
> > > > > false.
> > > > > > > >> LIST_AUTHORIZED =
> > > > > > > >> > > false, false. Would there be anything lost
> > > versus
> > > > > having
> > > > > > the
> > > > > > > enum?
> > > > > > > >> > >
> > > > > > > >> > > best,
> > > > > > > >> > > Colin
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > On Wed, Aug 14, 2019, at 06:29, Mickael
> Maison
> > > > > wrote:
> > > > > > > >> > > > Hi Rajini,
> > > > > > > >> > > >
> > > > > > > >> > > > Thanks for the KIP!
> > > > > > > >> > > > I really like that authorize() will be
> able
> > to
> > > > > take a
> > > > > > batch of
> > > > > > > >> > > > requests, this will speed up many
> > > > implementations!
> > > > > > > >> > > >
> > > > > > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini
> > > Sivaram <
> > > > > > > >> rajinisivaram@gmail.com>
> > > > > > > >> > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > Thanks David! I have fixed the typo.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Also made a couple of changes to make
> the
> > > > > context
> > > > > > > interfaces more
> > > > > > > >> > > generic.
> > > > > > > >> > > > > KafkaRequestContext now returns the
> 16-bit
> > > API
> > > > > key as
> > > > > > Colin
> > > > > > > >> suggested
> > > > > > > >> > > as
> > > > > > > >> > > > > well as the friendly name used in
> metrics
> > > > which
> > > > > are
> > > > > > useful
> > > > > > > in
> > > > > > > >> audit
> > > > > > > >> > > logs.
> > > > > > > >> > > > > `Authorizer#start` is now provided a
> > generic
> > > > > > `BrokerInfo`
> > > > > > > >> interface
> > > > > > > >> > > that
> > > > > > > >> > > > > gives cluster id, broker id and endpoint
> > > > > information.
> > > > > > The
> > > > > > > generic
> > > > > > > >> > > interface
> > > > > > > >> > > > > can potentially be used in other broker
> > > > plugins
> > > > > in
> > > > > > future
> > > > > > > and
> > > > > > > >> provides
> > > > > > > >> > > > > dynamically generated configs like
> broker
> > id
> > > > > and ports
> > > > > > > which are
> > > > > > > >> > > currently
> > > > > > > >> > > > > not available to plugins unless these
> > > configs
> > > > > are
> > > > > > statically
> > > > > > > >> > > configured.
> > > > > > > >> > > > > Please let me know if there are any
> > > concerns.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Regards,
> > > > > > > >> > > > >
> > > > > > > >> > > > > Rajini
> > > > > > > >> > > > >
> > > > > > > >> > > > > On Tue, Aug 13, 2019 at 4:30 PM David
> > Jacot
> > > <
> > > > > > > djacot@confluent.io>
> > > > > > > >> > > wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > Hi Rajini,
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Thank you for the update! It looks
> good
> > to
> > > > me.
> > > > > > There is a
> > > > > > > typo
> > > > > > > >> in the
> > > > > > > >> > > > > > `AuditFlag` enum:
> `MANDATORY_AUTHOEIZE`
> > ->
> > > > > > > >> `MANDATORY_AUTHORIZE`.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Regards,
> > > > > > > >> > > > > > David
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini
> > > > > Sivaram <
> > > > > > > >> > > rajinisivaram@gmail.com>
> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > Hi David,
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Thanks for reviewing the KIP! Since
> > > > > questions
> > > > > > about
> > > > > > > >> `authorization
> > > > > > > >> > > mode`
> > > > > > > >> > > > > > > and `count` have come up multiple
> > > times, I
> > > > > have
> > > > > > renamed
> > > > > > > both.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > 1) Renamed `count` to
> > > > > `resourceReferenceCount`.
> > > > > > It is
> > > > > > > the
> > > > > > > >> number
> > > > > > > >> > > of times
> > > > > > > >> > > > > > > the resource being authorized is
> > > > referenced
> > > > > > within the
> > > > > > > >> request.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > 2) Renamed `AuthorizationMode` to
> > > > > `AuditFlag`. It
> > > > > > is
> > > > > > > provided
> > > > > > > >> to
> > > > > > > >> > > improve
> > > > > > > >> > > > > > > audit logging in the authorizer. The
> > > enum
> > > > > values
> > > > > > have
> > > > > > > javadoc
> > > > > > > >> which
> > > > > > > >> > > > > > > indicate how the authorization
> result
> > is
> > > > > used in
> > > > > > each
> > > > > > > of the
> > > > > > > >> modes
> > > > > > > >> > > to
> > > > > > > >> > > > > > > enable authorizers to log audit
> > messages
> > > > at
> > > > > the
> > > > > > right
> > > > > > > severity
> > > > > > > >> > > level.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Regards,
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Rajini
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM
> David
> > > > Jacot
> > > > > <
> > > > > > > >> djacot@confluent.io>
> > > > > > > >> > > wrote:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > Hi Rajini,
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Thank you for the KIP. Overall, it
> > > looks
> > > > > good
> > > > > > to me.
> > > > > > > I have
> > > > > > > >> few
> > > > > > > >> > > > > > > > questions/suggestions:
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > 1. It is hard to grasp what
> > > > > `Action#count` is
> > > > > > for. I
> > > > > > > guess I
> > > > > > > >> > > understand
> > > > > > > >> > > > > > > > where you want to go with it but
> it
> > > took
> > > > > me a
> > > > > > while to
> > > > > > > >> figure it
> > > > > > > >> > > out.
> > > > > > > >> > > > > > > > Perhaps, we could come up with a
> > > better
> > > > > name
> > > > > > than
> > > > > > > `count`?
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > 2. I had a hard time trying to
> > > > understand
> > > > > the
> > > > > > > >> `AuthorizationMode`
> > > > > > > >> > > > > > > concept,
> > > > > > > >> > > > > > > > especially wrt. the OPTIONAL one.
> My
> > > > > > understanding is
> > > > > > > that
> > > > > > > >> an
> > > > > > > >> > > ACL is
> > > > > > > >> > > > > > > either
> > > > > > > >> > > > > > > > defined or not. Could you
> elaborate
> > a
> > > > bit
> > > > > more
> > > > > > on
> > > > > > > that?
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > Thanks,
> > > > > > > >> > > > > > > > David
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM
> Don
> > > > Bosco
> > > > > Durai
> > > > > > <
> > > > > > > >> > > bosco@apache.org>
> > > > > > > >> > > > > > > wrote:
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > > Hi Rajini
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > 3.2 - This makes sense. Thanks
> for
> > > > > clarifying.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Rest looks fine. Once the
> > > > > implementations are
> > > > > > done,
> > > > > > > it
> > > > > > > >> will be
> > > > > > > >> > > more
> > > > > > > >> > > > > > > clear
> > > > > > > >> > > > > > > > > on the different values
> > RequestType
> > > > and
> > > > > Mode.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Thanks
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Bosco
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini
> > > Sivaram"
> > > > <
> > > > > > > >> rajinisivaram@gmail.com
> > > > > > > >> > > >
> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Hi Don,
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Thanks for the suggestions.
> A
> > > few
> > > > > > responses
> > > > > > > below:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > 3.1 Can rename and improve
> > docs
> > > if
> > > > > we keep
> > > > > > > this. Let's
> > > > > > > >> > > finish the
> > > > > > > >> > > > > > > > > discussion on Colin's
> > > suggestions
> > > > > > regarding this
> > > > > > > >> first.
> > > > > > > >> > > > > > > > > 3.2 No, I was thinking of
> some
> > > > > requests
> > > > > > that
> > > > > > > have an
> > > > > > > >> old
> > > > > > > >> > > way of
> > > > > > > >> > > > > > > > > authorizing
> > > > > > > >> > > > > > > > > and a new way where we have
> > > > > retained the
> > > > > > old
> > > > > > > way for
> > > > > > > >> > > backward
> > > > > > > >> > > > > > > > > compatibility. One example
> is
> > > > > > Cluster:Create
> > > > > > > >> permission to
> > > > > > > >> > > create
> > > > > > > >> > > > > > > > > topics.
> > > > > > > >> > > > > > > > > We have replaced this with
> > > > > fine-grained
> > > > > > topic
> > > > > > > create
> > > > > > > >> > > access using
> > > > > > > >> > > > > > > > > Topic:Create
> > > > > > > >> > > > > > > > > for topic patterns. But we
> > still
> > > > > check if
> > > > > > user
> > > > > > > has
> > > > > > > >> > > Cluster:Create
> > > > > > > >> > > > > > > > > first. If
> > > > > > > >> > > > > > > > > Denied, the deny is ignored
> > and
> > > we
> > > > > check
> > > > > > > >> Topic:Create. We
> > > > > > > >> > > dont
> > > > > > > >> > > > > > want
> > > > > > > >> > > > > > > > to
> > > > > > > >> > > > > > > > > log
> > > > > > > >> > > > > > > > > DENY for Cluster:Create at
> > INFO
> > > > > level for
> > > > > > this,
> > > > > > > since
> > > > > > > >> this
> > > > > > > >> > > is
> > > > > > > >> > > > > > not a
> > > > > > > >> > > > > > > > > mandatory ACL for creating
> > > topics.
> > > > > We
> > > > > > will get
> > > > > > > >> appropriate
> > > > > > > >> > > logs
> > > > > > > >> > > > > > > from
> > > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > > > > subsequent Topic:Create in
> > this
> > > > > case.
> > > > > > > >> > > > > > > > > 3.3 They are not quite the
> > same.
> > > > > FILTER
> > > > > > implies
> > > > > > > that
> > > > > > > >> user
> > > > > > > >> > > > > > actually
> > > > > > > >> > > > > > > > > requested access to and
> > > performed
> > > > > some
> > > > > > > operation on
> > > > > > > >> the
> > > > > > > >> > > filtered
> > > > > > > >> > > > > > > > > resources.
> > > > > > > >> > > > > > > > > LIST_AUTHORZED did not
> result
> > in
> > > > any
> > > > > > actual
> > > > > > > access.
> > > > > > > >> User
> > > > > > > >> > > simply
> > > > > > > >> > > > > > > > wanted
> > > > > > > >> > > > > > > > > to
> > > > > > > >> > > > > > > > > know what they are allowed
> to
> > > > > access.
> > > > > > > >> > > > > > > > > 3.4 Request types are
> Produce,
> > > > > JoinGroup,
> > > > > > > OffsetCommit
> > > > > > > >> > > etc. So
> > > > > > > >> > > > > > that
> > > > > > > >> > > > > > > > is
> > > > > > > >> > > > > > > > > different from authorization
> > > mode,
> > > > > > operation
> > > > > > > etc.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > On Thu, Aug 8, 2019 at 11:36
> > PM
> > > > Don
> > > > > Bosco
> > > > > > Durai
> > > > > > > <
> > > > > > > >> > > > > > bosco@apache.org>
> > > > > > > >> > > > > > > > > wrote:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > Hi Rajini
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Thanks for clarifying.
> This
> > is
> > > > > very
> > > > > > helpful...
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > #1 - On the Ranger side,
> we
> > > > > should be
> > > > > > able to
> > > > > > > handle
> > > > > > > >> > > multiple
> > > > > > > >> > > > > > > > > requests at
> > > > > > > >> > > > > > > > > > the same time. I was just
> > not
> > > > > sure how
> > > > > > much
> > > > > > > >> processing
> > > > > > > >> > > overhead
> > > > > > > >> > > > > > > > will
> > > > > > > >> > > > > > > > > be
> > > > > > > >> > > > > > > > > > there on the Broker side
> to
> > > > split
> > > > > and
> > > > > > then
> > > > > > > >> consolidate
> > > > > > > >> > > the
> > > > > > > >> > > > > > > results.
> > > > > > > >> > > > > > > > > If it
> > > > > > > >> > > > > > > > > > is negligible, then this
> is
> > > the
> > > > > better
> > > > > > way.
> > > > > > > It will
> > > > > > > >> > > make it
> > > > > > > >> > > > > > > future
> > > > > > > >> > > > > > > > > proof.
> > > > > > > >> > > > > > > > > > #2 - I agree, having a
> > single
> > > > > "start"
> > > > > > call
> > > > > > > makes it
> > > > > > > >> > > cleaner.
> > > > > > > >> > > > > > The
> > > > > > > >> > > > > > > > > > Authorization
> implementation
> > > > will
> > > > > only
> > > > > > have
> > > > > > > to do
> > > > > > > >> the
> > > > > > > >> > > > > > > > initialization
> > > > > > > >> > > > > > > > > only
> > > > > > > >> > > > > > > > > > once.
> > > > > > > >> > > > > > > > > > #3.1 - Thanks for the
> > > > > clarification. I
> > > > > > think
> > > > > > > it
> > > > > > > >> makes
> > > > > > > >> > > sense to
> > > > > > > >> > > > > > > have
> > > > > > > >> > > > > > > > > this.
> > > > > > > >> > > > > > > > > > The term "Mode" might not
> be
> > > > > explicit
> > > > > > enough.
> > > > > > > >> > > Essentially it
> > > > > > > >> > > > > > > seems
> > > > > > > >> > > > > > > > > you want
> > > > > > > >> > > > > > > > > > the Authorizer to know the
> > > > > > Intent/Purpose of
> > > > > > > the
> > > > > > > >> > > authorize call
> > > > > > > >> > > > > > > and
> > > > > > > >> > > > > > > > > let the
> > > > > > > >> > > > > > > > > > Authorizer decide what to
> > log
> > > as
> > > > > Audit
> > > > > > event.
> > > > > > > >> Changing
> > > > > > > >> > > the
> > > > > > > >> > > > > > > > > class/field name
> > > > > > > >> > > > > > > > > > or giving more
> documentation
> > > > will
> > > > > do.
> > > > > > > >> > > > > > > > > > #3.2 - Regarding the
> option
> > > > > "OPTIONAL".
> > > > > > Are
> > > > > > > you
> > > > > > > >> thinking
> > > > > > > >> > > from
> > > > > > > >> > > > > > > > > chaining
> > > > > > > >> > > > > > > > > > multiple Authorizer? If
> > so, I
> > > > am
> > > > > not
> > > > > > sure
> > > > > > > whether
> > > > > > > >> the
> > > > > > > >> > > Broker
> > > > > > > >> > > > > > > would
> > > > > > > >> > > > > > > > > have
> > > > > > > >> > > > > > > > > > enough information to make
> > > that
> > > > > > decision. I
> > > > > > > feel the
> > > > > > > >> > > Authorizer
> > > > > > > >> > > > > > > > will
> > > > > > > >> > > > > > > > > be the
> > > > > > > >> > > > > > > > > > one who would have that
> > > > > knowledge. E.g.
> > > > > > in
> > > > > > > Ranger
> > > > > > > >> we have
> > > > > > > >> > > > > > > explicit
> > > > > > > >> > > > > > > > > deny,
> > > > > > > >> > > > > > > > > > which means no matter
> what,
> > > the
> > > > > request
> > > > > > > should be
> > > > > > > >> denied
> > > > > > > >> > > for
> > > > > > > >> > > > > > the
> > > > > > > >> > > > > > > > > user/group
> > > > > > > >> > > > > > > > > > or condition. So if you
> are
> > > > > thinking of
> > > > > > > chaining
> > > > > > > >> > > Authorizers,
> > > > > > > >> > > > > > > then
> > > > > > > >> > > > > > > > > > responses should have the
> > > third
> > > > > state,
> > > > > > e.g.
> > > > > > > >> > > "DENIED_FINAL", in
> > > > > > > >> > > > > > > > which
> > > > > > > >> > > > > > > > > case
> > > > > > > >> > > > > > > > > > if there is an
> Authorization
> > > > > chain, it
> > > > > > will
> > > > > > > be stop
> > > > > > > >> and
> > > > > > > >> > > the
> > > > > > > >> > > > > > > request
> > > > > > > >> > > > > > > > > will be
> > > > > > > >> > > > > > > > > > denied and if it is just
> > > denied,
> > > > > then
> > > > > > you
> > > > > > > might fall
> > > > > > > >> > > back to
> > > > > > > >> > > > > > next
> > > > > > > >> > > > > > > > > > authorizer. If we don't
> have
> > > > > chaining of
> > > > > > > >> Authorizing in
> > > > > > > >> > > mind,
> > > > > > > >> > > > > > > then
> > > > > > > >> > > > > > > > we
> > > > > > > >> > > > > > > > > > should reconsider OPTIONAL
> > for
> > > > > now. Or
> > > > > > > clarify under
> > > > > > > >> > > which
> > > > > > > >> > > > > > > scenario
> > > > > > > >> > > > > > > > > > OPTIONAL will be called.
> > > > > > > >> > > > > > > > > > #3.3 Regarding, FILTER v/s
> > > > > > LIST_AUTHORIZED,
> > > > > > > isn't
> > > > > > > >> > > > > > > LIST_AUTHORIZED a
> > > > > > > >> > > > > > > > > > special case for "FILTER"?
> > > > > > > >> > > > > > > > > > #3.4 KafkaRequestContext.
> > > > > requestType()
> > > > > > v/s
> > > > > > > Action.
> > > > > > > >> > > > > > > > > authorizationMode. I
> > > > > > > >> > > > > > > > > > am not sure about the
> > overlap
> > > or
> > > > > > ambiguity.
> > > > > > > >> > > > > > > > > > #4 - Cool. This is good,
> it
> > > will
> > > > > be less
> > > > > > > stress on
> > > > > > > >> the
> > > > > > > >> > > > > > > Authorizer.
> > > > > > > >> > > > > > > > > Ranger
> > > > > > > >> > > > > > > > > > already supports the
> "count"
> > > > > concept
> > > > > > and also
> > > > > > > has
> > > > > > > >> > > batching
> > > > > > > >> > > > > > > > > capability to
> > > > > > > >> > > > > > > > > > aggregate similar requests
> > to
> > > > > reduce the
> > > > > > > number of
> > > > > > > >> audit
> > > > > > > >> > > logs
> > > > > > > >> > > > > > to
> > > > > > > >> > > > > > > > > write. We
> > > > > > > >> > > > > > > > > > should be able to pass
> this
> > > > > through.
> > > > > > > >> > > > > > > > > > #5 - Assuming if the
> object
> > > > > instance is
> > > > > > going
> > > > > > > out of
> > > > > > > >> > > scope, we
> > > > > > > >> > > > > > > > > should be
> > > > > > > >> > > > > > > > > > fine. Not a super
> important
> > > ask.
> > > > > Ranger
> > > > > > is
> > > > > > > already
> > > > > > > >> > > catching
> > > > > > > >> > > > > > KILL
> > > > > > > >> > > > > > > > > signal for
> > > > > > > >> > > > > > > > > > clean up.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Thanks again. These are
> good
> > > > > > enhancements.
> > > > > > > >> Appreciate
> > > > > > > >> > > your
> > > > > > > >> > > > > > > efforts
> > > > > > > >> > > > > > > > > here.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Bosco
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > On 8/8/19, 2:03 AM,
> "Rajini
> > > > > Sivaram" <
> > > > > > > >> > > rajinisivaram@gmail.com
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Hi Don,
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Thanks for reviewing
> the
> > > > KIP.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > 1. I had this
> originally
> > > as
> > > > a
> > > > > single
> > > > > > > Action, but
> > > > > > > >> > > thought it
> > > > > > > >> > > > > > > may
> > > > > > > >> > > > > > > > > be
> > > > > > > >> > > > > > > > > > useful
> > > > > > > >> > > > > > > > > > to support batched
> > > authorize
> > > > > calls
> > > > > > as
> > > > > > > well and
> > > > > > > >> keep
> > > > > > > >> > > it
> > > > > > > >> > > > > > > > > consistent with
> > > > > > > >> > > > > > > > > > other methods. Single
> > > > > requests can
> > > > > > contain
> > > > > > > >> multiple
> > > > > > > >> > > topics.
> > > > > > > >> > > > > > > For
> > > > > > > >> > > > > > > > > > example a
> > > > > > > >> > > > > > > > > > produce request can
> > > contain
> > > > > records
> > > > > > for
> > > > > > > several
> > > > > > > >> > > partitions
> > > > > > > >> > > > > > of
> > > > > > > >> > > > > > > > > different
> > > > > > > >> > > > > > > > > > topics. Broker could
> > > > > potentially
> > > > > > > authorize these
> > > > > > > >> > > together.
> > > > > > > >> > > > > > > For
> > > > > > > >> > > > > > > > > > SimpleAclAuthorizer,
> > > batched
> > > > > > authorize
> > > > > > > methods
> > > > > > > >> don't
> > > > > > > >> > > > > > provide
> > > > > > > >> > > > > > > > any
> > > > > > > >> > > > > > > > > > optimisation since
> > lookup
> > > is
> > > > > based
> > > > > > on
> > > > > > > resources
> > > > > > > >> > > followed by
> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > > > > > > matching
> > > > > > > >> > > > > > > > > > logic. But some
> > > authorizers
> > > > > may
> > > > > > manage
> > > > > > > ACLs by
> > > > > > > >> user
> > > > > > > >> > > > > > principal
> > > > > > > >> > > > > > > > > rather
> > > > > > > >> > > > > > > > > > than
> > > > > > > >> > > > > > > > > > resource and may be
> able
> > > to
> > > > > optimize
> > > > > > > batched
> > > > > > > >> > > requests. I am
> > > > > > > >> > > > > > > ok
> > > > > > > >> > > > > > > > > with
> > > > > > > >> > > > > > > > > > using
> > > > > > > >> > > > > > > > > > single Action if this
> is
> > > > > likely to
> > > > > > cause
> > > > > > > issues.
> > > > > > > >> > > > > > > > > > 2. If you have two
> > > > listeners,
> > > > > one
> > > > > > for
> > > > > > > >> inter-broker
> > > > > > > >> > > traffic
> > > > > > > >> > > > > > > and
> > > > > > > >> > > > > > > > > another
> > > > > > > >> > > > > > > > > > for
> > > > > > > >> > > > > > > > > > external clients,
> start
> > > > > method is
> > > > > > invoked
> > > > > > > twice,
> > > > > > > >> > > once for
> > > > > > > >> > > > > > > each
> > > > > > > >> > > > > > > > > > listener. On
> > > > > > > >> > > > > > > > > > second thought, that
> may
> > > be
> > > > > > confusing and
> > > > > > > a
> > > > > > > >> single
> > > > > > > >> > > start()
> > > > > > > >> > > > > > > > > invocation
> > > > > > > >> > > > > > > > > > that
> > > > > > > >> > > > > > > > > > provides all listener
> > > > > information
> > > > > > and
> > > > > > > returns
> > > > > > > >> > > multiple
> > > > > > > >> > > > > > > futures
> > > > > > > >> > > > > > > > > would be
> > > > > > > >> > > > > > > > > > better. Will update
> the
> > > KIP.
> > > > > > > >> > > > > > > > > > 3. A typical example
> is
> > a
> > > > > consumer
> > > > > > > subscribing
> > > > > > > >> to a
> > > > > > > >> > > regex
> > > > > > > >> > > > > > > > > pattern. We
> > > > > > > >> > > > > > > > > > request all topic
> > metadata
> > > > > from the
> > > > > > > broker in
> > > > > > > >> order
> > > > > > > >> > > to
> > > > > > > >> > > > > > decide
> > > > > > > >> > > > > > > > > whether
> > > > > > > >> > > > > > > > > > the
> > > > > > > >> > > > > > > > > > pattern matches,
> > expecting
> > > > to
> > > > > > receive a
> > > > > > > list of
> > > > > > > >> > > authorised
> > > > > > > >> > > > > > > > > topics. The
> > > > > > > >> > > > > > > > > > user
> > > > > > > >> > > > > > > > > > is not asking to
> > subscribe
> > > > to
> > > > > an
> > > > > > > unauthorized
> > > > > > > >> topic.
> > > > > > > >> > > If
> > > > > > > >> > > > > > there
> > > > > > > >> > > > > > > > > are 10000
> > > > > > > >> > > > > > > > > > topics in the cluster
> > and
> > > > the
> > > > > user
> > > > > > has
> > > > > > > access
> > > > > > > >> to 100
> > > > > > > >> > > of
> > > > > > > >> > > > > > them,
> > > > > > > >> > > > > > > > at
> > > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > > > > > moment
> > > > > > > >> > > > > > > > > > we log 9900 DENIED log
> > > > > entries at
> > > > > > INFO
> > > > > > > level in
> > > > > > > >> > > > > > > > > SimpleAclAuthorizer.
> > > > > > > >> > > > > > > > > > The
> > > > > > > >> > > > > > > > > > proposal is to
> authorize
> > > > this
> > > > > > request with
> > > > > > > >> > > > > > > > > AuthorizationMode.FILTER, so
> > > > > > > >> > > > > > > > > > that authorizers can
> log
> > > > > resources
> > > > > > that
> > > > > > > are
> > > > > > > >> filtered
> > > > > > > >> > > out at
> > > > > > > >> > > > > > > > > lower level
> > > > > > > >> > > > > > > > > > like DEBUG since this
> is
> > > not
> > > > > an
> > > > > > attempt to
> > > > > > > >> access
> > > > > > > >> > > > > > > unauthorized
> > > > > > > >> > > > > > > > > > resources.
> > > > > > > >> > > > > > > > > > Brokers already handle
> > > these
> > > > > > differently
> > > > > > > since
> > > > > > > >> no
> > > > > > > >> > > > > > > authorization
> > > > > > > >> > > > > > > > > error
> > > > > > > >> > > > > > > > > > is
> > > > > > > >> > > > > > > > > > returned to the client
> > in
> > > > > these
> > > > > > cases.
> > > > > > > Providing
> > > > > > > >> > > > > > > authorization
> > > > > > > >> > > > > > > > > mode to
> > > > > > > >> > > > > > > > > > authorizers enables
> > > > authorizer
> > > > > > > implementations
> > > > > > > >> to
> > > > > > > >> > > generate
> > > > > > > >> > > > > > > > > better audit
> > > > > > > >> > > > > > > > > > logs.
> > > > > > > >> > > > > > > > > > 4. Each request may
> > > contain
> > > > > multiple
> > > > > > > instances
> > > > > > > >> of
> > > > > > > >> > > the same
> > > > > > > >> > > > > > > > > authorizable
> > > > > > > >> > > > > > > > > > resource. For example
> a
> > > > > produce
> > > > > > request
> > > > > > > may
> > > > > > > >> contain
> > > > > > > >> > > records
> > > > > > > >> > > > > > > for
> > > > > > > >> > > > > > > > > 10
> > > > > > > >> > > > > > > > > > partitions of the same
> > > > topic.
> > > > > At the
> > > > > > > moment, we
> > > > > > > >> > > invoke
> > > > > > > >> > > > > > > > authorize
> > > > > > > >> > > > > > > > > > method 10
> > > > > > > >> > > > > > > > > > times. The proposal is
> > to
> > > > > invoke it
> > > > > > once
> > > > > > > with
> > > > > > > >> > > count=10. The
> > > > > > > >> > > > > > > > > count is
> > > > > > > >> > > > > > > > > > provided to authorizer
> > > just
> > > > > for
> > > > > > audit
> > > > > > > logging
> > > > > > > >> > > purposes.
> > > > > > > >> > > > > > > > > > 5. Authorizer
> implements
> > > > > Closeable,
> > > > > > so you
> > > > > > > >> could use
> > > > > > > >> > > > > > close()
> > > > > > > >> > > > > > > to
> > > > > > > >> > > > > > > > > flush
> > > > > > > >> > > > > > > > > > audits?
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > On Thu, Aug 8, 2019 at
> > > 7:01
> > > > > AM Don
> > > > > > Bosco
> > > > > > > Durai <
> > > > > > > >> > > > > > > > bosco@apache.org
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Rajini
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Thanks for putting
> > this
> > > > > together.
> > > > > > It is
> > > > > > > >> looking
> > > > > > > >> > > good. I
> > > > > > > >> > > > > > > have
> > > > > > > >> > > > > > > > > few
> > > > > > > >> > > > > > > > > > > questions...
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > 1.
> > > > List<AuthorizationResult>
> > > > > > > authorize(...,
> > > > > > > >> > > List<Action>
> > > > > > > >> > > > > > > > > actions).
> > > > > > > >> > > > > > > > > > Do you
> > > > > > > >> > > > > > > > > > > see a scenario where
> > the
> > > > > broker
> > > > > > will
> > > > > > > call
> > > > > > > >> > > authorize for
> > > > > > > >> > > > > > > > > multiple
> > > > > > > >> > > > > > > > > > topics at
> > > > > > > >> > > > > > > > > > > the same time? I can
> > > > > understand
> > > > > > that
> > > > > > > during
> > > > > > > >> > > > > > > creating/deleting
> > > > > > > >> > > > > > > > > ACLS,
> > > > > > > >> > > > > > > > > > > multiple permissions
> > for
> > > > > multiple
> > > > > > > resources
> > > > > > > >> might
> > > > > > > >> > > be
> > > > > > > >> > > > > > done.
> > > > > > > >> > > > > > > > For
> > > > > > > >> > > > > > > > > > authorize
> > > > > > > >> > > > > > > > > > > call, would this be
> a
> > > > case?
> > > > > And
> > > > > > does the
> > > > > > > >> Authorize
> > > > > > > >> > > > > > > > > implementation
> > > > > > > >> > > > > > > > > > will be
> > > > > > > >> > > > > > > > > > > able to do
> performance
> > > > > > optimization
> > > > > > > because of
> > > > > > > >> > > this? Or
> > > > > > > >> > > > > > > > should
> > > > > > > >> > > > > > > > > we
> > > > > > > >> > > > > > > > > > just keep
> > > > > > > >> > > > > > > > > > > it simple? I don't
> see
> > > it
> > > > > as an
> > > > > > issue
> > > > > > > from
> > > > > > > >> Apache
> > > > > > > >> > > Ranger
> > > > > > > >> > > > > > > > side,
> > > > > > > >> > > > > > > > > but
> > > > > > > >> > > > > > > > > > just
> > > > > > > >> > > > > > > > > > > checking to see
> > whether
> > > we
> > > > > need
> > > > > > to be
> > > > > > > aware of
> > > > > > > >> > > something.
> > > > > > > >> > > > > > > > > > > 2. Should I assume
> > that
> > > > the
> > > > > > > SecurityProtocol
> > > > > > > >> passed
> > > > > > > >> > > > > > during
> > > > > > > >> > > > > > > > > start and
> > > > > > > >> > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > one return by
> > > > > > > >> > > KafkaRequestContext.securityProtocol() will
> > > > > > > >> > > > > > > be
> > > > > > > >> > > > > > > > > the
> > > > > > > >> > > > > > > > > > same?
> > > > > > > >> > > > > > > > > > >
> > CompletableFuture<Void>
> > > > > > start(String
> > > > > > > >> listenerName,
> > > > > > > >> > > > > > > > > SecurityProtocol
> > > > > > > >> > > > > > > > > > > securityProtocol);
> > > > > > > >> > > > > > > > > > >
> > > > > > KafkaRequestContext.securityProtocol()
> > > > > > > >> > > > > > > > > > > 3. What is the
> purpose
> > > of
> > > > > > > AuthorizationMode?
> > > > > > > >> How
> > > > > > > >> > > does the
> > > > > > > >> > > > > > > > > broker
> > > > > > > >> > > > > > > > > > decide
> > > > > > > >> > > > > > > > > > > what mode to use
> when
> > > the
> > > > > > authorize()
> > > > > > > method
> > > > > > > >> is
> > > > > > > >> > > called?
> > > > > > > >> > > > > > > > > > > 4. Can we clarify
> > > "count"
> > > > in
> > > > > > Action a
> > > > > > > bit
> > > > > > > >> more?
> > > > > > > >> > > How is it
> > > > > > > >> > > > > > > > used?
> > > > > > > >> > > > > > > > > > > 5. Do you feel
> having
> > > > "stop"
> > > > > > along with
> > > > > > > >> "start" be
> > > > > > > >> > > > > > helpful?
> > > > > > > >> > > > > > > > > E.g. In
> > > > > > > >> > > > > > > > > > Ranger
> > > > > > > >> > > > > > > > > > > we try to optimize
> the
> > > > Audit
> > > > > > writing by
> > > > > > > >> caching
> > > > > > > >> > > the logs
> > > > > > > >> > > > > > > for
> > > > > > > >> > > > > > > > a
> > > > > > > >> > > > > > > > > fixed
> > > > > > > >> > > > > > > > > > > interval. But when
> the
> > > > > Broker
> > > > > > > terminates, we
> > > > > > > >> do a
> > > > > > > >> > > forced
> > > > > > > >> > > > > > > > flush.
> > > > > > > >> > > > > > > > > > Having an
> > > > > > > >> > > > > > > > > > > explicit "stop"
> might
> > > give
> > > > > us a
> > > > > > formal
> > > > > > > way to
> > > > > > > >> > > flush our
> > > > > > > >> > > > > > > > audits.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Thanks
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Bosco
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > On 8/7/19, 3:59 PM,
> > > > "Rajini
> > > > > > Sivaram" <
> > > > > > > >> > > > > > > > rajinisivaram@gmail.com
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Hi
> > > Ron/Harsha/Satish,
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Thanks for
> > reviewing
> > > > > the KIP!
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > We should
> perhaps
> > > have
> > > > > a wider
> > > > > > > discussion
> > > > > > > >> > > outside
> > > > > > > >> > > > > > this
> > > > > > > >> > > > > > > > KIP
> > > > > > > >> > > > > > > > > for
> > > > > > > >> > > > > > > > > > > refactoring
> > > > > > > >> > > > > > > > > > > clients so that
> > > others
> > > > > who
> > > > > > are not
> > > > > > > >> following
> > > > > > > >> > > this KIP
> > > > > > > >> > > > > > > > also
> > > > > > > >> > > > > > > > > > notice the
> > > > > > > >> > > > > > > > > > > discussion.
> > Satish,
> > > > > would you
> > > > > > like
> > > > > > > to
> > > > > > > >> start a
> > > > > > > >> > > > > > > discussion
> > > > > > > >> > > > > > > > > thread
> > > > > > > >> > > > > > > > > > on dev?
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Regards,
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Rajini
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > On Wed, Aug 7,
> > 2019
> > > at
> > > > > 6:21 PM
> > > > > > > Satish
> > > > > > > >> Duggana <
> > > > > > > >> > > > > > > > > > >
> > > satish.duggana@gmail.com>
> > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > I felt the
> same
> > > need
> > > > > when
> > > > > > we want
> > > > > > > to
> > > > > > > >> add a
> > > > > > > >> > > > > > pluggable
> > > > > > > >> > > > > > > > API
> > > > > > > >> > > > > > > > > for
> > > > > > > >> > > > > > > > > > core
> > > > > > > >> > > > > > > > > > > > server
> > > > functionality.
> > > > > This
> > > > > > does
> > > > > > > not
> > > > > > > >> need to
> > > > > > > >> > > be part
> > > > > > > >> > > > > > > of
> > > > > > > >> > > > > > > > > this
> > > > > > > >> > > > > > > > > > KIP, it
> > > > > > > >> > > > > > > > > > > > can be a
> > separate
> > > > > KIP. I can
> > > > > > > contribute
> > > > > > > >> those
> > > > > > > >> > > > > > > > refactoring
> > > > > > > >> > > > > > > > > > changes if
> > > > > > > >> > > > > > > > > > > > others are OK
> > with
> > > > > that.
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > It is better
> to
> > > > have a
> > > > > > structure
> > > > > > > like
> > > > > > > >> below.
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > kafka-common:
> > > > > > > >> > > > > > > > > > > > common classes
> > > which
> > > > > can be
> > > > > > used
> > > > > > > in any
> > > > > > > >> of
> > > > > > > >> > > the
> > > > > > > >> > > > > > other
> > > > > > > >> > > > > > > > > modules
> > > > > > > >> > > > > > > > > > in Kafka
> > > > > > > >> > > > > > > > > > > > like client,
> > > > > > Kafka-server-common
> > > > > > > and
> > > > > > > >> server
> > > > > > > >> > > etc.
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > kafka-client-common:
> > > > > > > >> > > > > > > > > > > > common classes
> > > which
> > > > > can be
> > > > > > used
> > > > > > > in the
> > > > > > > >> > > client
> > > > > > > >> > > > > > > module.
> > > > > > > >> > > > > > > > > This
> > > > > > > >> > > > > > > > > > can be
> > > > > > > >> > > > > > > > > > > > part of client
> > > > module
> > > > > > itself.
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > kafka-server-common:
> > > > > > > >> > > > > > > > > > > > classes
> required
> > > > only
> > > > > for
> > > > > > > kafka-server.
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > Thanks.
> > > > > > > >> > > > > > > > > > > > Satish.
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > On Wed, Aug 7,
> > > 2019
> > > > > at 9:28
> > > > > > PM
> > > > > > > Harsha
> > > > > > > >> > > Chintalapani
> > > > > > > >> > > > > > <
> > > > > > > >> > > > > > > > > > kafka@harsha.io>
> > > > > > > >> > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > Thanks for
> the
> > > KIP
> > > > > Rajini.
> > > > > > > >> > > > > > > > > > > > > Quick
> thought,
> > > it
> > > > > would
> > > > > > be good
> > > > > > > to
> > > > > > > >> have a
> > > > > > > >> > > common
> > > > > > > >> > > > > > > > module
> > > > > > > >> > > > > > > > > > outside of
> > > > > > > >> > > > > > > > > > > > clients
> > > > > > > >> > > > > > > > > > > > > that only
> > > applies
> > > > to
> > > > > > server side
> > > > > > > >> > > interfaces &
> > > > > > > >> > > > > > > > changes.
> > > > > > > >> > > > > > > > > It
> > > > > > > >> > > > > > > > > > looks
> > > > > > > >> > > > > > > > > > > like we
> > > > > > > >> > > > > > > > > > > > are
> > > > > > > >> > > > > > > > > > > > > increasingly
> > in
> > > > > favor of
> > > > > > using
> > > > > > > Java
> > > > > > > >> > > interface for
> > > > > > > >> > > > > > > > > pluggable
> > > > > > > >> > > > > > > > > > > modules on
> > > > > > > >> > > > > > > > > > > > the
> > > > > > > >> > > > > > > > > > > > > broker side.
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > Thanks,
> > > > > > > >> > > > > > > > > > > > > Harsha
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > On Tue, Aug
> > 06,
> > > > > 2019 at
> > > > > > 2:31 PM,
> > > > > > > >> Rajini
> > > > > > > >> > > Sivaram <
> > > > > > > >> > > > > > > > > > >
> > rajinisivaram@gmail.com
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > Hi all,
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > I have
> > > created a
> > > > > KIP to
> > > > > > > replace the
> > > > > > > >> Scala
> > > > > > > >> > > > > > > > Authorizer
> > > > > > > >> > > > > > > > > API
> > > > > > > >> > > > > > > > > > with a
> > > > > > > >> > > > > > > > > > > new
> > > > > > > >> > > > > > > > > > > > Java
> > > > > > > >> > > > > > > > > > > > > > API:
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > -
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > This is
> > > > > replacement for
> > > > > > KIP-50
> > > > > > > >> which was
> > > > > > > >> > > > > > accepted
> > > > > > > >> > > > > > > > > but never
> > > > > > > >> > > > > > > > > > > merged.
> > > > > > > >> > > > > > > > > > > > Apart
> > > > > > > >> > > > > > > > > > > > > > from
> moving
> > > to a
> > > > > Java
> > > > > > API
> > > > > > > consistent
> > > > > > > >> > > with other
> > > > > > > >> > > > > > > > > pluggable
> > > > > > > >> > > > > > > > > > > interfaces
> > > > > > > >> > > > > > > > > > > > in the
> > > > > > > >> > > > > > > > > > > > > > broker,
> > > KIP-504
> > > > > also
> > > > > > attempts
> > > > > > > to
> > > > > > > >> address
> > > > > > > >> > > known
> > > > > > > >> > > > > > > > > limitations
> > > > > > > >> > > > > > > > > > in the
> > > > > > > >> > > > > > > > > > > > > >
> authorizer.
> > If
> > > > > you have
> > > > > > come
> > > > > > > across
> > > > > > > >> other
> > > > > > > >> > > > > > > > > limitations that
> > > > > > > >> > > > > > > > > > you
> > > > > > > >> > > > > > > > > > > would
> > > > > > > >> > > > > > > > > > > > like
> > > > > > > >> > > > > > > > > > > > > > to see
> > > addressed
> > > > > in the
> > > > > > new
> > > > > > > API,
> > > > > > > >> please
> > > > > > > >> > > raise
> > > > > > > >> > > > > > > these
> > > > > > > >> > > > > > > > > on the
> > > > > > > >> > > > > > > > > > > discussion
> > > > > > > >> > > > > > > > > > > > > > thread so
> > that
> > > > we
> > > > > can
> > > > > > > consider those
> > > > > > > >> > > too. All
> > > > > > > >> > > > > > > > > suggestions
> > > > > > > >> > > > > > > > > > and
> > > > > > > >> > > > > > > > > > > feedback
> > > > > > > >> > > > > > > > > > > > are
> > > > > > > >> > > > > > > > > > > > > > welcome.
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > Thank you,
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > > > Rajini
> > > > > > > >> > > > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Jun Rao <ju...@confluent.io>.
Hi, Ismael,
Yes, I agree that there are 2 separate discussion points, one on the API
and the other on the implementation.
First on the API, in general, my feeling is that the authorize() call is
expected to be fast to reduce request latency. Adding an async API could be
mis-leading since it might give users the impression that it would be
arbitrarily long. In the short term, currently, we cache all partition
level metadata on every broker. The authorization metadata is proportional
to the number of topics, which typically is less in size. So it we can
cache all partition level metadata, we should be able to cache all
authorization metadata. Longer term, if we want to support more metadata
beyond memory, another option is to put all metadata in an SSD backed
key/value store. The lookup time could still be under 1ms, which we could
potentially absorb in the request threads. So, I am not sure async
authorize() api is strictly needed even in the long term.
Second on the implementation. I was more concerned about the impact on our
threading model. If we make authorize() async, some threads have to
complete the future when it's ready. It seems to me that those threads have
to be inside the Authorizer implementation since only it knows when to
complete a future? If so, it feels weird that all the logic of request
handling post authorize() is now handled by external threads inside a
plugin. It's not clear how we can configure and monitor them, and how
throttling would work. Perhaps we can provide a bit more detail on the new
threading model.
Thanks,
Jun
On Wed, Sep 4, 2019 at 9:56 PM Ismael Juma <is...@juma.me.uk> wrote:
> Hi Jun,
>
> I think it's important to discuss the pluggable API versus the calling
> implementation as two separate points.
>
> From an API perspective, are you suggesting that we should tell users that
> they cannot block in authorize? Or are you saying that it's OK to block on
> authorize on occasion and the recommendation would be for people to
> increase the number of threads in the request thread pool? Architecturally,
> this feels wrong in my opinion.
>
> From the calling implementation perspective, you don't need a callback. You
> can basically say:
>
> authorize(...).map { result =>
> ...
> }
>
> And then have a common method take that Future and handle it synchronously
> if it's already complete or submit it to the purgatory if not. This is
> similar to how many modern async web libraries work.
>
> As Rajini said, this could be done later. The initial implementation could
> simply do `toCompletableFuture().get()` to do it synchronously. But it
> would mean that we could add this functionality without changing the
> pluggable interface.
>
> Ismael
>
> On Wed, Sep 4, 2019 at 5:29 AM Jun Rao <ju...@confluent.io> wrote:
>
> > Hi, Rajini,
> >
> > Thanks for the KIP. I was concerned about #4 too. If we change the
> handling
> > of all requests to use an async authorize() api, will that cause the code
> > much harder to understand? There are quite a few callbacks already. I am
> > not sure that we want to introduce more of those. The benefit from async
> > authorize() api seems limited.
> >
> > Jun
> >
> > On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <ra...@gmail.com>
> > wrote:
> >
> > > Hi Don,
> > >
> > > Thanks for your note.
> > >
> > > 1) The intention is to avoid blocking in the calling thread. We already
> > > have several requests that are put into a purgatory when waiting for
> > remote
> > > communication, for example produce request waiting for replication.
> Since
> > > we have a limited number of request threads in the broker, we want to
> > make
> > > progress with other requests, while one is waiting on any form of
> remote
> > > communication.
> > >
> > > 2) Async management calls will be useful with the default authorizer
> when
> > > KIP-500 removes ZK and we rely on Kafka instead. Our current ZK-based
> > > implementation as well as any custom implementations that don't want to
> > be
> > > async will just need to return a sync'ed value. So instead of
> returning `
> > > value`, the code would just return
> > > `CompletableFuture.completedFuture(value)
> > > `. So it would be just a single line change in the implementation with
> > the
> > > new API. The caller would treat completedFuture exactly as it does
> today,
> > > processing the request synchronously without using a purgatory.
> > >
> > > 3) For implementations that return a completedFuture as described in
> 2),
> > > the behaviour would remain exactly the same. No additional threads or
> > > purgatory will be used for this case. So there would be no performance
> > > penalty. For implementations that return a future that is not complete,
> > we
> > > prioritise running more requests concurrently. So in a deployment with
> a
> > > large number of clients, we would improve performance by allowing other
> > > requests to be processed on the request threads while some are waiting
> > for
> > > authorization metadata.
> > >
> > > 4) I was concerned about this too. The goal is to make the API flexible
> > > enough to handle large scale deployments in future when caching all
> > > authorization metadata in each broker is not viable. Using an async API
> > > that returns CompletionStage, the caller has the option to handle the
> > > result synchronously or asynchronously, so we don't necessarily need to
> > > update the calling code right away. Custom authorizers using the async
> > API
> > > have full control over whether authorization is performed in-line since
> > > completedFuture will always be handled synchronously. We do need to
> > update
> > > KafkaApis to take advantage of the asynchronous API to improve scale.
> > Even
> > > though this is a big change, since we will be doing the same for all
> > > requests, it shouldn't be too hard to maintain since the same pattern
> > will
> > > be used for all requests.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <bo...@apache.org>
> > wrote:
> > >
> > > > Hi Rajini
> > > >
> > > > Help me understand this a bit more.
> > > >
> > > > 1. For all practical purpose, without authorization you can't go to
> the
> > > > next step. The calling code needs to block anyway. So should we just
> > let
> > > > the implementation code do the async part?
> > > > 2. If you feel management calls need to be async, then we should
> > consider
> > > > another set of async APIs. I don't feel we should complicate it
> > further (
> > > > 3. Another concern I have is wrt performance. Kafka has been built to
> > > > handle 1000s per second requests. Not sure whether making it async
> will
> > > add
> > > > any unnecessary overhead.
> > > > 4. How much complication would this add on the calling side? And is
> it
> > > > worth it?
> > > >
> > > > Thanks
> > > >
> > > > Bosco
> > > >
> > > >
> > > > On 9/3/19, 8:50 AM, "Rajini Sivaram" <ra...@gmail.com>
> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Ismael brought up a point that it will be good to make the
> > Authorizer
> > > > interface asynchronous to avoid blocking request threads during
> > > remote
> > > > operations.
> > > >
> > > > 1) Since we want to support different backends for authorization
> > > > metadata,
> > > > making createAcls() and deleteAcls() asynchronous makes sense
> since
> > > > these
> > > > always involve remote operations. When KIP-500 removes ZooKeeper,
> > we
> > > > would
> > > > want to move ACLs to Kafka and async updates will avoid
> unnecessary
> > > > blocking.
> > > > 2) For authorize() method, we currently use cached ACLs in the
> > > built-in
> > > > authorizers, so synchronous authorise operations work well now.
> But
> > > > async
> > > > authorize() would support this scenario as well as authorizers in
> > > large
> > > > organisations where an LRU cache would enable a smaller cache
> even
> > > > when the
> > > > backend holds a large amount of ACLs for infrequently used
> > resources
> > > or
> > > > users who don't use the system frequently.
> > > >
> > > > For both cases, the built-in authorizer will continue to be
> > > > synchronous,
> > > > using CompletableFuture.completedFuture() to return the actual
> > > result.
> > > > But
> > > > async API will make custom authorizer implementations more
> > flexible.
> > > I
> > > > would like to know if there are any concerns with these changes
> > > before
> > > > updating the KIP.
> > > >
> > > > *Proposed API:*
> > > > public interface Authorizer extends Configurable, Closeable {
> > > >
> > > > Map<Endpoint, CompletionStage<Void>>
> start(AuthorizerServerInfo
> > > > serverInfo);
> > > > List<CompletionStage<AuthorizationResult>>
> > > > authorize(AuthorizableRequestContext requestContext, List<Action>
> > > > actions);
> > > > List<CompletionStage<AclCreateResult>>
> > > > createAcls(AuthorizableRequestContext requestContext,
> > > List<AclBinding>
> > > > aclBindings);
> > > > List<CompletionStage<AclDeleteResult>>
> > > > deleteAcls(AuthorizableRequestContext requestContext,
> > > > List<AclBindingFilter> aclBindingFilters);
> > > > CompletionStage<Collection<AclBinding>> acls(AclBindingFilter
> > > > filter);
> > > > }
> > > >
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> > > > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <
> bosco@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Rajini
> > > > >
> > > > > Thanks for clarifying. I am good for now.
> > > > >
> > > > > Regards
> > > > >
> > > > > Bosco
> > > > >
> > > > >
> > > > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <
> rajinisivaram@gmail.com
> > >
> > > > wrote:
> > > > >
> > > > > Hi Don,
> > > > >
> > > > > That should be fine. I guess Ranger loads policies from the
> > > > database
> > > > > synchronously when the authorizer is configured, similar to
> > > > > SimpleAclAuthorizer loading from ZooKeeper. Ranger can
> > continue
> > > > to load
> > > > > synchronously from `configure()` or `start()` and return an
> > > > empty map
> > > > > from
> > > > > `start()`. That would retain the existing behaviour.. When
> > the
> > > > same
> > > > > database stores policies for all listeners and the policies
> > are
> > > > not
> > > > > stored
> > > > > in Kafka, there is no value in making the load
> asynchronous.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > > >
> > > > > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai <
> > > > bosco@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Rajini
> > > > > >
> > > > > > Assuming this doesn't affect custom plugins, I don't have
> > any
> > > > > concerns
> > > > > > regarding this.
> > > > > >
> > > > > > I do have one question regarding:
> > > > > >
> > > > > > "For authorizers that don’t store metadata in ZooKeeper,
> > > > ensure that
> > > > > > authorizer metadata for each listener is available before
> > > > starting
> > > > > up the
> > > > > > listener. This enables different authorization metadata
> > > stores
> > > > for
> > > > > > different listeners."
> > > > > >
> > > > > > Since Ranger uses its own database for storing policies,
> do
> > > you
> > > > > anticipate
> > > > > > any issues?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Bosco
> > > > > >
> > > > > >
> > > > > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
> > > > rajinisivaram@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I made another change to the KIP. The KIP was
> > originally
> > > > > proposing to
> > > > > > extend SimpleAclAuthorizer to also implement the new
> > API
> > > > (in
> > > > > addition
> > > > > > to
> > > > > > the existing API). But since we use the new API when
> > > > available,
> > > > > this
> > > > > > can
> > > > > > break custom authorizers that extend this class and
> > > > override
> > > > > specific
> > > > > > methods of the old API. To avoid breaking any
> existing
> > > > custom
> > > > > > implementations that extend this class, particularly
> > > > because it
> > > > > is in
> > > > > > the
> > > > > > public package kafka.security.auth, the KIP now
> > proposes
> > > to
> > > > > retain the
> > > > > > old
> > > > > > authorizer as-is. SimpleAclAuthorizer will continue
> to
> > > > > implement the
> > > > > > old
> > > > > > API, but will be deprecated. A new authorizer
> > > > implementation
> > > > > > kafka.security.authorizer.AclAuthorizer will be added
> > for
> > > > the
> > > > > new API
> > > > > > (this
> > > > > > will not be in the public package).
> > > > > >
> > > > > > Please let me know if you have any concerns.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > >
> > > > > > On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
> > > > > > rajinisivaram@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks Colin.
> > > > > > >
> > > > > > > If there are no other concerns, I will start vote
> > later
> > > > today.
> > > > > Many
> > > > > > thanks
> > > > > > > to every one for the feedback.
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <
> > > > > cmccabe@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > >> Thanks, Rajini. It looks good to me.
> > > > > > >>
> > > > > > >> best,
> > > > > > >> Colin
> > > > > > >>
> > > > > > >>
> > > > > > >> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram
> > wrote:
> > > > > > >> > Hi Colin,
> > > > > > >> >
> > > > > > >> > Thanks for the review. I have updated the KIP to
> > > move
> > > > the
> > > > > > interfaces for
> > > > > > >> > request context and server info to the
> authorizer
> > > > package.
> > > > > These
> > > > > > are now
> > > > > > >> > called AuthorizableRequestContext and
> > > > AuthorizerServerInfo.
> > > > > > Endpoint is
> > > > > > >> now
> > > > > > >> > a class in org.apache.kafka.common to make it
> > > reusable
> > > > > since we
> > > > > > already
> > > > > > >> > have multiple implementations of it. I have
> > removed
> > > > > requestName
> > > > > > from the
> > > > > > >> > request context interface since authorizers can
> > > > distinguish
> > > > > > follower
> > > > > > >> fetch
> > > > > > >> > and consumer fetch from the operation being
> > > > authorized. So
> > > > > 16-bit
> > > > > > >> request
> > > > > > >> > type should be sufficient for audit logging.
> Also
> > > > replaced
> > > > > > AuditFlag
> > > > > > >> with
> > > > > > >> > two booleans as you suggested.
> > > > > > >> >
> > > > > > >> > Can you take another look and see if the KIP is
> > > ready
> > > > for
> > > > > voting?
> > > > > > >> >
> > > > > > >> > Thanks for all your help!
> > > > > > >> >
> > > > > > >> > Regards,
> > > > > > >> >
> > > > > > >> > Rajini
> > > > > > >> >
> > > > > > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <
> > > > > cmccabe@apache.org>
> > > > > > >> wrote:
> > > > > > >> >
> > > > > > >> > > Hi Rajini,
> > > > > > >> > >
> > > > > > >> > > I think it would be good to rename
> > > > KafkaRequestContext to
> > > > > > something
> > > > > > >> like
> > > > > > >> > > AuthorizableRequestContext, and put it in the
> > > > > > >> > > org.apache.kafka.server.authorizer namespace.
> > If
> > > > we put
> > > > > it in
> > > > > > the
> > > > > > >> > > org.apache.kafka.common namespace, then it's
> not
> > > > really
> > > > > clear
> > > > > > that
> > > > > > >> it's
> > > > > > >> > > part of the Authorizer API. Since this class
> is
> > > > purely an
> > > > > > interface,
> > > > > > >> with
> > > > > > >> > > no concrete implementation of anything,
> there's
> > > > nothing
> > > > > common
> > > > > > to
> > > > > > >> really
> > > > > > >> > > reuse in any case. We definitely don't want
> > > > someone to
> > > > > > accidentally
> > > > > > >> add or
> > > > > > >> > > remove methods because they think this is just
> > > > another
> > > > > internal
> > > > > > class
> > > > > > >> used
> > > > > > >> > > for requests.
> > > > > > >> > >
> > > > > > >> > > The BrokerInfo class is a nice improvement.
> It
> > > > looks
> > > > > like it
> > > > > > will be
> > > > > > >> > > useful for passing in information about the
> > > context
> > > > we're
> > > > > > running
> > > > > > >> in. It
> > > > > > >> > > would be nice to call this class ServerInfo
> > rather
> > > > than
> > > > > > BrokerInfo,
> > > > > > >> since
> > > > > > >> > > we will want to run the authorizer on
> > controllers
> > > > as well
> > > > > as on
> > > > > > >> brokers,
> > > > > > >> > > and the controller may run as a separate
> process
> > > > post
> > > > > KIP-500.
> > > > > > I also
> > > > > > >> > > think that this class should be in the
> > > > > > >> org.apache.kafka.server.authorizer
> > > > > > >> > > namespace. Again, it is an interface, not a
> > > > concrete
> > > > > > implementation,
> > > > > > >> and
> > > > > > >> > > it's an interface that is very specifically
> for
> > > the
> > > > > authorizer.
> > > > > > >> > >
> > > > > > >> > > I agree that we probably don't have enough
> > > > information
> > > > > > preserved for
> > > > > > >> > > requests currently to always know what entity
> > made
> > > > them.
> > > > > So we
> > > > > > can
> > > > > > >> leave
> > > > > > >> > > that out for now (except in the special case
> of
> > > > Fetch).
> > > > > > Perhaps we
> > > > > > >> can add
> > > > > > >> > > this later if it's needed.
> > > > > > >> > >
> > > > > > >> > > I understand the intention behind
> > > AuthorizationMode
> > > > > (which is
> > > > > > now
> > > > > > >> called
> > > > > > >> > > AuditFlag in the latest revision). But it
> still
> > > > feels
> > > > > > complex. What
> > > > > > >> if we
> > > > > > >> > > just had two booleans in Action: logSuccesses
> > and
> > > > > logFailures?
> > > > > > That
> > > > > > >> seems
> > > > > > >> > > to cover all the cases here.
> > MANDATORY_AUTHORIZE
> > > =
> > > > true,
> > > > > true.
> > > > > > >> > > OPTIONAL_AUTHORIZE = true, false. FILTER =
> > true,
> > > > false.
> > > > > > >> LIST_AUTHORIZED =
> > > > > > >> > > false, false. Would there be anything lost
> > versus
> > > > having
> > > > > the
> > > > > > enum?
> > > > > > >> > >
> > > > > > >> > > best,
> > > > > > >> > > Colin
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > On Wed, Aug 14, 2019, at 06:29, Mickael Maison
> > > > wrote:
> > > > > > >> > > > Hi Rajini,
> > > > > > >> > > >
> > > > > > >> > > > Thanks for the KIP!
> > > > > > >> > > > I really like that authorize() will be able
> to
> > > > take a
> > > > > batch of
> > > > > > >> > > > requests, this will speed up many
> > > implementations!
> > > > > > >> > > >
> > > > > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini
> > Sivaram <
> > > > > > >> rajinisivaram@gmail.com>
> > > > > > >> > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > Thanks David! I have fixed the typo.
> > > > > > >> > > > >
> > > > > > >> > > > > Also made a couple of changes to make the
> > > > context
> > > > > > interfaces more
> > > > > > >> > > generic.
> > > > > > >> > > > > KafkaRequestContext now returns the 16-bit
> > API
> > > > key as
> > > > > Colin
> > > > > > >> suggested
> > > > > > >> > > as
> > > > > > >> > > > > well as the friendly name used in metrics
> > > which
> > > > are
> > > > > useful
> > > > > > in
> > > > > > >> audit
> > > > > > >> > > logs.
> > > > > > >> > > > > `Authorizer#start` is now provided a
> generic
> > > > > `BrokerInfo`
> > > > > > >> interface
> > > > > > >> > > that
> > > > > > >> > > > > gives cluster id, broker id and endpoint
> > > > information.
> > > > > The
> > > > > > generic
> > > > > > >> > > interface
> > > > > > >> > > > > can potentially be used in other broker
> > > plugins
> > > > in
> > > > > future
> > > > > > and
> > > > > > >> provides
> > > > > > >> > > > > dynamically generated configs like broker
> id
> > > > and ports
> > > > > > which are
> > > > > > >> > > currently
> > > > > > >> > > > > not available to plugins unless these
> > configs
> > > > are
> > > > > statically
> > > > > > >> > > configured.
> > > > > > >> > > > > Please let me know if there are any
> > concerns.
> > > > > > >> > > > >
> > > > > > >> > > > > Regards,
> > > > > > >> > > > >
> > > > > > >> > > > > Rajini
> > > > > > >> > > > >
> > > > > > >> > > > > On Tue, Aug 13, 2019 at 4:30 PM David
> Jacot
> > <
> > > > > > djacot@confluent.io>
> > > > > > >> > > wrote:
> > > > > > >> > > > >
> > > > > > >> > > > > > Hi Rajini,
> > > > > > >> > > > > >
> > > > > > >> > > > > > Thank you for the update! It looks good
> to
> > > me.
> > > > > There is a
> > > > > > typo
> > > > > > >> in the
> > > > > > >> > > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE`
> ->
> > > > > > >> `MANDATORY_AUTHORIZE`.
> > > > > > >> > > > > >
> > > > > > >> > > > > > Regards,
> > > > > > >> > > > > > David
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini
> > > > Sivaram <
> > > > > > >> > > rajinisivaram@gmail.com>
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Hi David,
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Thanks for reviewing the KIP! Since
> > > > questions
> > > > > about
> > > > > > >> `authorization
> > > > > > >> > > mode`
> > > > > > >> > > > > > > and `count` have come up multiple
> > times, I
> > > > have
> > > > > renamed
> > > > > > both.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > 1) Renamed `count` to
> > > > `resourceReferenceCount`.
> > > > > It is
> > > > > > the
> > > > > > >> number
> > > > > > >> > > of times
> > > > > > >> > > > > > > the resource being authorized is
> > > referenced
> > > > > within the
> > > > > > >> request.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > 2) Renamed `AuthorizationMode` to
> > > > `AuditFlag`. It
> > > > > is
> > > > > > provided
> > > > > > >> to
> > > > > > >> > > improve
> > > > > > >> > > > > > > audit logging in the authorizer. The
> > enum
> > > > values
> > > > > have
> > > > > > javadoc
> > > > > > >> which
> > > > > > >> > > > > > > indicate how the authorization result
> is
> > > > used in
> > > > > each
> > > > > > of the
> > > > > > >> modes
> > > > > > >> > > to
> > > > > > >> > > > > > > enable authorizers to log audit
> messages
> > > at
> > > > the
> > > > > right
> > > > > > severity
> > > > > > >> > > level.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Regards,
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Rajini
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM David
> > > Jacot
> > > > <
> > > > > > >> djacot@confluent.io>
> > > > > > >> > > wrote:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > Hi Rajini,
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Thank you for the KIP. Overall, it
> > looks
> > > > good
> > > > > to me.
> > > > > > I have
> > > > > > >> few
> > > > > > >> > > > > > > > questions/suggestions:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > 1. It is hard to grasp what
> > > > `Action#count` is
> > > > > for. I
> > > > > > guess I
> > > > > > >> > > understand
> > > > > > >> > > > > > > > where you want to go with it but it
> > took
> > > > me a
> > > > > while to
> > > > > > >> figure it
> > > > > > >> > > out.
> > > > > > >> > > > > > > > Perhaps, we could come up with a
> > better
> > > > name
> > > > > than
> > > > > > `count`?
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > 2. I had a hard time trying to
> > > understand
> > > > the
> > > > > > >> `AuthorizationMode`
> > > > > > >> > > > > > > concept,
> > > > > > >> > > > > > > > especially wrt. the OPTIONAL one. My
> > > > > understanding is
> > > > > > that
> > > > > > >> an
> > > > > > >> > > ACL is
> > > > > > >> > > > > > > either
> > > > > > >> > > > > > > > defined or not. Could you elaborate
> a
> > > bit
> > > > more
> > > > > on
> > > > > > that?
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > Thanks,
> > > > > > >> > > > > > > > David
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don
> > > Bosco
> > > > Durai
> > > > > <
> > > > > > >> > > bosco@apache.org>
> > > > > > >> > > > > > > wrote:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > Hi Rajini
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > 3.2 - This makes sense. Thanks for
> > > > clarifying.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Rest looks fine. Once the
> > > > implementations are
> > > > > done,
> > > > > > it
> > > > > > >> will be
> > > > > > >> > > more
> > > > > > >> > > > > > > clear
> > > > > > >> > > > > > > > > on the different values
> RequestType
> > > and
> > > > Mode.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Thanks
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Bosco
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini
> > Sivaram"
> > > <
> > > > > > >> rajinisivaram@gmail.com
> > > > > > >> > > >
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Hi Don,
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Thanks for the suggestions. A
> > few
> > > > > responses
> > > > > > below:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > 3.1 Can rename and improve
> docs
> > if
> > > > we keep
> > > > > > this. Let's
> > > > > > >> > > finish the
> > > > > > >> > > > > > > > > discussion on Colin's
> > suggestions
> > > > > regarding this
> > > > > > >> first.
> > > > > > >> > > > > > > > > 3.2 No, I was thinking of some
> > > > requests
> > > > > that
> > > > > > have an
> > > > > > >> old
> > > > > > >> > > way of
> > > > > > >> > > > > > > > > authorizing
> > > > > > >> > > > > > > > > and a new way where we have
> > > > retained the
> > > > > old
> > > > > > way for
> > > > > > >> > > backward
> > > > > > >> > > > > > > > > compatibility. One example is
> > > > > Cluster:Create
> > > > > > >> permission to
> > > > > > >> > > create
> > > > > > >> > > > > > > > > topics.
> > > > > > >> > > > > > > > > We have replaced this with
> > > > fine-grained
> > > > > topic
> > > > > > create
> > > > > > >> > > access using
> > > > > > >> > > > > > > > > Topic:Create
> > > > > > >> > > > > > > > > for topic patterns. But we
> still
> > > > check if
> > > > > user
> > > > > > has
> > > > > > >> > > Cluster:Create
> > > > > > >> > > > > > > > > first. If
> > > > > > >> > > > > > > > > Denied, the deny is ignored
> and
> > we
> > > > check
> > > > > > >> Topic:Create. We
> > > > > > >> > > dont
> > > > > > >> > > > > > want
> > > > > > >> > > > > > > > to
> > > > > > >> > > > > > > > > log
> > > > > > >> > > > > > > > > DENY for Cluster:Create at
> INFO
> > > > level for
> > > > > this,
> > > > > > since
> > > > > > >> this
> > > > > > >> > > is
> > > > > > >> > > > > > not a
> > > > > > >> > > > > > > > > mandatory ACL for creating
> > topics.
> > > > We
> > > > > will get
> > > > > > >> appropriate
> > > > > > >> > > logs
> > > > > > >> > > > > > > from
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > subsequent Topic:Create in
> this
> > > > case.
> > > > > > >> > > > > > > > > 3.3 They are not quite the
> same.
> > > > FILTER
> > > > > implies
> > > > > > that
> > > > > > >> user
> > > > > > >> > > > > > actually
> > > > > > >> > > > > > > > > requested access to and
> > performed
> > > > some
> > > > > > operation on
> > > > > > >> the
> > > > > > >> > > filtered
> > > > > > >> > > > > > > > > resources.
> > > > > > >> > > > > > > > > LIST_AUTHORZED did not result
> in
> > > any
> > > > > actual
> > > > > > access.
> > > > > > >> User
> > > > > > >> > > simply
> > > > > > >> > > > > > > > wanted
> > > > > > >> > > > > > > > > to
> > > > > > >> > > > > > > > > know what they are allowed to
> > > > access.
> > > > > > >> > > > > > > > > 3.4 Request types are Produce,
> > > > JoinGroup,
> > > > > > OffsetCommit
> > > > > > >> > > etc. So
> > > > > > >> > > > > > that
> > > > > > >> > > > > > > > is
> > > > > > >> > > > > > > > > different from authorization
> > mode,
> > > > > operation
> > > > > > etc.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > On Thu, Aug 8, 2019 at 11:36
> PM
> > > Don
> > > > Bosco
> > > > > Durai
> > > > > > <
> > > > > > >> > > > > > bosco@apache.org>
> > > > > > >> > > > > > > > > wrote:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > Hi Rajini
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Thanks for clarifying. This
> is
> > > > very
> > > > > helpful...
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > #1 - On the Ranger side, we
> > > > should be
> > > > > able to
> > > > > > handle
> > > > > > >> > > multiple
> > > > > > >> > > > > > > > > requests at
> > > > > > >> > > > > > > > > > the same time. I was just
> not
> > > > sure how
> > > > > much
> > > > > > >> processing
> > > > > > >> > > overhead
> > > > > > >> > > > > > > > will
> > > > > > >> > > > > > > > > be
> > > > > > >> > > > > > > > > > there on the Broker side to
> > > split
> > > > and
> > > > > then
> > > > > > >> consolidate
> > > > > > >> > > the
> > > > > > >> > > > > > > results.
> > > > > > >> > > > > > > > > If it
> > > > > > >> > > > > > > > > > is negligible, then this is
> > the
> > > > better
> > > > > way.
> > > > > > It will
> > > > > > >> > > make it
> > > > > > >> > > > > > > future
> > > > > > >> > > > > > > > > proof.
> > > > > > >> > > > > > > > > > #2 - I agree, having a
> single
> > > > "start"
> > > > > call
> > > > > > makes it
> > > > > > >> > > cleaner.
> > > > > > >> > > > > > The
> > > > > > >> > > > > > > > > > Authorization implementation
> > > will
> > > > only
> > > > > have
> > > > > > to do
> > > > > > >> the
> > > > > > >> > > > > > > > initialization
> > > > > > >> > > > > > > > > only
> > > > > > >> > > > > > > > > > once.
> > > > > > >> > > > > > > > > > #3.1 - Thanks for the
> > > > clarification. I
> > > > > think
> > > > > > it
> > > > > > >> makes
> > > > > > >> > > sense to
> > > > > > >> > > > > > > have
> > > > > > >> > > > > > > > > this.
> > > > > > >> > > > > > > > > > The term "Mode" might not be
> > > > explicit
> > > > > enough.
> > > > > > >> > > Essentially it
> > > > > > >> > > > > > > seems
> > > > > > >> > > > > > > > > you want
> > > > > > >> > > > > > > > > > the Authorizer to know the
> > > > > Intent/Purpose of
> > > > > > the
> > > > > > >> > > authorize call
> > > > > > >> > > > > > > and
> > > > > > >> > > > > > > > > let the
> > > > > > >> > > > > > > > > > Authorizer decide what to
> log
> > as
> > > > Audit
> > > > > event.
> > > > > > >> Changing
> > > > > > >> > > the
> > > > > > >> > > > > > > > > class/field name
> > > > > > >> > > > > > > > > > or giving more documentation
> > > will
> > > > do.
> > > > > > >> > > > > > > > > > #3.2 - Regarding the option
> > > > "OPTIONAL".
> > > > > Are
> > > > > > you
> > > > > > >> thinking
> > > > > > >> > > from
> > > > > > >> > > > > > > > > chaining
> > > > > > >> > > > > > > > > > multiple Authorizer? If
> so, I
> > > am
> > > > not
> > > > > sure
> > > > > > whether
> > > > > > >> the
> > > > > > >> > > Broker
> > > > > > >> > > > > > > would
> > > > > > >> > > > > > > > > have
> > > > > > >> > > > > > > > > > enough information to make
> > that
> > > > > decision. I
> > > > > > feel the
> > > > > > >> > > Authorizer
> > > > > > >> > > > > > > > will
> > > > > > >> > > > > > > > > be the
> > > > > > >> > > > > > > > > > one who would have that
> > > > knowledge. E.g.
> > > > > in
> > > > > > Ranger
> > > > > > >> we have
> > > > > > >> > > > > > > explicit
> > > > > > >> > > > > > > > > deny,
> > > > > > >> > > > > > > > > > which means no matter what,
> > the
> > > > request
> > > > > > should be
> > > > > > >> denied
> > > > > > >> > > for
> > > > > > >> > > > > > the
> > > > > > >> > > > > > > > > user/group
> > > > > > >> > > > > > > > > > or condition. So if you are
> > > > thinking of
> > > > > > chaining
> > > > > > >> > > Authorizers,
> > > > > > >> > > > > > > then
> > > > > > >> > > > > > > > > > responses should have the
> > third
> > > > state,
> > > > > e.g.
> > > > > > >> > > "DENIED_FINAL", in
> > > > > > >> > > > > > > > which
> > > > > > >> > > > > > > > > case
> > > > > > >> > > > > > > > > > if there is an Authorization
> > > > chain, it
> > > > > will
> > > > > > be stop
> > > > > > >> and
> > > > > > >> > > the
> > > > > > >> > > > > > > request
> > > > > > >> > > > > > > > > will be
> > > > > > >> > > > > > > > > > denied and if it is just
> > denied,
> > > > then
> > > > > you
> > > > > > might fall
> > > > > > >> > > back to
> > > > > > >> > > > > > next
> > > > > > >> > > > > > > > > > authorizer. If we don't have
> > > > chaining of
> > > > > > >> Authorizing in
> > > > > > >> > > mind,
> > > > > > >> > > > > > > then
> > > > > > >> > > > > > > > we
> > > > > > >> > > > > > > > > > should reconsider OPTIONAL
> for
> > > > now. Or
> > > > > > clarify under
> > > > > > >> > > which
> > > > > > >> > > > > > > scenario
> > > > > > >> > > > > > > > > > OPTIONAL will be called.
> > > > > > >> > > > > > > > > > #3.3 Regarding, FILTER v/s
> > > > > LIST_AUTHORIZED,
> > > > > > isn't
> > > > > > >> > > > > > > LIST_AUTHORIZED a
> > > > > > >> > > > > > > > > > special case for "FILTER"?
> > > > > > >> > > > > > > > > > #3.4 KafkaRequestContext.
> > > > requestType()
> > > > > v/s
> > > > > > Action.
> > > > > > >> > > > > > > > > authorizationMode. I
> > > > > > >> > > > > > > > > > am not sure about the
> overlap
> > or
> > > > > ambiguity.
> > > > > > >> > > > > > > > > > #4 - Cool. This is good, it
> > will
> > > > be less
> > > > > > stress on
> > > > > > >> the
> > > > > > >> > > > > > > Authorizer.
> > > > > > >> > > > > > > > > Ranger
> > > > > > >> > > > > > > > > > already supports the "count"
> > > > concept
> > > > > and also
> > > > > > has
> > > > > > >> > > batching
> > > > > > >> > > > > > > > > capability to
> > > > > > >> > > > > > > > > > aggregate similar requests
> to
> > > > reduce the
> > > > > > number of
> > > > > > >> audit
> > > > > > >> > > logs
> > > > > > >> > > > > > to
> > > > > > >> > > > > > > > > write. We
> > > > > > >> > > > > > > > > > should be able to pass this
> > > > through.
> > > > > > >> > > > > > > > > > #5 - Assuming if the object
> > > > instance is
> > > > > going
> > > > > > out of
> > > > > > >> > > scope, we
> > > > > > >> > > > > > > > > should be
> > > > > > >> > > > > > > > > > fine. Not a super important
> > ask.
> > > > Ranger
> > > > > is
> > > > > > already
> > > > > > >> > > catching
> > > > > > >> > > > > > KILL
> > > > > > >> > > > > > > > > signal for
> > > > > > >> > > > > > > > > > clean up.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Thanks again. These are good
> > > > > enhancements.
> > > > > > >> Appreciate
> > > > > > >> > > your
> > > > > > >> > > > > > > efforts
> > > > > > >> > > > > > > > > here.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Bosco
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > On 8/8/19, 2:03 AM, "Rajini
> > > > Sivaram" <
> > > > > > >> > > rajinisivaram@gmail.com
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > > wrote:
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Hi Don,
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Thanks for reviewing the
> > > KIP.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > 1. I had this originally
> > as
> > > a
> > > > single
> > > > > > Action, but
> > > > > > >> > > thought it
> > > > > > >> > > > > > > may
> > > > > > >> > > > > > > > > be
> > > > > > >> > > > > > > > > > useful
> > > > > > >> > > > > > > > > > to support batched
> > authorize
> > > > calls
> > > > > as
> > > > > > well and
> > > > > > >> keep
> > > > > > >> > > it
> > > > > > >> > > > > > > > > consistent with
> > > > > > >> > > > > > > > > > other methods. Single
> > > > requests can
> > > > > contain
> > > > > > >> multiple
> > > > > > >> > > topics.
> > > > > > >> > > > > > > For
> > > > > > >> > > > > > > > > > example a
> > > > > > >> > > > > > > > > > produce request can
> > contain
> > > > records
> > > > > for
> > > > > > several
> > > > > > >> > > partitions
> > > > > > >> > > > > > of
> > > > > > >> > > > > > > > > different
> > > > > > >> > > > > > > > > > topics. Broker could
> > > > potentially
> > > > > > authorize these
> > > > > > >> > > together.
> > > > > > >> > > > > > > For
> > > > > > >> > > > > > > > > > SimpleAclAuthorizer,
> > batched
> > > > > authorize
> > > > > > methods
> > > > > > >> don't
> > > > > > >> > > > > > provide
> > > > > > >> > > > > > > > any
> > > > > > >> > > > > > > > > > optimisation since
> lookup
> > is
> > > > based
> > > > > on
> > > > > > resources
> > > > > > >> > > followed by
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > > > matching
> > > > > > >> > > > > > > > > > logic. But some
> > authorizers
> > > > may
> > > > > manage
> > > > > > ACLs by
> > > > > > >> user
> > > > > > >> > > > > > principal
> > > > > > >> > > > > > > > > rather
> > > > > > >> > > > > > > > > > than
> > > > > > >> > > > > > > > > > resource and may be able
> > to
> > > > optimize
> > > > > > batched
> > > > > > >> > > requests. I am
> > > > > > >> > > > > > > ok
> > > > > > >> > > > > > > > > with
> > > > > > >> > > > > > > > > > using
> > > > > > >> > > > > > > > > > single Action if this is
> > > > likely to
> > > > > cause
> > > > > > issues.
> > > > > > >> > > > > > > > > > 2. If you have two
> > > listeners,
> > > > one
> > > > > for
> > > > > > >> inter-broker
> > > > > > >> > > traffic
> > > > > > >> > > > > > > and
> > > > > > >> > > > > > > > > another
> > > > > > >> > > > > > > > > > for
> > > > > > >> > > > > > > > > > external clients, start
> > > > method is
> > > > > invoked
> > > > > > twice,
> > > > > > >> > > once for
> > > > > > >> > > > > > > each
> > > > > > >> > > > > > > > > > listener. On
> > > > > > >> > > > > > > > > > second thought, that may
> > be
> > > > > confusing and
> > > > > > a
> > > > > > >> single
> > > > > > >> > > start()
> > > > > > >> > > > > > > > > invocation
> > > > > > >> > > > > > > > > > that
> > > > > > >> > > > > > > > > > provides all listener
> > > > information
> > > > > and
> > > > > > returns
> > > > > > >> > > multiple
> > > > > > >> > > > > > > futures
> > > > > > >> > > > > > > > > would be
> > > > > > >> > > > > > > > > > better. Will update the
> > KIP.
> > > > > > >> > > > > > > > > > 3. A typical example is
> a
> > > > consumer
> > > > > > subscribing
> > > > > > >> to a
> > > > > > >> > > regex
> > > > > > >> > > > > > > > > pattern. We
> > > > > > >> > > > > > > > > > request all topic
> metadata
> > > > from the
> > > > > > broker in
> > > > > > >> order
> > > > > > >> > > to
> > > > > > >> > > > > > decide
> > > > > > >> > > > > > > > > whether
> > > > > > >> > > > > > > > > > the
> > > > > > >> > > > > > > > > > pattern matches,
> expecting
> > > to
> > > > > receive a
> > > > > > list of
> > > > > > >> > > authorised
> > > > > > >> > > > > > > > > topics. The
> > > > > > >> > > > > > > > > > user
> > > > > > >> > > > > > > > > > is not asking to
> subscribe
> > > to
> > > > an
> > > > > > unauthorized
> > > > > > >> topic.
> > > > > > >> > > If
> > > > > > >> > > > > > there
> > > > > > >> > > > > > > > > are 10000
> > > > > > >> > > > > > > > > > topics in the cluster
> and
> > > the
> > > > user
> > > > > has
> > > > > > access
> > > > > > >> to 100
> > > > > > >> > > of
> > > > > > >> > > > > > them,
> > > > > > >> > > > > > > > at
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > moment
> > > > > > >> > > > > > > > > > we log 9900 DENIED log
> > > > entries at
> > > > > INFO
> > > > > > level in
> > > > > > >> > > > > > > > > SimpleAclAuthorizer.
> > > > > > >> > > > > > > > > > The
> > > > > > >> > > > > > > > > > proposal is to authorize
> > > this
> > > > > request with
> > > > > > >> > > > > > > > > AuthorizationMode.FILTER, so
> > > > > > >> > > > > > > > > > that authorizers can log
> > > > resources
> > > > > that
> > > > > > are
> > > > > > >> filtered
> > > > > > >> > > out at
> > > > > > >> > > > > > > > > lower level
> > > > > > >> > > > > > > > > > like DEBUG since this is
> > not
> > > > an
> > > > > attempt to
> > > > > > >> access
> > > > > > >> > > > > > > unauthorized
> > > > > > >> > > > > > > > > > resources.
> > > > > > >> > > > > > > > > > Brokers already handle
> > these
> > > > > differently
> > > > > > since
> > > > > > >> no
> > > > > > >> > > > > > > authorization
> > > > > > >> > > > > > > > > error
> > > > > > >> > > > > > > > > > is
> > > > > > >> > > > > > > > > > returned to the client
> in
> > > > these
> > > > > cases.
> > > > > > Providing
> > > > > > >> > > > > > > authorization
> > > > > > >> > > > > > > > > mode to
> > > > > > >> > > > > > > > > > authorizers enables
> > > authorizer
> > > > > > implementations
> > > > > > >> to
> > > > > > >> > > generate
> > > > > > >> > > > > > > > > better audit
> > > > > > >> > > > > > > > > > logs.
> > > > > > >> > > > > > > > > > 4. Each request may
> > contain
> > > > multiple
> > > > > > instances
> > > > > > >> of
> > > > > > >> > > the same
> > > > > > >> > > > > > > > > authorizable
> > > > > > >> > > > > > > > > > resource. For example a
> > > > produce
> > > > > request
> > > > > > may
> > > > > > >> contain
> > > > > > >> > > records
> > > > > > >> > > > > > > for
> > > > > > >> > > > > > > > > 10
> > > > > > >> > > > > > > > > > partitions of the same
> > > topic.
> > > > At the
> > > > > > moment, we
> > > > > > >> > > invoke
> > > > > > >> > > > > > > > authorize
> > > > > > >> > > > > > > > > > method 10
> > > > > > >> > > > > > > > > > times. The proposal is
> to
> > > > invoke it
> > > > > once
> > > > > > with
> > > > > > >> > > count=10. The
> > > > > > >> > > > > > > > > count is
> > > > > > >> > > > > > > > > > provided to authorizer
> > just
> > > > for
> > > > > audit
> > > > > > logging
> > > > > > >> > > purposes.
> > > > > > >> > > > > > > > > > 5. Authorizer implements
> > > > Closeable,
> > > > > so you
> > > > > > >> could use
> > > > > > >> > > > > > close()
> > > > > > >> > > > > > > to
> > > > > > >> > > > > > > > > flush
> > > > > > >> > > > > > > > > > audits?
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > On Thu, Aug 8, 2019 at
> > 7:01
> > > > AM Don
> > > > > Bosco
> > > > > > Durai <
> > > > > > >> > > > > > > > bosco@apache.org
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > > Rajini
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Thanks for putting
> this
> > > > together.
> > > > > It is
> > > > > > >> looking
> > > > > > >> > > good. I
> > > > > > >> > > > > > > have
> > > > > > >> > > > > > > > > few
> > > > > > >> > > > > > > > > > > questions...
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > 1.
> > > List<AuthorizationResult>
> > > > > > authorize(...,
> > > > > > >> > > List<Action>
> > > > > > >> > > > > > > > > actions).
> > > > > > >> > > > > > > > > > Do you
> > > > > > >> > > > > > > > > > > see a scenario where
> the
> > > > broker
> > > > > will
> > > > > > call
> > > > > > >> > > authorize for
> > > > > > >> > > > > > > > > multiple
> > > > > > >> > > > > > > > > > topics at
> > > > > > >> > > > > > > > > > > the same time? I can
> > > > understand
> > > > > that
> > > > > > during
> > > > > > >> > > > > > > creating/deleting
> > > > > > >> > > > > > > > > ACLS,
> > > > > > >> > > > > > > > > > > multiple permissions
> for
> > > > multiple
> > > > > > resources
> > > > > > >> might
> > > > > > >> > > be
> > > > > > >> > > > > > done.
> > > > > > >> > > > > > > > For
> > > > > > >> > > > > > > > > > authorize
> > > > > > >> > > > > > > > > > > call, would this be a
> > > case?
> > > > And
> > > > > does the
> > > > > > >> Authorize
> > > > > > >> > > > > > > > > implementation
> > > > > > >> > > > > > > > > > will be
> > > > > > >> > > > > > > > > > > able to do performance
> > > > > optimization
> > > > > > because of
> > > > > > >> > > this? Or
> > > > > > >> > > > > > > > should
> > > > > > >> > > > > > > > > we
> > > > > > >> > > > > > > > > > just keep
> > > > > > >> > > > > > > > > > > it simple? I don't see
> > it
> > > > as an
> > > > > issue
> > > > > > from
> > > > > > >> Apache
> > > > > > >> > > Ranger
> > > > > > >> > > > > > > > side,
> > > > > > >> > > > > > > > > but
> > > > > > >> > > > > > > > > > just
> > > > > > >> > > > > > > > > > > checking to see
> whether
> > we
> > > > need
> > > > > to be
> > > > > > aware of
> > > > > > >> > > something.
> > > > > > >> > > > > > > > > > > 2. Should I assume
> that
> > > the
> > > > > > SecurityProtocol
> > > > > > >> passed
> > > > > > >> > > > > > during
> > > > > > >> > > > > > > > > start and
> > > > > > >> > > > > > > > > > the
> > > > > > >> > > > > > > > > > > one return by
> > > > > > >> > > KafkaRequestContext.securityProtocol() will
> > > > > > >> > > > > > > be
> > > > > > >> > > > > > > > > the
> > > > > > >> > > > > > > > > > same?
> > > > > > >> > > > > > > > > > >
> CompletableFuture<Void>
> > > > > start(String
> > > > > > >> listenerName,
> > > > > > >> > > > > > > > > SecurityProtocol
> > > > > > >> > > > > > > > > > > securityProtocol);
> > > > > > >> > > > > > > > > > >
> > > > > KafkaRequestContext.securityProtocol()
> > > > > > >> > > > > > > > > > > 3. What is the purpose
> > of
> > > > > > AuthorizationMode?
> > > > > > >> How
> > > > > > >> > > does the
> > > > > > >> > > > > > > > > broker
> > > > > > >> > > > > > > > > > decide
> > > > > > >> > > > > > > > > > > what mode to use when
> > the
> > > > > authorize()
> > > > > > method
> > > > > > >> is
> > > > > > >> > > called?
> > > > > > >> > > > > > > > > > > 4. Can we clarify
> > "count"
> > > in
> > > > > Action a
> > > > > > bit
> > > > > > >> more?
> > > > > > >> > > How is it
> > > > > > >> > > > > > > > used?
> > > > > > >> > > > > > > > > > > 5. Do you feel having
> > > "stop"
> > > > > along with
> > > > > > >> "start" be
> > > > > > >> > > > > > helpful?
> > > > > > >> > > > > > > > > E.g. In
> > > > > > >> > > > > > > > > > Ranger
> > > > > > >> > > > > > > > > > > we try to optimize the
> > > Audit
> > > > > writing by
> > > > > > >> caching
> > > > > > >> > > the logs
> > > > > > >> > > > > > > for
> > > > > > >> > > > > > > > a
> > > > > > >> > > > > > > > > fixed
> > > > > > >> > > > > > > > > > > interval. But when the
> > > > Broker
> > > > > > terminates, we
> > > > > > >> do a
> > > > > > >> > > forced
> > > > > > >> > > > > > > > flush.
> > > > > > >> > > > > > > > > > Having an
> > > > > > >> > > > > > > > > > > explicit "stop" might
> > give
> > > > us a
> > > > > formal
> > > > > > way to
> > > > > > >> > > flush our
> > > > > > >> > > > > > > > audits.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Thanks
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Bosco
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > On 8/7/19, 3:59 PM,
> > > "Rajini
> > > > > Sivaram" <
> > > > > > >> > > > > > > > rajinisivaram@gmail.com
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Hi
> > Ron/Harsha/Satish,
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Thanks for
> reviewing
> > > > the KIP!
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > We should perhaps
> > have
> > > > a wider
> > > > > > discussion
> > > > > > >> > > outside
> > > > > > >> > > > > > this
> > > > > > >> > > > > > > > KIP
> > > > > > >> > > > > > > > > for
> > > > > > >> > > > > > > > > > > refactoring
> > > > > > >> > > > > > > > > > > clients so that
> > others
> > > > who
> > > > > are not
> > > > > > >> following
> > > > > > >> > > this KIP
> > > > > > >> > > > > > > > also
> > > > > > >> > > > > > > > > > notice the
> > > > > > >> > > > > > > > > > > discussion.
> Satish,
> > > > would you
> > > > > like
> > > > > > to
> > > > > > >> start a
> > > > > > >> > > > > > > discussion
> > > > > > >> > > > > > > > > thread
> > > > > > >> > > > > > > > > > on dev?
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Regards,
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Rajini
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > On Wed, Aug 7,
> 2019
> > at
> > > > 6:21 PM
> > > > > > Satish
> > > > > > >> Duggana <
> > > > > > >> > > > > > > > > > >
> > satish.duggana@gmail.com>
> > > > > > >> > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > I felt the same
> > need
> > > > when
> > > > > we want
> > > > > > to
> > > > > > >> add a
> > > > > > >> > > > > > pluggable
> > > > > > >> > > > > > > > API
> > > > > > >> > > > > > > > > for
> > > > > > >> > > > > > > > > > core
> > > > > > >> > > > > > > > > > > > server
> > > functionality.
> > > > This
> > > > > does
> > > > > > not
> > > > > > >> need to
> > > > > > >> > > be part
> > > > > > >> > > > > > > of
> > > > > > >> > > > > > > > > this
> > > > > > >> > > > > > > > > > KIP, it
> > > > > > >> > > > > > > > > > > > can be a
> separate
> > > > KIP. I can
> > > > > > contribute
> > > > > > >> those
> > > > > > >> > > > > > > > refactoring
> > > > > > >> > > > > > > > > > changes if
> > > > > > >> > > > > > > > > > > > others are OK
> with
> > > > that.
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > It is better to
> > > have a
> > > > > structure
> > > > > > like
> > > > > > >> below.
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > kafka-common:
> > > > > > >> > > > > > > > > > > > common classes
> > which
> > > > can be
> > > > > used
> > > > > > in any
> > > > > > >> of
> > > > > > >> > > the
> > > > > > >> > > > > > other
> > > > > > >> > > > > > > > > modules
> > > > > > >> > > > > > > > > > in Kafka
> > > > > > >> > > > > > > > > > > > like client,
> > > > > Kafka-server-common
> > > > > > and
> > > > > > >> server
> > > > > > >> > > etc.
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > kafka-client-common:
> > > > > > >> > > > > > > > > > > > common classes
> > which
> > > > can be
> > > > > used
> > > > > > in the
> > > > > > >> > > client
> > > > > > >> > > > > > > module.
> > > > > > >> > > > > > > > > This
> > > > > > >> > > > > > > > > > can be
> > > > > > >> > > > > > > > > > > > part of client
> > > module
> > > > > itself.
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > kafka-server-common:
> > > > > > >> > > > > > > > > > > > classes required
> > > only
> > > > for
> > > > > > kafka-server.
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Thanks.
> > > > > > >> > > > > > > > > > > > Satish.
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > On Wed, Aug 7,
> > 2019
> > > > at 9:28
> > > > > PM
> > > > > > Harsha
> > > > > > >> > > Chintalapani
> > > > > > >> > > > > > <
> > > > > > >> > > > > > > > > > kafka@harsha.io>
> > > > > > >> > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Thanks for the
> > KIP
> > > > Rajini.
> > > > > > >> > > > > > > > > > > > > Quick thought,
> > it
> > > > would
> > > > > be good
> > > > > > to
> > > > > > >> have a
> > > > > > >> > > common
> > > > > > >> > > > > > > > module
> > > > > > >> > > > > > > > > > outside of
> > > > > > >> > > > > > > > > > > > clients
> > > > > > >> > > > > > > > > > > > > that only
> > applies
> > > to
> > > > > server side
> > > > > > >> > > interfaces &
> > > > > > >> > > > > > > > changes.
> > > > > > >> > > > > > > > > It
> > > > > > >> > > > > > > > > > looks
> > > > > > >> > > > > > > > > > > like we
> > > > > > >> > > > > > > > > > > > are
> > > > > > >> > > > > > > > > > > > > increasingly
> in
> > > > favor of
> > > > > using
> > > > > > Java
> > > > > > >> > > interface for
> > > > > > >> > > > > > > > > pluggable
> > > > > > >> > > > > > > > > > > modules on
> > > > > > >> > > > > > > > > > > > the
> > > > > > >> > > > > > > > > > > > > broker side.
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > Thanks,
> > > > > > >> > > > > > > > > > > > > Harsha
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > On Tue, Aug
> 06,
> > > > 2019 at
> > > > > 2:31 PM,
> > > > > > >> Rajini
> > > > > > >> > > Sivaram <
> > > > > > >> > > > > > > > > > >
> rajinisivaram@gmail.com
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Hi all,
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > I have
> > created a
> > > > KIP to
> > > > > > replace the
> > > > > > >> Scala
> > > > > > >> > > > > > > > Authorizer
> > > > > > >> > > > > > > > > API
> > > > > > >> > > > > > > > > > with a
> > > > > > >> > > > > > > > > > > new
> > > > > > >> > > > > > > > > > > > Java
> > > > > > >> > > > > > > > > > > > > > API:
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > -
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > This is
> > > > replacement for
> > > > > KIP-50
> > > > > > >> which was
> > > > > > >> > > > > > accepted
> > > > > > >> > > > > > > > > but never
> > > > > > >> > > > > > > > > > > merged.
> > > > > > >> > > > > > > > > > > > Apart
> > > > > > >> > > > > > > > > > > > > > from moving
> > to a
> > > > Java
> > > > > API
> > > > > > consistent
> > > > > > >> > > with other
> > > > > > >> > > > > > > > > pluggable
> > > > > > >> > > > > > > > > > > interfaces
> > > > > > >> > > > > > > > > > > > in the
> > > > > > >> > > > > > > > > > > > > > broker,
> > KIP-504
> > > > also
> > > > > attempts
> > > > > > to
> > > > > > >> address
> > > > > > >> > > known
> > > > > > >> > > > > > > > > limitations
> > > > > > >> > > > > > > > > > in the
> > > > > > >> > > > > > > > > > > > > > authorizer.
> If
> > > > you have
> > > > > come
> > > > > > across
> > > > > > >> other
> > > > > > >> > > > > > > > > limitations that
> > > > > > >> > > > > > > > > > you
> > > > > > >> > > > > > > > > > > would
> > > > > > >> > > > > > > > > > > > like
> > > > > > >> > > > > > > > > > > > > > to see
> > addressed
> > > > in the
> > > > > new
> > > > > > API,
> > > > > > >> please
> > > > > > >> > > raise
> > > > > > >> > > > > > > these
> > > > > > >> > > > > > > > > on the
> > > > > > >> > > > > > > > > > > discussion
> > > > > > >> > > > > > > > > > > > > > thread so
> that
> > > we
> > > > can
> > > > > > consider those
> > > > > > >> > > too. All
> > > > > > >> > > > > > > > > suggestions
> > > > > > >> > > > > > > > > > and
> > > > > > >> > > > > > > > > > > feedback
> > > > > > >> > > > > > > > > > > > are
> > > > > > >> > > > > > > > > > > > > > welcome.
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Thank you,
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > > > Rajini
> > > > > > >> > > > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Jun,
I think it's important to discuss the pluggable API versus the calling
implementation as two separate points.
From an API perspective, are you suggesting that we should tell users that
they cannot block in authorize? Or are you saying that it's OK to block on
authorize on occasion and the recommendation would be for people to
increase the number of threads in the request thread pool? Architecturally,
this feels wrong in my opinion.
From the calling implementation perspective, you don't need a callback. You
can basically say:
authorize(...).map { result =>
...
}
And then have a common method take that Future and handle it synchronously
if it's already complete or submit it to the purgatory if not. This is
similar to how many modern async web libraries work.
As Rajini said, this could be done later. The initial implementation could
simply do `toCompletableFuture().get()` to do it synchronously. But it
would mean that we could add this functionality without changing the
pluggable interface.
Ismael
On Wed, Sep 4, 2019 at 5:29 AM Jun Rao <ju...@confluent.io> wrote:
> Hi, Rajini,
>
> Thanks for the KIP. I was concerned about #4 too. If we change the handling
> of all requests to use an async authorize() api, will that cause the code
> much harder to understand? There are quite a few callbacks already. I am
> not sure that we want to introduce more of those. The benefit from async
> authorize() api seems limited.
>
> Jun
>
> On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <ra...@gmail.com>
> wrote:
>
> > Hi Don,
> >
> > Thanks for your note.
> >
> > 1) The intention is to avoid blocking in the calling thread. We already
> > have several requests that are put into a purgatory when waiting for
> remote
> > communication, for example produce request waiting for replication. Since
> > we have a limited number of request threads in the broker, we want to
> make
> > progress with other requests, while one is waiting on any form of remote
> > communication.
> >
> > 2) Async management calls will be useful with the default authorizer when
> > KIP-500 removes ZK and we rely on Kafka instead. Our current ZK-based
> > implementation as well as any custom implementations that don't want to
> be
> > async will just need to return a sync'ed value. So instead of returning `
> > value`, the code would just return
> > `CompletableFuture.completedFuture(value)
> > `. So it would be just a single line change in the implementation with
> the
> > new API. The caller would treat completedFuture exactly as it does today,
> > processing the request synchronously without using a purgatory.
> >
> > 3) For implementations that return a completedFuture as described in 2),
> > the behaviour would remain exactly the same. No additional threads or
> > purgatory will be used for this case. So there would be no performance
> > penalty. For implementations that return a future that is not complete,
> we
> > prioritise running more requests concurrently. So in a deployment with a
> > large number of clients, we would improve performance by allowing other
> > requests to be processed on the request threads while some are waiting
> for
> > authorization metadata.
> >
> > 4) I was concerned about this too. The goal is to make the API flexible
> > enough to handle large scale deployments in future when caching all
> > authorization metadata in each broker is not viable. Using an async API
> > that returns CompletionStage, the caller has the option to handle the
> > result synchronously or asynchronously, so we don't necessarily need to
> > update the calling code right away. Custom authorizers using the async
> API
> > have full control over whether authorization is performed in-line since
> > completedFuture will always be handled synchronously. We do need to
> update
> > KafkaApis to take advantage of the asynchronous API to improve scale.
> Even
> > though this is a big change, since we will be doing the same for all
> > requests, it shouldn't be too hard to maintain since the same pattern
> will
> > be used for all requests.
> >
> > Regards,
> >
> > Rajini
> >
> > On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <bo...@apache.org>
> wrote:
> >
> > > Hi Rajini
> > >
> > > Help me understand this a bit more.
> > >
> > > 1. For all practical purpose, without authorization you can't go to the
> > > next step. The calling code needs to block anyway. So should we just
> let
> > > the implementation code do the async part?
> > > 2. If you feel management calls need to be async, then we should
> consider
> > > another set of async APIs. I don't feel we should complicate it
> further (
> > > 3. Another concern I have is wrt performance. Kafka has been built to
> > > handle 1000s per second requests. Not sure whether making it async will
> > add
> > > any unnecessary overhead.
> > > 4. How much complication would this add on the calling side? And is it
> > > worth it?
> > >
> > > Thanks
> > >
> > > Bosco
> > >
> > >
> > > On 9/3/19, 8:50 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
> > >
> > > Hi all,
> > >
> > > Ismael brought up a point that it will be good to make the
> Authorizer
> > > interface asynchronous to avoid blocking request threads during
> > remote
> > > operations.
> > >
> > > 1) Since we want to support different backends for authorization
> > > metadata,
> > > making createAcls() and deleteAcls() asynchronous makes sense since
> > > these
> > > always involve remote operations. When KIP-500 removes ZooKeeper,
> we
> > > would
> > > want to move ACLs to Kafka and async updates will avoid unnecessary
> > > blocking.
> > > 2) For authorize() method, we currently use cached ACLs in the
> > built-in
> > > authorizers, so synchronous authorise operations work well now. But
> > > async
> > > authorize() would support this scenario as well as authorizers in
> > large
> > > organisations where an LRU cache would enable a smaller cache even
> > > when the
> > > backend holds a large amount of ACLs for infrequently used
> resources
> > or
> > > users who don't use the system frequently.
> > >
> > > For both cases, the built-in authorizer will continue to be
> > > synchronous,
> > > using CompletableFuture.completedFuture() to return the actual
> > result.
> > > But
> > > async API will make custom authorizer implementations more
> flexible.
> > I
> > > would like to know if there are any concerns with these changes
> > before
> > > updating the KIP.
> > >
> > > *Proposed API:*
> > > public interface Authorizer extends Configurable, Closeable {
> > >
> > > Map<Endpoint, CompletionStage<Void>> start(AuthorizerServerInfo
> > > serverInfo);
> > > List<CompletionStage<AuthorizationResult>>
> > > authorize(AuthorizableRequestContext requestContext, List<Action>
> > > actions);
> > > List<CompletionStage<AclCreateResult>>
> > > createAcls(AuthorizableRequestContext requestContext,
> > List<AclBinding>
> > > aclBindings);
> > > List<CompletionStage<AclDeleteResult>>
> > > deleteAcls(AuthorizableRequestContext requestContext,
> > > List<AclBindingFilter> aclBindingFilters);
> > > CompletionStage<Collection<AclBinding>> acls(AclBindingFilter
> > > filter);
> > > }
> > >
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
> > > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <bo...@apache.org>
> > > wrote:
> > >
> > > > Hi Rajini
> > > >
> > > > Thanks for clarifying. I am good for now.
> > > >
> > > > Regards
> > > >
> > > > Bosco
> > > >
> > > >
> > > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <rajinisivaram@gmail.com
> >
> > > wrote:
> > > >
> > > > Hi Don,
> > > >
> > > > That should be fine. I guess Ranger loads policies from the
> > > database
> > > > synchronously when the authorizer is configured, similar to
> > > > SimpleAclAuthorizer loading from ZooKeeper. Ranger can
> continue
> > > to load
> > > > synchronously from `configure()` or `start()` and return an
> > > empty map
> > > > from
> > > > `start()`. That would retain the existing behaviour.. When
> the
> > > same
> > > > database stores policies for all listeners and the policies
> are
> > > not
> > > > stored
> > > > in Kafka, there is no value in making the load asynchronous.
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai <
> > > bosco@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Rajini
> > > > >
> > > > > Assuming this doesn't affect custom plugins, I don't have
> any
> > > > concerns
> > > > > regarding this.
> > > > >
> > > > > I do have one question regarding:
> > > > >
> > > > > "For authorizers that don’t store metadata in ZooKeeper,
> > > ensure that
> > > > > authorizer metadata for each listener is available before
> > > starting
> > > > up the
> > > > > listener. This enables different authorization metadata
> > stores
> > > for
> > > > > different listeners."
> > > > >
> > > > > Since Ranger uses its own database for storing policies, do
> > you
> > > > anticipate
> > > > > any issues?
> > > > >
> > > > > Thanks
> > > > >
> > > > > Bosco
> > > > >
> > > > >
> > > > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
> > > rajinisivaram@gmail.com>
> > > > wrote:
> > > > >
> > > > > Hi all,
> > > > >
> > > > > I made another change to the KIP. The KIP was
> originally
> > > > proposing to
> > > > > extend SimpleAclAuthorizer to also implement the new
> API
> > > (in
> > > > addition
> > > > > to
> > > > > the existing API). But since we use the new API when
> > > available,
> > > > this
> > > > > can
> > > > > break custom authorizers that extend this class and
> > > override
> > > > specific
> > > > > methods of the old API. To avoid breaking any existing
> > > custom
> > > > > implementations that extend this class, particularly
> > > because it
> > > > is in
> > > > > the
> > > > > public package kafka.security.auth, the KIP now
> proposes
> > to
> > > > retain the
> > > > > old
> > > > > authorizer as-is. SimpleAclAuthorizer will continue to
> > > > implement the
> > > > > old
> > > > > API, but will be deprecated. A new authorizer
> > > implementation
> > > > > kafka.security.authorizer.AclAuthorizer will be added
> for
> > > the
> > > > new API
> > > > > (this
> > > > > will not be in the public package).
> > > > >
> > > > > Please let me know if you have any concerns.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > > >
> > > > > On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
> > > > > rajinisivaram@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks Colin.
> > > > > >
> > > > > > If there are no other concerns, I will start vote
> later
> > > today.
> > > > Many
> > > > > thanks
> > > > > > to every one for the feedback.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > >
> > > > > > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <
> > > > cmccabe@apache.org>
> > > > > wrote:
> > > > > >
> > > > > >> Thanks, Rajini. It looks good to me.
> > > > > >>
> > > > > >> best,
> > > > > >> Colin
> > > > > >>
> > > > > >>
> > > > > >> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram
> wrote:
> > > > > >> > Hi Colin,
> > > > > >> >
> > > > > >> > Thanks for the review. I have updated the KIP to
> > move
> > > the
> > > > > interfaces for
> > > > > >> > request context and server info to the authorizer
> > > package.
> > > > These
> > > > > are now
> > > > > >> > called AuthorizableRequestContext and
> > > AuthorizerServerInfo.
> > > > > Endpoint is
> > > > > >> now
> > > > > >> > a class in org.apache.kafka.common to make it
> > reusable
> > > > since we
> > > > > already
> > > > > >> > have multiple implementations of it. I have
> removed
> > > > requestName
> > > > > from the
> > > > > >> > request context interface since authorizers can
> > > distinguish
> > > > > follower
> > > > > >> fetch
> > > > > >> > and consumer fetch from the operation being
> > > authorized. So
> > > > 16-bit
> > > > > >> request
> > > > > >> > type should be sufficient for audit logging. Also
> > > replaced
> > > > > AuditFlag
> > > > > >> with
> > > > > >> > two booleans as you suggested.
> > > > > >> >
> > > > > >> > Can you take another look and see if the KIP is
> > ready
> > > for
> > > > voting?
> > > > > >> >
> > > > > >> > Thanks for all your help!
> > > > > >> >
> > > > > >> > Regards,
> > > > > >> >
> > > > > >> > Rajini
> > > > > >> >
> > > > > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <
> > > > cmccabe@apache.org>
> > > > > >> wrote:
> > > > > >> >
> > > > > >> > > Hi Rajini,
> > > > > >> > >
> > > > > >> > > I think it would be good to rename
> > > KafkaRequestContext to
> > > > > something
> > > > > >> like
> > > > > >> > > AuthorizableRequestContext, and put it in the
> > > > > >> > > org.apache.kafka.server.authorizer namespace.
> If
> > > we put
> > > > it in
> > > > > the
> > > > > >> > > org.apache.kafka.common namespace, then it's not
> > > really
> > > > clear
> > > > > that
> > > > > >> it's
> > > > > >> > > part of the Authorizer API. Since this class is
> > > purely an
> > > > > interface,
> > > > > >> with
> > > > > >> > > no concrete implementation of anything, there's
> > > nothing
> > > > common
> > > > > to
> > > > > >> really
> > > > > >> > > reuse in any case. We definitely don't want
> > > someone to
> > > > > accidentally
> > > > > >> add or
> > > > > >> > > remove methods because they think this is just
> > > another
> > > > internal
> > > > > class
> > > > > >> used
> > > > > >> > > for requests.
> > > > > >> > >
> > > > > >> > > The BrokerInfo class is a nice improvement. It
> > > looks
> > > > like it
> > > > > will be
> > > > > >> > > useful for passing in information about the
> > context
> > > we're
> > > > > running
> > > > > >> in. It
> > > > > >> > > would be nice to call this class ServerInfo
> rather
> > > than
> > > > > BrokerInfo,
> > > > > >> since
> > > > > >> > > we will want to run the authorizer on
> controllers
> > > as well
> > > > as on
> > > > > >> brokers,
> > > > > >> > > and the controller may run as a separate process
> > > post
> > > > KIP-500.
> > > > > I also
> > > > > >> > > think that this class should be in the
> > > > > >> org.apache.kafka.server.authorizer
> > > > > >> > > namespace. Again, it is an interface, not a
> > > concrete
> > > > > implementation,
> > > > > >> and
> > > > > >> > > it's an interface that is very specifically for
> > the
> > > > authorizer.
> > > > > >> > >
> > > > > >> > > I agree that we probably don't have enough
> > > information
> > > > > preserved for
> > > > > >> > > requests currently to always know what entity
> made
> > > them.
> > > > So we
> > > > > can
> > > > > >> leave
> > > > > >> > > that out for now (except in the special case of
> > > Fetch).
> > > > > Perhaps we
> > > > > >> can add
> > > > > >> > > this later if it's needed.
> > > > > >> > >
> > > > > >> > > I understand the intention behind
> > AuthorizationMode
> > > > (which is
> > > > > now
> > > > > >> called
> > > > > >> > > AuditFlag in the latest revision). But it still
> > > feels
> > > > > complex. What
> > > > > >> if we
> > > > > >> > > just had two booleans in Action: logSuccesses
> and
> > > > logFailures?
> > > > > That
> > > > > >> seems
> > > > > >> > > to cover all the cases here.
> MANDATORY_AUTHORIZE
> > =
> > > true,
> > > > true.
> > > > > >> > > OPTIONAL_AUTHORIZE = true, false. FILTER =
> true,
> > > false.
> > > > > >> LIST_AUTHORIZED =
> > > > > >> > > false, false. Would there be anything lost
> versus
> > > having
> > > > the
> > > > > enum?
> > > > > >> > >
> > > > > >> > > best,
> > > > > >> > > Colin
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Wed, Aug 14, 2019, at 06:29, Mickael Maison
> > > wrote:
> > > > > >> > > > Hi Rajini,
> > > > > >> > > >
> > > > > >> > > > Thanks for the KIP!
> > > > > >> > > > I really like that authorize() will be able to
> > > take a
> > > > batch of
> > > > > >> > > > requests, this will speed up many
> > implementations!
> > > > > >> > > >
> > > > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini
> Sivaram <
> > > > > >> rajinisivaram@gmail.com>
> > > > > >> > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > Thanks David! I have fixed the typo.
> > > > > >> > > > >
> > > > > >> > > > > Also made a couple of changes to make the
> > > context
> > > > > interfaces more
> > > > > >> > > generic.
> > > > > >> > > > > KafkaRequestContext now returns the 16-bit
> API
> > > key as
> > > > Colin
> > > > > >> suggested
> > > > > >> > > as
> > > > > >> > > > > well as the friendly name used in metrics
> > which
> > > are
> > > > useful
> > > > > in
> > > > > >> audit
> > > > > >> > > logs.
> > > > > >> > > > > `Authorizer#start` is now provided a generic
> > > > `BrokerInfo`
> > > > > >> interface
> > > > > >> > > that
> > > > > >> > > > > gives cluster id, broker id and endpoint
> > > information.
> > > > The
> > > > > generic
> > > > > >> > > interface
> > > > > >> > > > > can potentially be used in other broker
> > plugins
> > > in
> > > > future
> > > > > and
> > > > > >> provides
> > > > > >> > > > > dynamically generated configs like broker id
> > > and ports
> > > > > which are
> > > > > >> > > currently
> > > > > >> > > > > not available to plugins unless these
> configs
> > > are
> > > > statically
> > > > > >> > > configured.
> > > > > >> > > > > Please let me know if there are any
> concerns.
> > > > > >> > > > >
> > > > > >> > > > > Regards,
> > > > > >> > > > >
> > > > > >> > > > > Rajini
> > > > > >> > > > >
> > > > > >> > > > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot
> <
> > > > > djacot@confluent.io>
> > > > > >> > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Hi Rajini,
> > > > > >> > > > > >
> > > > > >> > > > > > Thank you for the update! It looks good to
> > me.
> > > > There is a
> > > > > typo
> > > > > >> in the
> > > > > >> > > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` ->
> > > > > >> `MANDATORY_AUTHORIZE`.
> > > > > >> > > > > >
> > > > > >> > > > > > Regards,
> > > > > >> > > > > > David
> > > > > >> > > > > >
> > > > > >> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini
> > > Sivaram <
> > > > > >> > > rajinisivaram@gmail.com>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Hi David,
> > > > > >> > > > > > >
> > > > > >> > > > > > > Thanks for reviewing the KIP! Since
> > > questions
> > > > about
> > > > > >> `authorization
> > > > > >> > > mode`
> > > > > >> > > > > > > and `count` have come up multiple
> times, I
> > > have
> > > > renamed
> > > > > both.
> > > > > >> > > > > > >
> > > > > >> > > > > > > 1) Renamed `count` to
> > > `resourceReferenceCount`.
> > > > It is
> > > > > the
> > > > > >> number
> > > > > >> > > of times
> > > > > >> > > > > > > the resource being authorized is
> > referenced
> > > > within the
> > > > > >> request.
> > > > > >> > > > > > >
> > > > > >> > > > > > > 2) Renamed `AuthorizationMode` to
> > > `AuditFlag`. It
> > > > is
> > > > > provided
> > > > > >> to
> > > > > >> > > improve
> > > > > >> > > > > > > audit logging in the authorizer. The
> enum
> > > values
> > > > have
> > > > > javadoc
> > > > > >> which
> > > > > >> > > > > > > indicate how the authorization result is
> > > used in
> > > > each
> > > > > of the
> > > > > >> modes
> > > > > >> > > to
> > > > > >> > > > > > > enable authorizers to log audit messages
> > at
> > > the
> > > > right
> > > > > severity
> > > > > >> > > level.
> > > > > >> > > > > > >
> > > > > >> > > > > > > Regards,
> > > > > >> > > > > > >
> > > > > >> > > > > > > Rajini
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM David
> > Jacot
> > > <
> > > > > >> djacot@confluent.io>
> > > > > >> > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Hi Rajini,
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Thank you for the KIP. Overall, it
> looks
> > > good
> > > > to me.
> > > > > I have
> > > > > >> few
> > > > > >> > > > > > > > questions/suggestions:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > 1. It is hard to grasp what
> > > `Action#count` is
> > > > for. I
> > > > > guess I
> > > > > >> > > understand
> > > > > >> > > > > > > > where you want to go with it but it
> took
> > > me a
> > > > while to
> > > > > >> figure it
> > > > > >> > > out.
> > > > > >> > > > > > > > Perhaps, we could come up with a
> better
> > > name
> > > > than
> > > > > `count`?
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > 2. I had a hard time trying to
> > understand
> > > the
> > > > > >> `AuthorizationMode`
> > > > > >> > > > > > > concept,
> > > > > >> > > > > > > > especially wrt. the OPTIONAL one. My
> > > > understanding is
> > > > > that
> > > > > >> an
> > > > > >> > > ACL is
> > > > > >> > > > > > > either
> > > > > >> > > > > > > > defined or not. Could you elaborate a
> > bit
> > > more
> > > > on
> > > > > that?
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Thanks,
> > > > > >> > > > > > > > David
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don
> > Bosco
> > > Durai
> > > > <
> > > > > >> > > bosco@apache.org>
> > > > > >> > > > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > Hi Rajini
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > 3.2 - This makes sense. Thanks for
> > > clarifying.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Rest looks fine. Once the
> > > implementations are
> > > > done,
> > > > > it
> > > > > >> will be
> > > > > >> > > more
> > > > > >> > > > > > > clear
> > > > > >> > > > > > > > > on the different values RequestType
> > and
> > > Mode.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Thanks
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Bosco
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini
> Sivaram"
> > <
> > > > > >> rajinisivaram@gmail.com
> > > > > >> > > >
> > > > > >> > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Hi Don,
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Thanks for the suggestions. A
> few
> > > > responses
> > > > > below:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > 3.1 Can rename and improve docs
> if
> > > we keep
> > > > > this. Let's
> > > > > >> > > finish the
> > > > > >> > > > > > > > > discussion on Colin's
> suggestions
> > > > regarding this
> > > > > >> first.
> > > > > >> > > > > > > > > 3.2 No, I was thinking of some
> > > requests
> > > > that
> > > > > have an
> > > > > >> old
> > > > > >> > > way of
> > > > > >> > > > > > > > > authorizing
> > > > > >> > > > > > > > > and a new way where we have
> > > retained the
> > > > old
> > > > > way for
> > > > > >> > > backward
> > > > > >> > > > > > > > > compatibility. One example is
> > > > Cluster:Create
> > > > > >> permission to
> > > > > >> > > create
> > > > > >> > > > > > > > > topics.
> > > > > >> > > > > > > > > We have replaced this with
> > > fine-grained
> > > > topic
> > > > > create
> > > > > >> > > access using
> > > > > >> > > > > > > > > Topic:Create
> > > > > >> > > > > > > > > for topic patterns. But we still
> > > check if
> > > > user
> > > > > has
> > > > > >> > > Cluster:Create
> > > > > >> > > > > > > > > first. If
> > > > > >> > > > > > > > > Denied, the deny is ignored and
> we
> > > check
> > > > > >> Topic:Create. We
> > > > > >> > > dont
> > > > > >> > > > > > want
> > > > > >> > > > > > > > to
> > > > > >> > > > > > > > > log
> > > > > >> > > > > > > > > DENY for Cluster:Create at INFO
> > > level for
> > > > this,
> > > > > since
> > > > > >> this
> > > > > >> > > is
> > > > > >> > > > > > not a
> > > > > >> > > > > > > > > mandatory ACL for creating
> topics.
> > > We
> > > > will get
> > > > > >> appropriate
> > > > > >> > > logs
> > > > > >> > > > > > > from
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > subsequent Topic:Create in this
> > > case.
> > > > > >> > > > > > > > > 3.3 They are not quite the same.
> > > FILTER
> > > > implies
> > > > > that
> > > > > >> user
> > > > > >> > > > > > actually
> > > > > >> > > > > > > > > requested access to and
> performed
> > > some
> > > > > operation on
> > > > > >> the
> > > > > >> > > filtered
> > > > > >> > > > > > > > > resources.
> > > > > >> > > > > > > > > LIST_AUTHORZED did not result in
> > any
> > > > actual
> > > > > access.
> > > > > >> User
> > > > > >> > > simply
> > > > > >> > > > > > > > wanted
> > > > > >> > > > > > > > > to
> > > > > >> > > > > > > > > know what they are allowed to
> > > access.
> > > > > >> > > > > > > > > 3.4 Request types are Produce,
> > > JoinGroup,
> > > > > OffsetCommit
> > > > > >> > > etc. So
> > > > > >> > > > > > that
> > > > > >> > > > > > > > is
> > > > > >> > > > > > > > > different from authorization
> mode,
> > > > operation
> > > > > etc.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > On Thu, Aug 8, 2019 at 11:36 PM
> > Don
> > > Bosco
> > > > Durai
> > > > > <
> > > > > >> > > > > > bosco@apache.org>
> > > > > >> > > > > > > > > wrote:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > Hi Rajini
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Thanks for clarifying. This is
> > > very
> > > > helpful...
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > #1 - On the Ranger side, we
> > > should be
> > > > able to
> > > > > handle
> > > > > >> > > multiple
> > > > > >> > > > > > > > > requests at
> > > > > >> > > > > > > > > > the same time. I was just not
> > > sure how
> > > > much
> > > > > >> processing
> > > > > >> > > overhead
> > > > > >> > > > > > > > will
> > > > > >> > > > > > > > > be
> > > > > >> > > > > > > > > > there on the Broker side to
> > split
> > > and
> > > > then
> > > > > >> consolidate
> > > > > >> > > the
> > > > > >> > > > > > > results.
> > > > > >> > > > > > > > > If it
> > > > > >> > > > > > > > > > is negligible, then this is
> the
> > > better
> > > > way.
> > > > > It will
> > > > > >> > > make it
> > > > > >> > > > > > > future
> > > > > >> > > > > > > > > proof.
> > > > > >> > > > > > > > > > #2 - I agree, having a single
> > > "start"
> > > > call
> > > > > makes it
> > > > > >> > > cleaner.
> > > > > >> > > > > > The
> > > > > >> > > > > > > > > > Authorization implementation
> > will
> > > only
> > > > have
> > > > > to do
> > > > > >> the
> > > > > >> > > > > > > > initialization
> > > > > >> > > > > > > > > only
> > > > > >> > > > > > > > > > once.
> > > > > >> > > > > > > > > > #3.1 - Thanks for the
> > > clarification. I
> > > > think
> > > > > it
> > > > > >> makes
> > > > > >> > > sense to
> > > > > >> > > > > > > have
> > > > > >> > > > > > > > > this.
> > > > > >> > > > > > > > > > The term "Mode" might not be
> > > explicit
> > > > enough.
> > > > > >> > > Essentially it
> > > > > >> > > > > > > seems
> > > > > >> > > > > > > > > you want
> > > > > >> > > > > > > > > > the Authorizer to know the
> > > > Intent/Purpose of
> > > > > the
> > > > > >> > > authorize call
> > > > > >> > > > > > > and
> > > > > >> > > > > > > > > let the
> > > > > >> > > > > > > > > > Authorizer decide what to log
> as
> > > Audit
> > > > event.
> > > > > >> Changing
> > > > > >> > > the
> > > > > >> > > > > > > > > class/field name
> > > > > >> > > > > > > > > > or giving more documentation
> > will
> > > do.
> > > > > >> > > > > > > > > > #3.2 - Regarding the option
> > > "OPTIONAL".
> > > > Are
> > > > > you
> > > > > >> thinking
> > > > > >> > > from
> > > > > >> > > > > > > > > chaining
> > > > > >> > > > > > > > > > multiple Authorizer? If so, I
> > am
> > > not
> > > > sure
> > > > > whether
> > > > > >> the
> > > > > >> > > Broker
> > > > > >> > > > > > > would
> > > > > >> > > > > > > > > have
> > > > > >> > > > > > > > > > enough information to make
> that
> > > > decision. I
> > > > > feel the
> > > > > >> > > Authorizer
> > > > > >> > > > > > > > will
> > > > > >> > > > > > > > > be the
> > > > > >> > > > > > > > > > one who would have that
> > > knowledge. E.g.
> > > > in
> > > > > Ranger
> > > > > >> we have
> > > > > >> > > > > > > explicit
> > > > > >> > > > > > > > > deny,
> > > > > >> > > > > > > > > > which means no matter what,
> the
> > > request
> > > > > should be
> > > > > >> denied
> > > > > >> > > for
> > > > > >> > > > > > the
> > > > > >> > > > > > > > > user/group
> > > > > >> > > > > > > > > > or condition. So if you are
> > > thinking of
> > > > > chaining
> > > > > >> > > Authorizers,
> > > > > >> > > > > > > then
> > > > > >> > > > > > > > > > responses should have the
> third
> > > state,
> > > > e.g.
> > > > > >> > > "DENIED_FINAL", in
> > > > > >> > > > > > > > which
> > > > > >> > > > > > > > > case
> > > > > >> > > > > > > > > > if there is an Authorization
> > > chain, it
> > > > will
> > > > > be stop
> > > > > >> and
> > > > > >> > > the
> > > > > >> > > > > > > request
> > > > > >> > > > > > > > > will be
> > > > > >> > > > > > > > > > denied and if it is just
> denied,
> > > then
> > > > you
> > > > > might fall
> > > > > >> > > back to
> > > > > >> > > > > > next
> > > > > >> > > > > > > > > > authorizer. If we don't have
> > > chaining of
> > > > > >> Authorizing in
> > > > > >> > > mind,
> > > > > >> > > > > > > then
> > > > > >> > > > > > > > we
> > > > > >> > > > > > > > > > should reconsider OPTIONAL for
> > > now. Or
> > > > > clarify under
> > > > > >> > > which
> > > > > >> > > > > > > scenario
> > > > > >> > > > > > > > > > OPTIONAL will be called.
> > > > > >> > > > > > > > > > #3.3 Regarding, FILTER v/s
> > > > LIST_AUTHORIZED,
> > > > > isn't
> > > > > >> > > > > > > LIST_AUTHORIZED a
> > > > > >> > > > > > > > > > special case for "FILTER"?
> > > > > >> > > > > > > > > > #3.4 KafkaRequestContext.
> > > requestType()
> > > > v/s
> > > > > Action.
> > > > > >> > > > > > > > > authorizationMode. I
> > > > > >> > > > > > > > > > am not sure about the overlap
> or
> > > > ambiguity.
> > > > > >> > > > > > > > > > #4 - Cool. This is good, it
> will
> > > be less
> > > > > stress on
> > > > > >> the
> > > > > >> > > > > > > Authorizer.
> > > > > >> > > > > > > > > Ranger
> > > > > >> > > > > > > > > > already supports the "count"
> > > concept
> > > > and also
> > > > > has
> > > > > >> > > batching
> > > > > >> > > > > > > > > capability to
> > > > > >> > > > > > > > > > aggregate similar requests to
> > > reduce the
> > > > > number of
> > > > > >> audit
> > > > > >> > > logs
> > > > > >> > > > > > to
> > > > > >> > > > > > > > > write. We
> > > > > >> > > > > > > > > > should be able to pass this
> > > through.
> > > > > >> > > > > > > > > > #5 - Assuming if the object
> > > instance is
> > > > going
> > > > > out of
> > > > > >> > > scope, we
> > > > > >> > > > > > > > > should be
> > > > > >> > > > > > > > > > fine. Not a super important
> ask.
> > > Ranger
> > > > is
> > > > > already
> > > > > >> > > catching
> > > > > >> > > > > > KILL
> > > > > >> > > > > > > > > signal for
> > > > > >> > > > > > > > > > clean up.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Thanks again. These are good
> > > > enhancements.
> > > > > >> Appreciate
> > > > > >> > > your
> > > > > >> > > > > > > efforts
> > > > > >> > > > > > > > > here.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Bosco
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On 8/8/19, 2:03 AM, "Rajini
> > > Sivaram" <
> > > > > >> > > rajinisivaram@gmail.com
> > > > > >> > > > > > >
> > > > > >> > > > > > > > > wrote:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Hi Don,
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Thanks for reviewing the
> > KIP.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > 1. I had this originally
> as
> > a
> > > single
> > > > > Action, but
> > > > > >> > > thought it
> > > > > >> > > > > > > may
> > > > > >> > > > > > > > > be
> > > > > >> > > > > > > > > > useful
> > > > > >> > > > > > > > > > to support batched
> authorize
> > > calls
> > > > as
> > > > > well and
> > > > > >> keep
> > > > > >> > > it
> > > > > >> > > > > > > > > consistent with
> > > > > >> > > > > > > > > > other methods. Single
> > > requests can
> > > > contain
> > > > > >> multiple
> > > > > >> > > topics.
> > > > > >> > > > > > > For
> > > > > >> > > > > > > > > > example a
> > > > > >> > > > > > > > > > produce request can
> contain
> > > records
> > > > for
> > > > > several
> > > > > >> > > partitions
> > > > > >> > > > > > of
> > > > > >> > > > > > > > > different
> > > > > >> > > > > > > > > > topics. Broker could
> > > potentially
> > > > > authorize these
> > > > > >> > > together.
> > > > > >> > > > > > > For
> > > > > >> > > > > > > > > > SimpleAclAuthorizer,
> batched
> > > > authorize
> > > > > methods
> > > > > >> don't
> > > > > >> > > > > > provide
> > > > > >> > > > > > > > any
> > > > > >> > > > > > > > > > optimisation since lookup
> is
> > > based
> > > > on
> > > > > resources
> > > > > >> > > followed by
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > > > matching
> > > > > >> > > > > > > > > > logic. But some
> authorizers
> > > may
> > > > manage
> > > > > ACLs by
> > > > > >> user
> > > > > >> > > > > > principal
> > > > > >> > > > > > > > > rather
> > > > > >> > > > > > > > > > than
> > > > > >> > > > > > > > > > resource and may be able
> to
> > > optimize
> > > > > batched
> > > > > >> > > requests. I am
> > > > > >> > > > > > > ok
> > > > > >> > > > > > > > > with
> > > > > >> > > > > > > > > > using
> > > > > >> > > > > > > > > > single Action if this is
> > > likely to
> > > > cause
> > > > > issues.
> > > > > >> > > > > > > > > > 2. If you have two
> > listeners,
> > > one
> > > > for
> > > > > >> inter-broker
> > > > > >> > > traffic
> > > > > >> > > > > > > and
> > > > > >> > > > > > > > > another
> > > > > >> > > > > > > > > > for
> > > > > >> > > > > > > > > > external clients, start
> > > method is
> > > > invoked
> > > > > twice,
> > > > > >> > > once for
> > > > > >> > > > > > > each
> > > > > >> > > > > > > > > > listener. On
> > > > > >> > > > > > > > > > second thought, that may
> be
> > > > confusing and
> > > > > a
> > > > > >> single
> > > > > >> > > start()
> > > > > >> > > > > > > > > invocation
> > > > > >> > > > > > > > > > that
> > > > > >> > > > > > > > > > provides all listener
> > > information
> > > > and
> > > > > returns
> > > > > >> > > multiple
> > > > > >> > > > > > > futures
> > > > > >> > > > > > > > > would be
> > > > > >> > > > > > > > > > better. Will update the
> KIP.
> > > > > >> > > > > > > > > > 3. A typical example is a
> > > consumer
> > > > > subscribing
> > > > > >> to a
> > > > > >> > > regex
> > > > > >> > > > > > > > > pattern. We
> > > > > >> > > > > > > > > > request all topic metadata
> > > from the
> > > > > broker in
> > > > > >> order
> > > > > >> > > to
> > > > > >> > > > > > decide
> > > > > >> > > > > > > > > whether
> > > > > >> > > > > > > > > > the
> > > > > >> > > > > > > > > > pattern matches, expecting
> > to
> > > > receive a
> > > > > list of
> > > > > >> > > authorised
> > > > > >> > > > > > > > > topics. The
> > > > > >> > > > > > > > > > user
> > > > > >> > > > > > > > > > is not asking to subscribe
> > to
> > > an
> > > > > unauthorized
> > > > > >> topic.
> > > > > >> > > If
> > > > > >> > > > > > there
> > > > > >> > > > > > > > > are 10000
> > > > > >> > > > > > > > > > topics in the cluster and
> > the
> > > user
> > > > has
> > > > > access
> > > > > >> to 100
> > > > > >> > > of
> > > > > >> > > > > > them,
> > > > > >> > > > > > > > at
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > moment
> > > > > >> > > > > > > > > > we log 9900 DENIED log
> > > entries at
> > > > INFO
> > > > > level in
> > > > > >> > > > > > > > > SimpleAclAuthorizer.
> > > > > >> > > > > > > > > > The
> > > > > >> > > > > > > > > > proposal is to authorize
> > this
> > > > request with
> > > > > >> > > > > > > > > AuthorizationMode.FILTER, so
> > > > > >> > > > > > > > > > that authorizers can log
> > > resources
> > > > that
> > > > > are
> > > > > >> filtered
> > > > > >> > > out at
> > > > > >> > > > > > > > > lower level
> > > > > >> > > > > > > > > > like DEBUG since this is
> not
> > > an
> > > > attempt to
> > > > > >> access
> > > > > >> > > > > > > unauthorized
> > > > > >> > > > > > > > > > resources.
> > > > > >> > > > > > > > > > Brokers already handle
> these
> > > > differently
> > > > > since
> > > > > >> no
> > > > > >> > > > > > > authorization
> > > > > >> > > > > > > > > error
> > > > > >> > > > > > > > > > is
> > > > > >> > > > > > > > > > returned to the client in
> > > these
> > > > cases.
> > > > > Providing
> > > > > >> > > > > > > authorization
> > > > > >> > > > > > > > > mode to
> > > > > >> > > > > > > > > > authorizers enables
> > authorizer
> > > > > implementations
> > > > > >> to
> > > > > >> > > generate
> > > > > >> > > > > > > > > better audit
> > > > > >> > > > > > > > > > logs.
> > > > > >> > > > > > > > > > 4. Each request may
> contain
> > > multiple
> > > > > instances
> > > > > >> of
> > > > > >> > > the same
> > > > > >> > > > > > > > > authorizable
> > > > > >> > > > > > > > > > resource. For example a
> > > produce
> > > > request
> > > > > may
> > > > > >> contain
> > > > > >> > > records
> > > > > >> > > > > > > for
> > > > > >> > > > > > > > > 10
> > > > > >> > > > > > > > > > partitions of the same
> > topic.
> > > At the
> > > > > moment, we
> > > > > >> > > invoke
> > > > > >> > > > > > > > authorize
> > > > > >> > > > > > > > > > method 10
> > > > > >> > > > > > > > > > times. The proposal is to
> > > invoke it
> > > > once
> > > > > with
> > > > > >> > > count=10. The
> > > > > >> > > > > > > > > count is
> > > > > >> > > > > > > > > > provided to authorizer
> just
> > > for
> > > > audit
> > > > > logging
> > > > > >> > > purposes.
> > > > > >> > > > > > > > > > 5. Authorizer implements
> > > Closeable,
> > > > so you
> > > > > >> could use
> > > > > >> > > > > > close()
> > > > > >> > > > > > > to
> > > > > >> > > > > > > > > flush
> > > > > >> > > > > > > > > > audits?
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > On Thu, Aug 8, 2019 at
> 7:01
> > > AM Don
> > > > Bosco
> > > > > Durai <
> > > > > >> > > > > > > > bosco@apache.org
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > Rajini
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Thanks for putting this
> > > together.
> > > > It is
> > > > > >> looking
> > > > > >> > > good. I
> > > > > >> > > > > > > have
> > > > > >> > > > > > > > > few
> > > > > >> > > > > > > > > > > questions...
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > 1.
> > List<AuthorizationResult>
> > > > > authorize(...,
> > > > > >> > > List<Action>
> > > > > >> > > > > > > > > actions).
> > > > > >> > > > > > > > > > Do you
> > > > > >> > > > > > > > > > > see a scenario where the
> > > broker
> > > > will
> > > > > call
> > > > > >> > > authorize for
> > > > > >> > > > > > > > > multiple
> > > > > >> > > > > > > > > > topics at
> > > > > >> > > > > > > > > > > the same time? I can
> > > understand
> > > > that
> > > > > during
> > > > > >> > > > > > > creating/deleting
> > > > > >> > > > > > > > > ACLS,
> > > > > >> > > > > > > > > > > multiple permissions for
> > > multiple
> > > > > resources
> > > > > >> might
> > > > > >> > > be
> > > > > >> > > > > > done.
> > > > > >> > > > > > > > For
> > > > > >> > > > > > > > > > authorize
> > > > > >> > > > > > > > > > > call, would this be a
> > case?
> > > And
> > > > does the
> > > > > >> Authorize
> > > > > >> > > > > > > > > implementation
> > > > > >> > > > > > > > > > will be
> > > > > >> > > > > > > > > > > able to do performance
> > > > optimization
> > > > > because of
> > > > > >> > > this? Or
> > > > > >> > > > > > > > should
> > > > > >> > > > > > > > > we
> > > > > >> > > > > > > > > > just keep
> > > > > >> > > > > > > > > > > it simple? I don't see
> it
> > > as an
> > > > issue
> > > > > from
> > > > > >> Apache
> > > > > >> > > Ranger
> > > > > >> > > > > > > > side,
> > > > > >> > > > > > > > > but
> > > > > >> > > > > > > > > > just
> > > > > >> > > > > > > > > > > checking to see whether
> we
> > > need
> > > > to be
> > > > > aware of
> > > > > >> > > something.
> > > > > >> > > > > > > > > > > 2. Should I assume that
> > the
> > > > > SecurityProtocol
> > > > > >> passed
> > > > > >> > > > > > during
> > > > > >> > > > > > > > > start and
> > > > > >> > > > > > > > > > the
> > > > > >> > > > > > > > > > > one return by
> > > > > >> > > KafkaRequestContext.securityProtocol() will
> > > > > >> > > > > > > be
> > > > > >> > > > > > > > > the
> > > > > >> > > > > > > > > > same?
> > > > > >> > > > > > > > > > > CompletableFuture<Void>
> > > > start(String
> > > > > >> listenerName,
> > > > > >> > > > > > > > > SecurityProtocol
> > > > > >> > > > > > > > > > > securityProtocol);
> > > > > >> > > > > > > > > > >
> > > > KafkaRequestContext.securityProtocol()
> > > > > >> > > > > > > > > > > 3. What is the purpose
> of
> > > > > AuthorizationMode?
> > > > > >> How
> > > > > >> > > does the
> > > > > >> > > > > > > > > broker
> > > > > >> > > > > > > > > > decide
> > > > > >> > > > > > > > > > > what mode to use when
> the
> > > > authorize()
> > > > > method
> > > > > >> is
> > > > > >> > > called?
> > > > > >> > > > > > > > > > > 4. Can we clarify
> "count"
> > in
> > > > Action a
> > > > > bit
> > > > > >> more?
> > > > > >> > > How is it
> > > > > >> > > > > > > > used?
> > > > > >> > > > > > > > > > > 5. Do you feel having
> > "stop"
> > > > along with
> > > > > >> "start" be
> > > > > >> > > > > > helpful?
> > > > > >> > > > > > > > > E.g. In
> > > > > >> > > > > > > > > > Ranger
> > > > > >> > > > > > > > > > > we try to optimize the
> > Audit
> > > > writing by
> > > > > >> caching
> > > > > >> > > the logs
> > > > > >> > > > > > > for
> > > > > >> > > > > > > > a
> > > > > >> > > > > > > > > fixed
> > > > > >> > > > > > > > > > > interval. But when the
> > > Broker
> > > > > terminates, we
> > > > > >> do a
> > > > > >> > > forced
> > > > > >> > > > > > > > flush.
> > > > > >> > > > > > > > > > Having an
> > > > > >> > > > > > > > > > > explicit "stop" might
> give
> > > us a
> > > > formal
> > > > > way to
> > > > > >> > > flush our
> > > > > >> > > > > > > > audits.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Thanks
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Bosco
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > On 8/7/19, 3:59 PM,
> > "Rajini
> > > > Sivaram" <
> > > > > >> > > > > > > > rajinisivaram@gmail.com
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Hi
> Ron/Harsha/Satish,
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Thanks for reviewing
> > > the KIP!
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > We should perhaps
> have
> > > a wider
> > > > > discussion
> > > > > >> > > outside
> > > > > >> > > > > > this
> > > > > >> > > > > > > > KIP
> > > > > >> > > > > > > > > for
> > > > > >> > > > > > > > > > > refactoring
> > > > > >> > > > > > > > > > > clients so that
> others
> > > who
> > > > are not
> > > > > >> following
> > > > > >> > > this KIP
> > > > > >> > > > > > > > also
> > > > > >> > > > > > > > > > notice the
> > > > > >> > > > > > > > > > > discussion. Satish,
> > > would you
> > > > like
> > > > > to
> > > > > >> start a
> > > > > >> > > > > > > discussion
> > > > > >> > > > > > > > > thread
> > > > > >> > > > > > > > > > on dev?
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Regards,
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Rajini
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > On Wed, Aug 7, 2019
> at
> > > 6:21 PM
> > > > > Satish
> > > > > >> Duggana <
> > > > > >> > > > > > > > > > >
> satish.duggana@gmail.com>
> > > > > >> > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > > I felt the same
> need
> > > when
> > > > we want
> > > > > to
> > > > > >> add a
> > > > > >> > > > > > pluggable
> > > > > >> > > > > > > > API
> > > > > >> > > > > > > > > for
> > > > > >> > > > > > > > > > core
> > > > > >> > > > > > > > > > > > server
> > functionality.
> > > This
> > > > does
> > > > > not
> > > > > >> need to
> > > > > >> > > be part
> > > > > >> > > > > > > of
> > > > > >> > > > > > > > > this
> > > > > >> > > > > > > > > > KIP, it
> > > > > >> > > > > > > > > > > > can be a separate
> > > KIP. I can
> > > > > contribute
> > > > > >> those
> > > > > >> > > > > > > > refactoring
> > > > > >> > > > > > > > > > changes if
> > > > > >> > > > > > > > > > > > others are OK with
> > > that.
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > It is better to
> > have a
> > > > structure
> > > > > like
> > > > > >> below.
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > kafka-common:
> > > > > >> > > > > > > > > > > > common classes
> which
> > > can be
> > > > used
> > > > > in any
> > > > > >> of
> > > > > >> > > the
> > > > > >> > > > > > other
> > > > > >> > > > > > > > > modules
> > > > > >> > > > > > > > > > in Kafka
> > > > > >> > > > > > > > > > > > like client,
> > > > Kafka-server-common
> > > > > and
> > > > > >> server
> > > > > >> > > etc.
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> kafka-client-common:
> > > > > >> > > > > > > > > > > > common classes
> which
> > > can be
> > > > used
> > > > > in the
> > > > > >> > > client
> > > > > >> > > > > > > module.
> > > > > >> > > > > > > > > This
> > > > > >> > > > > > > > > > can be
> > > > > >> > > > > > > > > > > > part of client
> > module
> > > > itself.
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> kafka-server-common:
> > > > > >> > > > > > > > > > > > classes required
> > only
> > > for
> > > > > kafka-server.
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Thanks.
> > > > > >> > > > > > > > > > > > Satish.
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > On Wed, Aug 7,
> 2019
> > > at 9:28
> > > > PM
> > > > > Harsha
> > > > > >> > > Chintalapani
> > > > > >> > > > > > <
> > > > > >> > > > > > > > > > kafka@harsha.io>
> > > > > >> > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Thanks for the
> KIP
> > > Rajini.
> > > > > >> > > > > > > > > > > > > Quick thought,
> it
> > > would
> > > > be good
> > > > > to
> > > > > >> have a
> > > > > >> > > common
> > > > > >> > > > > > > > module
> > > > > >> > > > > > > > > > outside of
> > > > > >> > > > > > > > > > > > clients
> > > > > >> > > > > > > > > > > > > that only
> applies
> > to
> > > > server side
> > > > > >> > > interfaces &
> > > > > >> > > > > > > > changes.
> > > > > >> > > > > > > > > It
> > > > > >> > > > > > > > > > looks
> > > > > >> > > > > > > > > > > like we
> > > > > >> > > > > > > > > > > > are
> > > > > >> > > > > > > > > > > > > increasingly in
> > > favor of
> > > > using
> > > > > Java
> > > > > >> > > interface for
> > > > > >> > > > > > > > > pluggable
> > > > > >> > > > > > > > > > > modules on
> > > > > >> > > > > > > > > > > > the
> > > > > >> > > > > > > > > > > > > broker side.
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > Thanks,
> > > > > >> > > > > > > > > > > > > Harsha
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > On Tue, Aug 06,
> > > 2019 at
> > > > 2:31 PM,
> > > > > >> Rajini
> > > > > >> > > Sivaram <
> > > > > >> > > > > > > > > > > rajinisivaram@gmail.com
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Hi all,
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > I have
> created a
> > > KIP to
> > > > > replace the
> > > > > >> Scala
> > > > > >> > > > > > > > Authorizer
> > > > > >> > > > > > > > > API
> > > > > >> > > > > > > > > > with a
> > > > > >> > > > > > > > > > > new
> > > > > >> > > > > > > > > > > > Java
> > > > > >> > > > > > > > > > > > > > API:
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > -
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > This is
> > > replacement for
> > > > KIP-50
> > > > > >> which was
> > > > > >> > > > > > accepted
> > > > > >> > > > > > > > > but never
> > > > > >> > > > > > > > > > > merged.
> > > > > >> > > > > > > > > > > > Apart
> > > > > >> > > > > > > > > > > > > > from moving
> to a
> > > Java
> > > > API
> > > > > consistent
> > > > > >> > > with other
> > > > > >> > > > > > > > > pluggable
> > > > > >> > > > > > > > > > > interfaces
> > > > > >> > > > > > > > > > > > in the
> > > > > >> > > > > > > > > > > > > > broker,
> KIP-504
> > > also
> > > > attempts
> > > > > to
> > > > > >> address
> > > > > >> > > known
> > > > > >> > > > > > > > > limitations
> > > > > >> > > > > > > > > > in the
> > > > > >> > > > > > > > > > > > > > authorizer. If
> > > you have
> > > > come
> > > > > across
> > > > > >> other
> > > > > >> > > > > > > > > limitations that
> > > > > >> > > > > > > > > > you
> > > > > >> > > > > > > > > > > would
> > > > > >> > > > > > > > > > > > like
> > > > > >> > > > > > > > > > > > > > to see
> addressed
> > > in the
> > > > new
> > > > > API,
> > > > > >> please
> > > > > >> > > raise
> > > > > >> > > > > > > these
> > > > > >> > > > > > > > > on the
> > > > > >> > > > > > > > > > > discussion
> > > > > >> > > > > > > > > > > > > > thread so that
> > we
> > > can
> > > > > consider those
> > > > > >> > > too. All
> > > > > >> > > > > > > > > suggestions
> > > > > >> > > > > > > > > > and
> > > > > >> > > > > > > > > > > feedback
> > > > > >> > > > > > > > > > > > are
> > > > > >> > > > > > > > > > > > > > welcome.
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Thank you,
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > > > Rajini
> > > > > >> > > > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Jun Rao <ju...@confluent.io>.
Hi, Rajini,
Thanks for the KIP. I was concerned about #4 too. If we change the handling
of all requests to use an async authorize() api, will that cause the code
much harder to understand? There are quite a few callbacks already. I am
not sure that we want to introduce more of those. The benefit from async
authorize() api seems limited.
Jun
On Wed, Sep 4, 2019 at 5:38 PM Rajini Sivaram <ra...@gmail.com>
wrote:
> Hi Don,
>
> Thanks for your note.
>
> 1) The intention is to avoid blocking in the calling thread. We already
> have several requests that are put into a purgatory when waiting for remote
> communication, for example produce request waiting for replication. Since
> we have a limited number of request threads in the broker, we want to make
> progress with other requests, while one is waiting on any form of remote
> communication.
>
> 2) Async management calls will be useful with the default authorizer when
> KIP-500 removes ZK and we rely on Kafka instead. Our current ZK-based
> implementation as well as any custom implementations that don't want to be
> async will just need to return a sync'ed value. So instead of returning `
> value`, the code would just return
> `CompletableFuture.completedFuture(value)
> `. So it would be just a single line change in the implementation with the
> new API. The caller would treat completedFuture exactly as it does today,
> processing the request synchronously without using a purgatory.
>
> 3) For implementations that return a completedFuture as described in 2),
> the behaviour would remain exactly the same. No additional threads or
> purgatory will be used for this case. So there would be no performance
> penalty. For implementations that return a future that is not complete, we
> prioritise running more requests concurrently. So in a deployment with a
> large number of clients, we would improve performance by allowing other
> requests to be processed on the request threads while some are waiting for
> authorization metadata.
>
> 4) I was concerned about this too. The goal is to make the API flexible
> enough to handle large scale deployments in future when caching all
> authorization metadata in each broker is not viable. Using an async API
> that returns CompletionStage, the caller has the option to handle the
> result synchronously or asynchronously, so we don't necessarily need to
> update the calling code right away. Custom authorizers using the async API
> have full control over whether authorization is performed in-line since
> completedFuture will always be handled synchronously. We do need to update
> KafkaApis to take advantage of the asynchronous API to improve scale. Even
> though this is a big change, since we will be doing the same for all
> requests, it shouldn't be too hard to maintain since the same pattern will
> be used for all requests.
>
> Regards,
>
> Rajini
>
> On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <bo...@apache.org> wrote:
>
> > Hi Rajini
> >
> > Help me understand this a bit more.
> >
> > 1. For all practical purpose, without authorization you can't go to the
> > next step. The calling code needs to block anyway. So should we just let
> > the implementation code do the async part?
> > 2. If you feel management calls need to be async, then we should consider
> > another set of async APIs. I don't feel we should complicate it further (
> > 3. Another concern I have is wrt performance. Kafka has been built to
> > handle 1000s per second requests. Not sure whether making it async will
> add
> > any unnecessary overhead.
> > 4. How much complication would this add on the calling side? And is it
> > worth it?
> >
> > Thanks
> >
> > Bosco
> >
> >
> > On 9/3/19, 8:50 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
> >
> > Hi all,
> >
> > Ismael brought up a point that it will be good to make the Authorizer
> > interface asynchronous to avoid blocking request threads during
> remote
> > operations.
> >
> > 1) Since we want to support different backends for authorization
> > metadata,
> > making createAcls() and deleteAcls() asynchronous makes sense since
> > these
> > always involve remote operations. When KIP-500 removes ZooKeeper, we
> > would
> > want to move ACLs to Kafka and async updates will avoid unnecessary
> > blocking.
> > 2) For authorize() method, we currently use cached ACLs in the
> built-in
> > authorizers, so synchronous authorise operations work well now. But
> > async
> > authorize() would support this scenario as well as authorizers in
> large
> > organisations where an LRU cache would enable a smaller cache even
> > when the
> > backend holds a large amount of ACLs for infrequently used resources
> or
> > users who don't use the system frequently.
> >
> > For both cases, the built-in authorizer will continue to be
> > synchronous,
> > using CompletableFuture.completedFuture() to return the actual
> result.
> > But
> > async API will make custom authorizer implementations more flexible.
> I
> > would like to know if there are any concerns with these changes
> before
> > updating the KIP.
> >
> > *Proposed API:*
> > public interface Authorizer extends Configurable, Closeable {
> >
> > Map<Endpoint, CompletionStage<Void>> start(AuthorizerServerInfo
> > serverInfo);
> > List<CompletionStage<AuthorizationResult>>
> > authorize(AuthorizableRequestContext requestContext, List<Action>
> > actions);
> > List<CompletionStage<AclCreateResult>>
> > createAcls(AuthorizableRequestContext requestContext,
> List<AclBinding>
> > aclBindings);
> > List<CompletionStage<AclDeleteResult>>
> > deleteAcls(AuthorizableRequestContext requestContext,
> > List<AclBindingFilter> aclBindingFilters);
> > CompletionStage<Collection<AclBinding>> acls(AclBindingFilter
> > filter);
> > }
> >
> >
> > Thank you,
> >
> > Rajini
> >
> > On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <bo...@apache.org>
> > wrote:
> >
> > > Hi Rajini
> > >
> > > Thanks for clarifying. I am good for now.
> > >
> > > Regards
> > >
> > > Bosco
> > >
> > >
> > > On 8/16/19, 11:30 AM, "Rajini Sivaram" <ra...@gmail.com>
> > wrote:
> > >
> > > Hi Don,
> > >
> > > That should be fine. I guess Ranger loads policies from the
> > database
> > > synchronously when the authorizer is configured, similar to
> > > SimpleAclAuthorizer loading from ZooKeeper. Ranger can continue
> > to load
> > > synchronously from `configure()` or `start()` and return an
> > empty map
> > > from
> > > `start()`. That would retain the existing behaviour.. When the
> > same
> > > database stores policies for all listeners and the policies are
> > not
> > > stored
> > > in Kafka, there is no value in making the load asynchronous.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai <
> > bosco@apache.org>
> > > wrote:
> > >
> > > > Hi Rajini
> > > >
> > > > Assuming this doesn't affect custom plugins, I don't have any
> > > concerns
> > > > regarding this.
> > > >
> > > > I do have one question regarding:
> > > >
> > > > "For authorizers that don’t store metadata in ZooKeeper,
> > ensure that
> > > > authorizer metadata for each listener is available before
> > starting
> > > up the
> > > > listener. This enables different authorization metadata
> stores
> > for
> > > > different listeners."
> > > >
> > > > Since Ranger uses its own database for storing policies, do
> you
> > > anticipate
> > > > any issues?
> > > >
> > > > Thanks
> > > >
> > > > Bosco
> > > >
> > > >
> > > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
> > rajinisivaram@gmail.com>
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I made another change to the KIP. The KIP was originally
> > > proposing to
> > > > extend SimpleAclAuthorizer to also implement the new API
> > (in
> > > addition
> > > > to
> > > > the existing API). But since we use the new API when
> > available,
> > > this
> > > > can
> > > > break custom authorizers that extend this class and
> > override
> > > specific
> > > > methods of the old API. To avoid breaking any existing
> > custom
> > > > implementations that extend this class, particularly
> > because it
> > > is in
> > > > the
> > > > public package kafka.security.auth, the KIP now proposes
> to
> > > retain the
> > > > old
> > > > authorizer as-is. SimpleAclAuthorizer will continue to
> > > implement the
> > > > old
> > > > API, but will be deprecated. A new authorizer
> > implementation
> > > > kafka.security.authorizer.AclAuthorizer will be added for
> > the
> > > new API
> > > > (this
> > > > will not be in the public package).
> > > >
> > > > Please let me know if you have any concerns.
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
> > > > rajinisivaram@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Colin.
> > > > >
> > > > > If there are no other concerns, I will start vote later
> > today.
> > > Many
> > > > thanks
> > > > > to every one for the feedback.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > > >
> > > > > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <
> > > cmccabe@apache.org>
> > > > wrote:
> > > > >
> > > > >> Thanks, Rajini. It looks good to me.
> > > > >>
> > > > >> best,
> > > > >> Colin
> > > > >>
> > > > >>
> > > > >> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram wrote:
> > > > >> > Hi Colin,
> > > > >> >
> > > > >> > Thanks for the review. I have updated the KIP to
> move
> > the
> > > > interfaces for
> > > > >> > request context and server info to the authorizer
> > package.
> > > These
> > > > are now
> > > > >> > called AuthorizableRequestContext and
> > AuthorizerServerInfo.
> > > > Endpoint is
> > > > >> now
> > > > >> > a class in org.apache.kafka.common to make it
> reusable
> > > since we
> > > > already
> > > > >> > have multiple implementations of it. I have removed
> > > requestName
> > > > from the
> > > > >> > request context interface since authorizers can
> > distinguish
> > > > follower
> > > > >> fetch
> > > > >> > and consumer fetch from the operation being
> > authorized. So
> > > 16-bit
> > > > >> request
> > > > >> > type should be sufficient for audit logging. Also
> > replaced
> > > > AuditFlag
> > > > >> with
> > > > >> > two booleans as you suggested.
> > > > >> >
> > > > >> > Can you take another look and see if the KIP is
> ready
> > for
> > > voting?
> > > > >> >
> > > > >> > Thanks for all your help!
> > > > >> >
> > > > >> > Regards,
> > > > >> >
> > > > >> > Rajini
> > > > >> >
> > > > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <
> > > cmccabe@apache.org>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Hi Rajini,
> > > > >> > >
> > > > >> > > I think it would be good to rename
> > KafkaRequestContext to
> > > > something
> > > > >> like
> > > > >> > > AuthorizableRequestContext, and put it in the
> > > > >> > > org.apache.kafka.server.authorizer namespace. If
> > we put
> > > it in
> > > > the
> > > > >> > > org.apache.kafka.common namespace, then it's not
> > really
> > > clear
> > > > that
> > > > >> it's
> > > > >> > > part of the Authorizer API. Since this class is
> > purely an
> > > > interface,
> > > > >> with
> > > > >> > > no concrete implementation of anything, there's
> > nothing
> > > common
> > > > to
> > > > >> really
> > > > >> > > reuse in any case. We definitely don't want
> > someone to
> > > > accidentally
> > > > >> add or
> > > > >> > > remove methods because they think this is just
> > another
> > > internal
> > > > class
> > > > >> used
> > > > >> > > for requests.
> > > > >> > >
> > > > >> > > The BrokerInfo class is a nice improvement. It
> > looks
> > > like it
> > > > will be
> > > > >> > > useful for passing in information about the
> context
> > we're
> > > > running
> > > > >> in. It
> > > > >> > > would be nice to call this class ServerInfo rather
> > than
> > > > BrokerInfo,
> > > > >> since
> > > > >> > > we will want to run the authorizer on controllers
> > as well
> > > as on
> > > > >> brokers,
> > > > >> > > and the controller may run as a separate process
> > post
> > > KIP-500.
> > > > I also
> > > > >> > > think that this class should be in the
> > > > >> org.apache.kafka.server.authorizer
> > > > >> > > namespace. Again, it is an interface, not a
> > concrete
> > > > implementation,
> > > > >> and
> > > > >> > > it's an interface that is very specifically for
> the
> > > authorizer.
> > > > >> > >
> > > > >> > > I agree that we probably don't have enough
> > information
> > > > preserved for
> > > > >> > > requests currently to always know what entity made
> > them.
> > > So we
> > > > can
> > > > >> leave
> > > > >> > > that out for now (except in the special case of
> > Fetch).
> > > > Perhaps we
> > > > >> can add
> > > > >> > > this later if it's needed.
> > > > >> > >
> > > > >> > > I understand the intention behind
> AuthorizationMode
> > > (which is
> > > > now
> > > > >> called
> > > > >> > > AuditFlag in the latest revision). But it still
> > feels
> > > > complex. What
> > > > >> if we
> > > > >> > > just had two booleans in Action: logSuccesses and
> > > logFailures?
> > > > That
> > > > >> seems
> > > > >> > > to cover all the cases here. MANDATORY_AUTHORIZE
> =
> > true,
> > > true.
> > > > >> > > OPTIONAL_AUTHORIZE = true, false. FILTER = true,
> > false.
> > > > >> LIST_AUTHORIZED =
> > > > >> > > false, false. Would there be anything lost versus
> > having
> > > the
> > > > enum?
> > > > >> > >
> > > > >> > > best,
> > > > >> > > Colin
> > > > >> > >
> > > > >> > >
> > > > >> > > On Wed, Aug 14, 2019, at 06:29, Mickael Maison
> > wrote:
> > > > >> > > > Hi Rajini,
> > > > >> > > >
> > > > >> > > > Thanks for the KIP!
> > > > >> > > > I really like that authorize() will be able to
> > take a
> > > batch of
> > > > >> > > > requests, this will speed up many
> implementations!
> > > > >> > > >
> > > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <
> > > > >> rajinisivaram@gmail.com>
> > > > >> > > wrote:
> > > > >> > > > >
> > > > >> > > > > Thanks David! I have fixed the typo.
> > > > >> > > > >
> > > > >> > > > > Also made a couple of changes to make the
> > context
> > > > interfaces more
> > > > >> > > generic.
> > > > >> > > > > KafkaRequestContext now returns the 16-bit API
> > key as
> > > Colin
> > > > >> suggested
> > > > >> > > as
> > > > >> > > > > well as the friendly name used in metrics
> which
> > are
> > > useful
> > > > in
> > > > >> audit
> > > > >> > > logs.
> > > > >> > > > > `Authorizer#start` is now provided a generic
> > > `BrokerInfo`
> > > > >> interface
> > > > >> > > that
> > > > >> > > > > gives cluster id, broker id and endpoint
> > information.
> > > The
> > > > generic
> > > > >> > > interface
> > > > >> > > > > can potentially be used in other broker
> plugins
> > in
> > > future
> > > > and
> > > > >> provides
> > > > >> > > > > dynamically generated configs like broker id
> > and ports
> > > > which are
> > > > >> > > currently
> > > > >> > > > > not available to plugins unless these configs
> > are
> > > statically
> > > > >> > > configured.
> > > > >> > > > > Please let me know if there are any concerns.
> > > > >> > > > >
> > > > >> > > > > Regards,
> > > > >> > > > >
> > > > >> > > > > Rajini
> > > > >> > > > >
> > > > >> > > > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot <
> > > > djacot@confluent.io>
> > > > >> > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Hi Rajini,
> > > > >> > > > > >
> > > > >> > > > > > Thank you for the update! It looks good to
> me.
> > > There is a
> > > > typo
> > > > >> in the
> > > > >> > > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` ->
> > > > >> `MANDATORY_AUTHORIZE`.
> > > > >> > > > > >
> > > > >> > > > > > Regards,
> > > > >> > > > > > David
> > > > >> > > > > >
> > > > >> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini
> > Sivaram <
> > > > >> > > rajinisivaram@gmail.com>
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > Hi David,
> > > > >> > > > > > >
> > > > >> > > > > > > Thanks for reviewing the KIP! Since
> > questions
> > > about
> > > > >> `authorization
> > > > >> > > mode`
> > > > >> > > > > > > and `count` have come up multiple times, I
> > have
> > > renamed
> > > > both.
> > > > >> > > > > > >
> > > > >> > > > > > > 1) Renamed `count` to
> > `resourceReferenceCount`.
> > > It is
> > > > the
> > > > >> number
> > > > >> > > of times
> > > > >> > > > > > > the resource being authorized is
> referenced
> > > within the
> > > > >> request.
> > > > >> > > > > > >
> > > > >> > > > > > > 2) Renamed `AuthorizationMode` to
> > `AuditFlag`. It
> > > is
> > > > provided
> > > > >> to
> > > > >> > > improve
> > > > >> > > > > > > audit logging in the authorizer. The enum
> > values
> > > have
> > > > javadoc
> > > > >> which
> > > > >> > > > > > > indicate how the authorization result is
> > used in
> > > each
> > > > of the
> > > > >> modes
> > > > >> > > to
> > > > >> > > > > > > enable authorizers to log audit messages
> at
> > the
> > > right
> > > > severity
> > > > >> > > level.
> > > > >> > > > > > >
> > > > >> > > > > > > Regards,
> > > > >> > > > > > >
> > > > >> > > > > > > Rajini
> > > > >> > > > > > >
> > > > >> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM David
> Jacot
> > <
> > > > >> djacot@confluent.io>
> > > > >> > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > Hi Rajini,
> > > > >> > > > > > > >
> > > > >> > > > > > > > Thank you for the KIP. Overall, it looks
> > good
> > > to me.
> > > > I have
> > > > >> few
> > > > >> > > > > > > > questions/suggestions:
> > > > >> > > > > > > >
> > > > >> > > > > > > > 1. It is hard to grasp what
> > `Action#count` is
> > > for. I
> > > > guess I
> > > > >> > > understand
> > > > >> > > > > > > > where you want to go with it but it took
> > me a
> > > while to
> > > > >> figure it
> > > > >> > > out.
> > > > >> > > > > > > > Perhaps, we could come up with a better
> > name
> > > than
> > > > `count`?
> > > > >> > > > > > > >
> > > > >> > > > > > > > 2. I had a hard time trying to
> understand
> > the
> > > > >> `AuthorizationMode`
> > > > >> > > > > > > concept,
> > > > >> > > > > > > > especially wrt. the OPTIONAL one. My
> > > understanding is
> > > > that
> > > > >> an
> > > > >> > > ACL is
> > > > >> > > > > > > either
> > > > >> > > > > > > > defined or not. Could you elaborate a
> bit
> > more
> > > on
> > > > that?
> > > > >> > > > > > > >
> > > > >> > > > > > > > Thanks,
> > > > >> > > > > > > > David
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don
> Bosco
> > Durai
> > > <
> > > > >> > > bosco@apache.org>
> > > > >> > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Hi Rajini
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > 3.2 - This makes sense. Thanks for
> > clarifying.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Rest looks fine. Once the
> > implementations are
> > > done,
> > > > it
> > > > >> will be
> > > > >> > > more
> > > > >> > > > > > > clear
> > > > >> > > > > > > > > on the different values RequestType
> and
> > Mode.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thanks
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Bosco
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram"
> <
> > > > >> rajinisivaram@gmail.com
> > > > >> > > >
> > > > >> > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Hi Don,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thanks for the suggestions. A few
> > > responses
> > > > below:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > 3.1 Can rename and improve docs if
> > we keep
> > > > this. Let's
> > > > >> > > finish the
> > > > >> > > > > > > > > discussion on Colin's suggestions
> > > regarding this
> > > > >> first.
> > > > >> > > > > > > > > 3.2 No, I was thinking of some
> > requests
> > > that
> > > > have an
> > > > >> old
> > > > >> > > way of
> > > > >> > > > > > > > > authorizing
> > > > >> > > > > > > > > and a new way where we have
> > retained the
> > > old
> > > > way for
> > > > >> > > backward
> > > > >> > > > > > > > > compatibility. One example is
> > > Cluster:Create
> > > > >> permission to
> > > > >> > > create
> > > > >> > > > > > > > > topics.
> > > > >> > > > > > > > > We have replaced this with
> > fine-grained
> > > topic
> > > > create
> > > > >> > > access using
> > > > >> > > > > > > > > Topic:Create
> > > > >> > > > > > > > > for topic patterns. But we still
> > check if
> > > user
> > > > has
> > > > >> > > Cluster:Create
> > > > >> > > > > > > > > first. If
> > > > >> > > > > > > > > Denied, the deny is ignored and we
> > check
> > > > >> Topic:Create. We
> > > > >> > > dont
> > > > >> > > > > > want
> > > > >> > > > > > > > to
> > > > >> > > > > > > > > log
> > > > >> > > > > > > > > DENY for Cluster:Create at INFO
> > level for
> > > this,
> > > > since
> > > > >> this
> > > > >> > > is
> > > > >> > > > > > not a
> > > > >> > > > > > > > > mandatory ACL for creating topics.
> > We
> > > will get
> > > > >> appropriate
> > > > >> > > logs
> > > > >> > > > > > > from
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > subsequent Topic:Create in this
> > case.
> > > > >> > > > > > > > > 3.3 They are not quite the same.
> > FILTER
> > > implies
> > > > that
> > > > >> user
> > > > >> > > > > > actually
> > > > >> > > > > > > > > requested access to and performed
> > some
> > > > operation on
> > > > >> the
> > > > >> > > filtered
> > > > >> > > > > > > > > resources.
> > > > >> > > > > > > > > LIST_AUTHORZED did not result in
> any
> > > actual
> > > > access.
> > > > >> User
> > > > >> > > simply
> > > > >> > > > > > > > wanted
> > > > >> > > > > > > > > to
> > > > >> > > > > > > > > know what they are allowed to
> > access.
> > > > >> > > > > > > > > 3.4 Request types are Produce,
> > JoinGroup,
> > > > OffsetCommit
> > > > >> > > etc. So
> > > > >> > > > > > that
> > > > >> > > > > > > > is
> > > > >> > > > > > > > > different from authorization mode,
> > > operation
> > > > etc.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > On Thu, Aug 8, 2019 at 11:36 PM
> Don
> > Bosco
> > > Durai
> > > > <
> > > > >> > > > > > bosco@apache.org>
> > > > >> > > > > > > > > wrote:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > Hi Rajini
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Thanks for clarifying. This is
> > very
> > > helpful...
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > #1 - On the Ranger side, we
> > should be
> > > able to
> > > > handle
> > > > >> > > multiple
> > > > >> > > > > > > > > requests at
> > > > >> > > > > > > > > > the same time. I was just not
> > sure how
> > > much
> > > > >> processing
> > > > >> > > overhead
> > > > >> > > > > > > > will
> > > > >> > > > > > > > > be
> > > > >> > > > > > > > > > there on the Broker side to
> split
> > and
> > > then
> > > > >> consolidate
> > > > >> > > the
> > > > >> > > > > > > results.
> > > > >> > > > > > > > > If it
> > > > >> > > > > > > > > > is negligible, then this is the
> > better
> > > way.
> > > > It will
> > > > >> > > make it
> > > > >> > > > > > > future
> > > > >> > > > > > > > > proof.
> > > > >> > > > > > > > > > #2 - I agree, having a single
> > "start"
> > > call
> > > > makes it
> > > > >> > > cleaner.
> > > > >> > > > > > The
> > > > >> > > > > > > > > > Authorization implementation
> will
> > only
> > > have
> > > > to do
> > > > >> the
> > > > >> > > > > > > > initialization
> > > > >> > > > > > > > > only
> > > > >> > > > > > > > > > once.
> > > > >> > > > > > > > > > #3.1 - Thanks for the
> > clarification. I
> > > think
> > > > it
> > > > >> makes
> > > > >> > > sense to
> > > > >> > > > > > > have
> > > > >> > > > > > > > > this.
> > > > >> > > > > > > > > > The term "Mode" might not be
> > explicit
> > > enough.
> > > > >> > > Essentially it
> > > > >> > > > > > > seems
> > > > >> > > > > > > > > you want
> > > > >> > > > > > > > > > the Authorizer to know the
> > > Intent/Purpose of
> > > > the
> > > > >> > > authorize call
> > > > >> > > > > > > and
> > > > >> > > > > > > > > let the
> > > > >> > > > > > > > > > Authorizer decide what to log as
> > Audit
> > > event.
> > > > >> Changing
> > > > >> > > the
> > > > >> > > > > > > > > class/field name
> > > > >> > > > > > > > > > or giving more documentation
> will
> > do.
> > > > >> > > > > > > > > > #3.2 - Regarding the option
> > "OPTIONAL".
> > > Are
> > > > you
> > > > >> thinking
> > > > >> > > from
> > > > >> > > > > > > > > chaining
> > > > >> > > > > > > > > > multiple Authorizer? If so, I
> am
> > not
> > > sure
> > > > whether
> > > > >> the
> > > > >> > > Broker
> > > > >> > > > > > > would
> > > > >> > > > > > > > > have
> > > > >> > > > > > > > > > enough information to make that
> > > decision. I
> > > > feel the
> > > > >> > > Authorizer
> > > > >> > > > > > > > will
> > > > >> > > > > > > > > be the
> > > > >> > > > > > > > > > one who would have that
> > knowledge. E.g.
> > > in
> > > > Ranger
> > > > >> we have
> > > > >> > > > > > > explicit
> > > > >> > > > > > > > > deny,
> > > > >> > > > > > > > > > which means no matter what, the
> > request
> > > > should be
> > > > >> denied
> > > > >> > > for
> > > > >> > > > > > the
> > > > >> > > > > > > > > user/group
> > > > >> > > > > > > > > > or condition. So if you are
> > thinking of
> > > > chaining
> > > > >> > > Authorizers,
> > > > >> > > > > > > then
> > > > >> > > > > > > > > > responses should have the third
> > state,
> > > e.g.
> > > > >> > > "DENIED_FINAL", in
> > > > >> > > > > > > > which
> > > > >> > > > > > > > > case
> > > > >> > > > > > > > > > if there is an Authorization
> > chain, it
> > > will
> > > > be stop
> > > > >> and
> > > > >> > > the
> > > > >> > > > > > > request
> > > > >> > > > > > > > > will be
> > > > >> > > > > > > > > > denied and if it is just denied,
> > then
> > > you
> > > > might fall
> > > > >> > > back to
> > > > >> > > > > > next
> > > > >> > > > > > > > > > authorizer. If we don't have
> > chaining of
> > > > >> Authorizing in
> > > > >> > > mind,
> > > > >> > > > > > > then
> > > > >> > > > > > > > we
> > > > >> > > > > > > > > > should reconsider OPTIONAL for
> > now. Or
> > > > clarify under
> > > > >> > > which
> > > > >> > > > > > > scenario
> > > > >> > > > > > > > > > OPTIONAL will be called.
> > > > >> > > > > > > > > > #3.3 Regarding, FILTER v/s
> > > LIST_AUTHORIZED,
> > > > isn't
> > > > >> > > > > > > LIST_AUTHORIZED a
> > > > >> > > > > > > > > > special case for "FILTER"?
> > > > >> > > > > > > > > > #3.4 KafkaRequestContext.
> > requestType()
> > > v/s
> > > > Action.
> > > > >> > > > > > > > > authorizationMode. I
> > > > >> > > > > > > > > > am not sure about the overlap or
> > > ambiguity.
> > > > >> > > > > > > > > > #4 - Cool. This is good, it will
> > be less
> > > > stress on
> > > > >> the
> > > > >> > > > > > > Authorizer.
> > > > >> > > > > > > > > Ranger
> > > > >> > > > > > > > > > already supports the "count"
> > concept
> > > and also
> > > > has
> > > > >> > > batching
> > > > >> > > > > > > > > capability to
> > > > >> > > > > > > > > > aggregate similar requests to
> > reduce the
> > > > number of
> > > > >> audit
> > > > >> > > logs
> > > > >> > > > > > to
> > > > >> > > > > > > > > write. We
> > > > >> > > > > > > > > > should be able to pass this
> > through.
> > > > >> > > > > > > > > > #5 - Assuming if the object
> > instance is
> > > going
> > > > out of
> > > > >> > > scope, we
> > > > >> > > > > > > > > should be
> > > > >> > > > > > > > > > fine. Not a super important ask.
> > Ranger
> > > is
> > > > already
> > > > >> > > catching
> > > > >> > > > > > KILL
> > > > >> > > > > > > > > signal for
> > > > >> > > > > > > > > > clean up.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Thanks again. These are good
> > > enhancements.
> > > > >> Appreciate
> > > > >> > > your
> > > > >> > > > > > > efforts
> > > > >> > > > > > > > > here.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Bosco
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On 8/8/19, 2:03 AM, "Rajini
> > Sivaram" <
> > > > >> > > rajinisivaram@gmail.com
> > > > >> > > > > > >
> > > > >> > > > > > > > > wrote:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Hi Don,
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Thanks for reviewing the
> KIP.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > 1. I had this originally as
> a
> > single
> > > > Action, but
> > > > >> > > thought it
> > > > >> > > > > > > may
> > > > >> > > > > > > > > be
> > > > >> > > > > > > > > > useful
> > > > >> > > > > > > > > > to support batched authorize
> > calls
> > > as
> > > > well and
> > > > >> keep
> > > > >> > > it
> > > > >> > > > > > > > > consistent with
> > > > >> > > > > > > > > > other methods. Single
> > requests can
> > > contain
> > > > >> multiple
> > > > >> > > topics.
> > > > >> > > > > > > For
> > > > >> > > > > > > > > > example a
> > > > >> > > > > > > > > > produce request can contain
> > records
> > > for
> > > > several
> > > > >> > > partitions
> > > > >> > > > > > of
> > > > >> > > > > > > > > different
> > > > >> > > > > > > > > > topics. Broker could
> > potentially
> > > > authorize these
> > > > >> > > together.
> > > > >> > > > > > > For
> > > > >> > > > > > > > > > SimpleAclAuthorizer, batched
> > > authorize
> > > > methods
> > > > >> don't
> > > > >> > > > > > provide
> > > > >> > > > > > > > any
> > > > >> > > > > > > > > > optimisation since lookup is
> > based
> > > on
> > > > resources
> > > > >> > > followed by
> > > > >> > > > > > > the
> > > > >> > > > > > > > > > matching
> > > > >> > > > > > > > > > logic. But some authorizers
> > may
> > > manage
> > > > ACLs by
> > > > >> user
> > > > >> > > > > > principal
> > > > >> > > > > > > > > rather
> > > > >> > > > > > > > > > than
> > > > >> > > > > > > > > > resource and may be able to
> > optimize
> > > > batched
> > > > >> > > requests. I am
> > > > >> > > > > > > ok
> > > > >> > > > > > > > > with
> > > > >> > > > > > > > > > using
> > > > >> > > > > > > > > > single Action if this is
> > likely to
> > > cause
> > > > issues.
> > > > >> > > > > > > > > > 2. If you have two
> listeners,
> > one
> > > for
> > > > >> inter-broker
> > > > >> > > traffic
> > > > >> > > > > > > and
> > > > >> > > > > > > > > another
> > > > >> > > > > > > > > > for
> > > > >> > > > > > > > > > external clients, start
> > method is
> > > invoked
> > > > twice,
> > > > >> > > once for
> > > > >> > > > > > > each
> > > > >> > > > > > > > > > listener. On
> > > > >> > > > > > > > > > second thought, that may be
> > > confusing and
> > > > a
> > > > >> single
> > > > >> > > start()
> > > > >> > > > > > > > > invocation
> > > > >> > > > > > > > > > that
> > > > >> > > > > > > > > > provides all listener
> > information
> > > and
> > > > returns
> > > > >> > > multiple
> > > > >> > > > > > > futures
> > > > >> > > > > > > > > would be
> > > > >> > > > > > > > > > better. Will update the KIP.
> > > > >> > > > > > > > > > 3. A typical example is a
> > consumer
> > > > subscribing
> > > > >> to a
> > > > >> > > regex
> > > > >> > > > > > > > > pattern. We
> > > > >> > > > > > > > > > request all topic metadata
> > from the
> > > > broker in
> > > > >> order
> > > > >> > > to
> > > > >> > > > > > decide
> > > > >> > > > > > > > > whether
> > > > >> > > > > > > > > > the
> > > > >> > > > > > > > > > pattern matches, expecting
> to
> > > receive a
> > > > list of
> > > > >> > > authorised
> > > > >> > > > > > > > > topics. The
> > > > >> > > > > > > > > > user
> > > > >> > > > > > > > > > is not asking to subscribe
> to
> > an
> > > > unauthorized
> > > > >> topic.
> > > > >> > > If
> > > > >> > > > > > there
> > > > >> > > > > > > > > are 10000
> > > > >> > > > > > > > > > topics in the cluster and
> the
> > user
> > > has
> > > > access
> > > > >> to 100
> > > > >> > > of
> > > > >> > > > > > them,
> > > > >> > > > > > > > at
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > moment
> > > > >> > > > > > > > > > we log 9900 DENIED log
> > entries at
> > > INFO
> > > > level in
> > > > >> > > > > > > > > SimpleAclAuthorizer.
> > > > >> > > > > > > > > > The
> > > > >> > > > > > > > > > proposal is to authorize
> this
> > > request with
> > > > >> > > > > > > > > AuthorizationMode.FILTER, so
> > > > >> > > > > > > > > > that authorizers can log
> > resources
> > > that
> > > > are
> > > > >> filtered
> > > > >> > > out at
> > > > >> > > > > > > > > lower level
> > > > >> > > > > > > > > > like DEBUG since this is not
> > an
> > > attempt to
> > > > >> access
> > > > >> > > > > > > unauthorized
> > > > >> > > > > > > > > > resources.
> > > > >> > > > > > > > > > Brokers already handle these
> > > differently
> > > > since
> > > > >> no
> > > > >> > > > > > > authorization
> > > > >> > > > > > > > > error
> > > > >> > > > > > > > > > is
> > > > >> > > > > > > > > > returned to the client in
> > these
> > > cases.
> > > > Providing
> > > > >> > > > > > > authorization
> > > > >> > > > > > > > > mode to
> > > > >> > > > > > > > > > authorizers enables
> authorizer
> > > > implementations
> > > > >> to
> > > > >> > > generate
> > > > >> > > > > > > > > better audit
> > > > >> > > > > > > > > > logs.
> > > > >> > > > > > > > > > 4. Each request may contain
> > multiple
> > > > instances
> > > > >> of
> > > > >> > > the same
> > > > >> > > > > > > > > authorizable
> > > > >> > > > > > > > > > resource. For example a
> > produce
> > > request
> > > > may
> > > > >> contain
> > > > >> > > records
> > > > >> > > > > > > for
> > > > >> > > > > > > > > 10
> > > > >> > > > > > > > > > partitions of the same
> topic.
> > At the
> > > > moment, we
> > > > >> > > invoke
> > > > >> > > > > > > > authorize
> > > > >> > > > > > > > > > method 10
> > > > >> > > > > > > > > > times. The proposal is to
> > invoke it
> > > once
> > > > with
> > > > >> > > count=10. The
> > > > >> > > > > > > > > count is
> > > > >> > > > > > > > > > provided to authorizer just
> > for
> > > audit
> > > > logging
> > > > >> > > purposes.
> > > > >> > > > > > > > > > 5. Authorizer implements
> > Closeable,
> > > so you
> > > > >> could use
> > > > >> > > > > > close()
> > > > >> > > > > > > to
> > > > >> > > > > > > > > flush
> > > > >> > > > > > > > > > audits?
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > On Thu, Aug 8, 2019 at 7:01
> > AM Don
> > > Bosco
> > > > Durai <
> > > > >> > > > > > > > bosco@apache.org
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > wrote:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > Rajini
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Thanks for putting this
> > together.
> > > It is
> > > > >> looking
> > > > >> > > good. I
> > > > >> > > > > > > have
> > > > >> > > > > > > > > few
> > > > >> > > > > > > > > > > questions...
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > 1.
> List<AuthorizationResult>
> > > > authorize(...,
> > > > >> > > List<Action>
> > > > >> > > > > > > > > actions).
> > > > >> > > > > > > > > > Do you
> > > > >> > > > > > > > > > > see a scenario where the
> > broker
> > > will
> > > > call
> > > > >> > > authorize for
> > > > >> > > > > > > > > multiple
> > > > >> > > > > > > > > > topics at
> > > > >> > > > > > > > > > > the same time? I can
> > understand
> > > that
> > > > during
> > > > >> > > > > > > creating/deleting
> > > > >> > > > > > > > > ACLS,
> > > > >> > > > > > > > > > > multiple permissions for
> > multiple
> > > > resources
> > > > >> might
> > > > >> > > be
> > > > >> > > > > > done.
> > > > >> > > > > > > > For
> > > > >> > > > > > > > > > authorize
> > > > >> > > > > > > > > > > call, would this be a
> case?
> > And
> > > does the
> > > > >> Authorize
> > > > >> > > > > > > > > implementation
> > > > >> > > > > > > > > > will be
> > > > >> > > > > > > > > > > able to do performance
> > > optimization
> > > > because of
> > > > >> > > this? Or
> > > > >> > > > > > > > should
> > > > >> > > > > > > > > we
> > > > >> > > > > > > > > > just keep
> > > > >> > > > > > > > > > > it simple? I don't see it
> > as an
> > > issue
> > > > from
> > > > >> Apache
> > > > >> > > Ranger
> > > > >> > > > > > > > side,
> > > > >> > > > > > > > > but
> > > > >> > > > > > > > > > just
> > > > >> > > > > > > > > > > checking to see whether we
> > need
> > > to be
> > > > aware of
> > > > >> > > something.
> > > > >> > > > > > > > > > > 2. Should I assume that
> the
> > > > SecurityProtocol
> > > > >> passed
> > > > >> > > > > > during
> > > > >> > > > > > > > > start and
> > > > >> > > > > > > > > > the
> > > > >> > > > > > > > > > > one return by
> > > > >> > > KafkaRequestContext.securityProtocol() will
> > > > >> > > > > > > be
> > > > >> > > > > > > > > the
> > > > >> > > > > > > > > > same?
> > > > >> > > > > > > > > > > CompletableFuture<Void>
> > > start(String
> > > > >> listenerName,
> > > > >> > > > > > > > > SecurityProtocol
> > > > >> > > > > > > > > > > securityProtocol);
> > > > >> > > > > > > > > > >
> > > KafkaRequestContext.securityProtocol()
> > > > >> > > > > > > > > > > 3. What is the purpose of
> > > > AuthorizationMode?
> > > > >> How
> > > > >> > > does the
> > > > >> > > > > > > > > broker
> > > > >> > > > > > > > > > decide
> > > > >> > > > > > > > > > > what mode to use when the
> > > authorize()
> > > > method
> > > > >> is
> > > > >> > > called?
> > > > >> > > > > > > > > > > 4. Can we clarify "count"
> in
> > > Action a
> > > > bit
> > > > >> more?
> > > > >> > > How is it
> > > > >> > > > > > > > used?
> > > > >> > > > > > > > > > > 5. Do you feel having
> "stop"
> > > along with
> > > > >> "start" be
> > > > >> > > > > > helpful?
> > > > >> > > > > > > > > E.g. In
> > > > >> > > > > > > > > > Ranger
> > > > >> > > > > > > > > > > we try to optimize the
> Audit
> > > writing by
> > > > >> caching
> > > > >> > > the logs
> > > > >> > > > > > > for
> > > > >> > > > > > > > a
> > > > >> > > > > > > > > fixed
> > > > >> > > > > > > > > > > interval. But when the
> > Broker
> > > > terminates, we
> > > > >> do a
> > > > >> > > forced
> > > > >> > > > > > > > flush.
> > > > >> > > > > > > > > > Having an
> > > > >> > > > > > > > > > > explicit "stop" might give
> > us a
> > > formal
> > > > way to
> > > > >> > > flush our
> > > > >> > > > > > > > audits.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Thanks
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Bosco
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > On 8/7/19, 3:59 PM,
> "Rajini
> > > Sivaram" <
> > > > >> > > > > > > > rajinisivaram@gmail.com
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > wrote:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Hi Ron/Harsha/Satish,
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Thanks for reviewing
> > the KIP!
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > We should perhaps have
> > a wider
> > > > discussion
> > > > >> > > outside
> > > > >> > > > > > this
> > > > >> > > > > > > > KIP
> > > > >> > > > > > > > > for
> > > > >> > > > > > > > > > > refactoring
> > > > >> > > > > > > > > > > clients so that others
> > who
> > > are not
> > > > >> following
> > > > >> > > this KIP
> > > > >> > > > > > > > also
> > > > >> > > > > > > > > > notice the
> > > > >> > > > > > > > > > > discussion. Satish,
> > would you
> > > like
> > > > to
> > > > >> start a
> > > > >> > > > > > > discussion
> > > > >> > > > > > > > > thread
> > > > >> > > > > > > > > > on dev?
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Regards,
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Rajini
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > On Wed, Aug 7, 2019 at
> > 6:21 PM
> > > > Satish
> > > > >> Duggana <
> > > > >> > > > > > > > > > > satish.duggana@gmail.com>
> > > > >> > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > > I felt the same need
> > when
> > > we want
> > > > to
> > > > >> add a
> > > > >> > > > > > pluggable
> > > > >> > > > > > > > API
> > > > >> > > > > > > > > for
> > > > >> > > > > > > > > > core
> > > > >> > > > > > > > > > > > server
> functionality.
> > This
> > > does
> > > > not
> > > > >> need to
> > > > >> > > be part
> > > > >> > > > > > > of
> > > > >> > > > > > > > > this
> > > > >> > > > > > > > > > KIP, it
> > > > >> > > > > > > > > > > > can be a separate
> > KIP. I can
> > > > contribute
> > > > >> those
> > > > >> > > > > > > > refactoring
> > > > >> > > > > > > > > > changes if
> > > > >> > > > > > > > > > > > others are OK with
> > that.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > It is better to
> have a
> > > structure
> > > > like
> > > > >> below.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > kafka-common:
> > > > >> > > > > > > > > > > > common classes which
> > can be
> > > used
> > > > in any
> > > > >> of
> > > > >> > > the
> > > > >> > > > > > other
> > > > >> > > > > > > > > modules
> > > > >> > > > > > > > > > in Kafka
> > > > >> > > > > > > > > > > > like client,
> > > Kafka-server-common
> > > > and
> > > > >> server
> > > > >> > > etc.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > kafka-client-common:
> > > > >> > > > > > > > > > > > common classes which
> > can be
> > > used
> > > > in the
> > > > >> > > client
> > > > >> > > > > > > module.
> > > > >> > > > > > > > > This
> > > > >> > > > > > > > > > can be
> > > > >> > > > > > > > > > > > part of client
> module
> > > itself.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > kafka-server-common:
> > > > >> > > > > > > > > > > > classes required
> only
> > for
> > > > kafka-server.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Thanks.
> > > > >> > > > > > > > > > > > Satish.
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > On Wed, Aug 7, 2019
> > at 9:28
> > > PM
> > > > Harsha
> > > > >> > > Chintalapani
> > > > >> > > > > > <
> > > > >> > > > > > > > > > kafka@harsha.io>
> > > > >> > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Thanks for the KIP
> > Rajini.
> > > > >> > > > > > > > > > > > > Quick thought, it
> > would
> > > be good
> > > > to
> > > > >> have a
> > > > >> > > common
> > > > >> > > > > > > > module
> > > > >> > > > > > > > > > outside of
> > > > >> > > > > > > > > > > > clients
> > > > >> > > > > > > > > > > > > that only applies
> to
> > > server side
> > > > >> > > interfaces &
> > > > >> > > > > > > > changes.
> > > > >> > > > > > > > > It
> > > > >> > > > > > > > > > looks
> > > > >> > > > > > > > > > > like we
> > > > >> > > > > > > > > > > > are
> > > > >> > > > > > > > > > > > > increasingly in
> > favor of
> > > using
> > > > Java
> > > > >> > > interface for
> > > > >> > > > > > > > > pluggable
> > > > >> > > > > > > > > > > modules on
> > > > >> > > > > > > > > > > > the
> > > > >> > > > > > > > > > > > > broker side.
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > Thanks,
> > > > >> > > > > > > > > > > > > Harsha
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > On Tue, Aug 06,
> > 2019 at
> > > 2:31 PM,
> > > > >> Rajini
> > > > >> > > Sivaram <
> > > > >> > > > > > > > > > > rajinisivaram@gmail.com
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Hi all,
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > I have created a
> > KIP to
> > > > replace the
> > > > >> Scala
> > > > >> > > > > > > > Authorizer
> > > > >> > > > > > > > > API
> > > > >> > > > > > > > > > with a
> > > > >> > > > > > > > > > > new
> > > > >> > > > > > > > > > > > Java
> > > > >> > > > > > > > > > > > > > API:
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > -
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > >> > > > > > > > > > > > > >
> > > > >> > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > This is
> > replacement for
> > > KIP-50
> > > > >> which was
> > > > >> > > > > > accepted
> > > > >> > > > > > > > > but never
> > > > >> > > > > > > > > > > merged.
> > > > >> > > > > > > > > > > > Apart
> > > > >> > > > > > > > > > > > > > from moving to a
> > Java
> > > API
> > > > consistent
> > > > >> > > with other
> > > > >> > > > > > > > > pluggable
> > > > >> > > > > > > > > > > interfaces
> > > > >> > > > > > > > > > > > in the
> > > > >> > > > > > > > > > > > > > broker, KIP-504
> > also
> > > attempts
> > > > to
> > > > >> address
> > > > >> > > known
> > > > >> > > > > > > > > limitations
> > > > >> > > > > > > > > > in the
> > > > >> > > > > > > > > > > > > > authorizer. If
> > you have
> > > come
> > > > across
> > > > >> other
> > > > >> > > > > > > > > limitations that
> > > > >> > > > > > > > > > you
> > > > >> > > > > > > > > > > would
> > > > >> > > > > > > > > > > > like
> > > > >> > > > > > > > > > > > > > to see addressed
> > in the
> > > new
> > > > API,
> > > > >> please
> > > > >> > > raise
> > > > >> > > > > > > these
> > > > >> > > > > > > > > on the
> > > > >> > > > > > > > > > > discussion
> > > > >> > > > > > > > > > > > > > thread so that
> we
> > can
> > > > consider those
> > > > >> > > too. All
> > > > >> > > > > > > > > suggestions
> > > > >> > > > > > > > > > and
> > > > >> > > > > > > > > > > feedback
> > > > >> > > > > > > > > > > > are
> > > > >> > > > > > > > > > > > > > welcome.
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Thank you,
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > > > > Rajini
> > > > >> > > > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Don,
Thanks for your note.
1) The intention is to avoid blocking in the calling thread. We already
have several requests that are put into a purgatory when waiting for remote
communication, for example produce request waiting for replication. Since
we have a limited number of request threads in the broker, we want to make
progress with other requests, while one is waiting on any form of remote
communication.
2) Async management calls will be useful with the default authorizer when
KIP-500 removes ZK and we rely on Kafka instead. Our current ZK-based
implementation as well as any custom implementations that don't want to be
async will just need to return a sync'ed value. So instead of returning `
value`, the code would just return `CompletableFuture.completedFuture(value)
`. So it would be just a single line change in the implementation with the
new API. The caller would treat completedFuture exactly as it does today,
processing the request synchronously without using a purgatory.
3) For implementations that return a completedFuture as described in 2),
the behaviour would remain exactly the same. No additional threads or
purgatory will be used for this case. So there would be no performance
penalty. For implementations that return a future that is not complete, we
prioritise running more requests concurrently. So in a deployment with a
large number of clients, we would improve performance by allowing other
requests to be processed on the request threads while some are waiting for
authorization metadata.
4) I was concerned about this too. The goal is to make the API flexible
enough to handle large scale deployments in future when caching all
authorization metadata in each broker is not viable. Using an async API
that returns CompletionStage, the caller has the option to handle the
result synchronously or asynchronously, so we don't necessarily need to
update the calling code right away. Custom authorizers using the async API
have full control over whether authorization is performed in-line since
completedFuture will always be handled synchronously. We do need to update
KafkaApis to take advantage of the asynchronous API to improve scale. Even
though this is a big change, since we will be doing the same for all
requests, it shouldn't be too hard to maintain since the same pattern will
be used for all requests.
Regards,
Rajini
On Tue, Sep 3, 2019 at 11:48 PM Don Bosco Durai <bo...@apache.org> wrote:
> Hi Rajini
>
> Help me understand this a bit more.
>
> 1. For all practical purpose, without authorization you can't go to the
> next step. The calling code needs to block anyway. So should we just let
> the implementation code do the async part?
> 2. If you feel management calls need to be async, then we should consider
> another set of async APIs. I don't feel we should complicate it further (
> 3. Another concern I have is wrt performance. Kafka has been built to
> handle 1000s per second requests. Not sure whether making it async will add
> any unnecessary overhead.
> 4. How much complication would this add on the calling side? And is it
> worth it?
>
> Thanks
>
> Bosco
>
>
> On 9/3/19, 8:50 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
>
> Hi all,
>
> Ismael brought up a point that it will be good to make the Authorizer
> interface asynchronous to avoid blocking request threads during remote
> operations.
>
> 1) Since we want to support different backends for authorization
> metadata,
> making createAcls() and deleteAcls() asynchronous makes sense since
> these
> always involve remote operations. When KIP-500 removes ZooKeeper, we
> would
> want to move ACLs to Kafka and async updates will avoid unnecessary
> blocking.
> 2) For authorize() method, we currently use cached ACLs in the built-in
> authorizers, so synchronous authorise operations work well now. But
> async
> authorize() would support this scenario as well as authorizers in large
> organisations where an LRU cache would enable a smaller cache even
> when the
> backend holds a large amount of ACLs for infrequently used resources or
> users who don't use the system frequently.
>
> For both cases, the built-in authorizer will continue to be
> synchronous,
> using CompletableFuture.completedFuture() to return the actual result.
> But
> async API will make custom authorizer implementations more flexible. I
> would like to know if there are any concerns with these changes before
> updating the KIP.
>
> *Proposed API:*
> public interface Authorizer extends Configurable, Closeable {
>
> Map<Endpoint, CompletionStage<Void>> start(AuthorizerServerInfo
> serverInfo);
> List<CompletionStage<AuthorizationResult>>
> authorize(AuthorizableRequestContext requestContext, List<Action>
> actions);
> List<CompletionStage<AclCreateResult>>
> createAcls(AuthorizableRequestContext requestContext, List<AclBinding>
> aclBindings);
> List<CompletionStage<AclDeleteResult>>
> deleteAcls(AuthorizableRequestContext requestContext,
> List<AclBindingFilter> aclBindingFilters);
> CompletionStage<Collection<AclBinding>> acls(AclBindingFilter
> filter);
> }
>
>
> Thank you,
>
> Rajini
>
> On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <bo...@apache.org>
> wrote:
>
> > Hi Rajini
> >
> > Thanks for clarifying. I am good for now.
> >
> > Regards
> >
> > Bosco
> >
> >
> > On 8/16/19, 11:30 AM, "Rajini Sivaram" <ra...@gmail.com>
> wrote:
> >
> > Hi Don,
> >
> > That should be fine. I guess Ranger loads policies from the
> database
> > synchronously when the authorizer is configured, similar to
> > SimpleAclAuthorizer loading from ZooKeeper. Ranger can continue
> to load
> > synchronously from `configure()` or `start()` and return an
> empty map
> > from
> > `start()`. That would retain the existing behaviour.. When the
> same
> > database stores policies for all listeners and the policies are
> not
> > stored
> > in Kafka, there is no value in making the load asynchronous.
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai <
> bosco@apache.org>
> > wrote:
> >
> > > Hi Rajini
> > >
> > > Assuming this doesn't affect custom plugins, I don't have any
> > concerns
> > > regarding this.
> > >
> > > I do have one question regarding:
> > >
> > > "For authorizers that don’t store metadata in ZooKeeper,
> ensure that
> > > authorizer metadata for each listener is available before
> starting
> > up the
> > > listener. This enables different authorization metadata stores
> for
> > > different listeners."
> > >
> > > Since Ranger uses its own database for storing policies, do you
> > anticipate
> > > any issues?
> > >
> > > Thanks
> > >
> > > Bosco
> > >
> > >
> > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <
> rajinisivaram@gmail.com>
> > wrote:
> > >
> > > Hi all,
> > >
> > > I made another change to the KIP. The KIP was originally
> > proposing to
> > > extend SimpleAclAuthorizer to also implement the new API
> (in
> > addition
> > > to
> > > the existing API). But since we use the new API when
> available,
> > this
> > > can
> > > break custom authorizers that extend this class and
> override
> > specific
> > > methods of the old API. To avoid breaking any existing
> custom
> > > implementations that extend this class, particularly
> because it
> > is in
> > > the
> > > public package kafka.security.auth, the KIP now proposes to
> > retain the
> > > old
> > > authorizer as-is. SimpleAclAuthorizer will continue to
> > implement the
> > > old
> > > API, but will be deprecated. A new authorizer
> implementation
> > > kafka.security.authorizer.AclAuthorizer will be added for
> the
> > new API
> > > (this
> > > will not be in the public package).
> > >
> > > Please let me know if you have any concerns.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > > On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
> > > rajinisivaram@gmail.com>
> > > wrote:
> > >
> > > > Thanks Colin.
> > > >
> > > > If there are no other concerns, I will start vote later
> today.
> > Many
> > > thanks
> > > > to every one for the feedback.
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <
> > cmccabe@apache.org>
> > > wrote:
> > > >
> > > >> Thanks, Rajini. It looks good to me.
> > > >>
> > > >> best,
> > > >> Colin
> > > >>
> > > >>
> > > >> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram wrote:
> > > >> > Hi Colin,
> > > >> >
> > > >> > Thanks for the review. I have updated the KIP to move
> the
> > > interfaces for
> > > >> > request context and server info to the authorizer
> package.
> > These
> > > are now
> > > >> > called AuthorizableRequestContext and
> AuthorizerServerInfo.
> > > Endpoint is
> > > >> now
> > > >> > a class in org.apache.kafka.common to make it reusable
> > since we
> > > already
> > > >> > have multiple implementations of it. I have removed
> > requestName
> > > from the
> > > >> > request context interface since authorizers can
> distinguish
> > > follower
> > > >> fetch
> > > >> > and consumer fetch from the operation being
> authorized. So
> > 16-bit
> > > >> request
> > > >> > type should be sufficient for audit logging. Also
> replaced
> > > AuditFlag
> > > >> with
> > > >> > two booleans as you suggested.
> > > >> >
> > > >> > Can you take another look and see if the KIP is ready
> for
> > voting?
> > > >> >
> > > >> > Thanks for all your help!
> > > >> >
> > > >> > Regards,
> > > >> >
> > > >> > Rajini
> > > >> >
> > > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <
> > cmccabe@apache.org>
> > > >> wrote:
> > > >> >
> > > >> > > Hi Rajini,
> > > >> > >
> > > >> > > I think it would be good to rename
> KafkaRequestContext to
> > > something
> > > >> like
> > > >> > > AuthorizableRequestContext, and put it in the
> > > >> > > org.apache.kafka.server.authorizer namespace. If
> we put
> > it in
> > > the
> > > >> > > org.apache.kafka.common namespace, then it's not
> really
> > clear
> > > that
> > > >> it's
> > > >> > > part of the Authorizer API. Since this class is
> purely an
> > > interface,
> > > >> with
> > > >> > > no concrete implementation of anything, there's
> nothing
> > common
> > > to
> > > >> really
> > > >> > > reuse in any case. We definitely don't want
> someone to
> > > accidentally
> > > >> add or
> > > >> > > remove methods because they think this is just
> another
> > internal
> > > class
> > > >> used
> > > >> > > for requests.
> > > >> > >
> > > >> > > The BrokerInfo class is a nice improvement. It
> looks
> > like it
> > > will be
> > > >> > > useful for passing in information about the context
> we're
> > > running
> > > >> in. It
> > > >> > > would be nice to call this class ServerInfo rather
> than
> > > BrokerInfo,
> > > >> since
> > > >> > > we will want to run the authorizer on controllers
> as well
> > as on
> > > >> brokers,
> > > >> > > and the controller may run as a separate process
> post
> > KIP-500.
> > > I also
> > > >> > > think that this class should be in the
> > > >> org.apache.kafka.server.authorizer
> > > >> > > namespace. Again, it is an interface, not a
> concrete
> > > implementation,
> > > >> and
> > > >> > > it's an interface that is very specifically for the
> > authorizer.
> > > >> > >
> > > >> > > I agree that we probably don't have enough
> information
> > > preserved for
> > > >> > > requests currently to always know what entity made
> them.
> > So we
> > > can
> > > >> leave
> > > >> > > that out for now (except in the special case of
> Fetch).
> > > Perhaps we
> > > >> can add
> > > >> > > this later if it's needed.
> > > >> > >
> > > >> > > I understand the intention behind AuthorizationMode
> > (which is
> > > now
> > > >> called
> > > >> > > AuditFlag in the latest revision). But it still
> feels
> > > complex. What
> > > >> if we
> > > >> > > just had two booleans in Action: logSuccesses and
> > logFailures?
> > > That
> > > >> seems
> > > >> > > to cover all the cases here. MANDATORY_AUTHORIZE =
> true,
> > true.
> > > >> > > OPTIONAL_AUTHORIZE = true, false. FILTER = true,
> false.
> > > >> LIST_AUTHORIZED =
> > > >> > > false, false. Would there be anything lost versus
> having
> > the
> > > enum?
> > > >> > >
> > > >> > > best,
> > > >> > > Colin
> > > >> > >
> > > >> > >
> > > >> > > On Wed, Aug 14, 2019, at 06:29, Mickael Maison
> wrote:
> > > >> > > > Hi Rajini,
> > > >> > > >
> > > >> > > > Thanks for the KIP!
> > > >> > > > I really like that authorize() will be able to
> take a
> > batch of
> > > >> > > > requests, this will speed up many implementations!
> > > >> > > >
> > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <
> > > >> rajinisivaram@gmail.com>
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > Thanks David! I have fixed the typo.
> > > >> > > > >
> > > >> > > > > Also made a couple of changes to make the
> context
> > > interfaces more
> > > >> > > generic.
> > > >> > > > > KafkaRequestContext now returns the 16-bit API
> key as
> > Colin
> > > >> suggested
> > > >> > > as
> > > >> > > > > well as the friendly name used in metrics which
> are
> > useful
> > > in
> > > >> audit
> > > >> > > logs.
> > > >> > > > > `Authorizer#start` is now provided a generic
> > `BrokerInfo`
> > > >> interface
> > > >> > > that
> > > >> > > > > gives cluster id, broker id and endpoint
> information.
> > The
> > > generic
> > > >> > > interface
> > > >> > > > > can potentially be used in other broker plugins
> in
> > future
> > > and
> > > >> provides
> > > >> > > > > dynamically generated configs like broker id
> and ports
> > > which are
> > > >> > > currently
> > > >> > > > > not available to plugins unless these configs
> are
> > statically
> > > >> > > configured.
> > > >> > > > > Please let me know if there are any concerns.
> > > >> > > > >
> > > >> > > > > Regards,
> > > >> > > > >
> > > >> > > > > Rajini
> > > >> > > > >
> > > >> > > > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot <
> > > djacot@confluent.io>
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > > Hi Rajini,
> > > >> > > > > >
> > > >> > > > > > Thank you for the update! It looks good to me.
> > There is a
> > > typo
> > > >> in the
> > > >> > > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` ->
> > > >> `MANDATORY_AUTHORIZE`.
> > > >> > > > > >
> > > >> > > > > > Regards,
> > > >> > > > > > David
> > > >> > > > > >
> > > >> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini
> Sivaram <
> > > >> > > rajinisivaram@gmail.com>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi David,
> > > >> > > > > > >
> > > >> > > > > > > Thanks for reviewing the KIP! Since
> questions
> > about
> > > >> `authorization
> > > >> > > mode`
> > > >> > > > > > > and `count` have come up multiple times, I
> have
> > renamed
> > > both.
> > > >> > > > > > >
> > > >> > > > > > > 1) Renamed `count` to
> `resourceReferenceCount`.
> > It is
> > > the
> > > >> number
> > > >> > > of times
> > > >> > > > > > > the resource being authorized is referenced
> > within the
> > > >> request.
> > > >> > > > > > >
> > > >> > > > > > > 2) Renamed `AuthorizationMode` to
> `AuditFlag`. It
> > is
> > > provided
> > > >> to
> > > >> > > improve
> > > >> > > > > > > audit logging in the authorizer. The enum
> values
> > have
> > > javadoc
> > > >> which
> > > >> > > > > > > indicate how the authorization result is
> used in
> > each
> > > of the
> > > >> modes
> > > >> > > to
> > > >> > > > > > > enable authorizers to log audit messages at
> the
> > right
> > > severity
> > > >> > > level.
> > > >> > > > > > >
> > > >> > > > > > > Regards,
> > > >> > > > > > >
> > > >> > > > > > > Rajini
> > > >> > > > > > >
> > > >> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM David Jacot
> <
> > > >> djacot@confluent.io>
> > > >> > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Hi Rajini,
> > > >> > > > > > > >
> > > >> > > > > > > > Thank you for the KIP. Overall, it looks
> good
> > to me.
> > > I have
> > > >> few
> > > >> > > > > > > > questions/suggestions:
> > > >> > > > > > > >
> > > >> > > > > > > > 1. It is hard to grasp what
> `Action#count` is
> > for. I
> > > guess I
> > > >> > > understand
> > > >> > > > > > > > where you want to go with it but it took
> me a
> > while to
> > > >> figure it
> > > >> > > out.
> > > >> > > > > > > > Perhaps, we could come up with a better
> name
> > than
> > > `count`?
> > > >> > > > > > > >
> > > >> > > > > > > > 2. I had a hard time trying to understand
> the
> > > >> `AuthorizationMode`
> > > >> > > > > > > concept,
> > > >> > > > > > > > especially wrt. the OPTIONAL one. My
> > understanding is
> > > that
> > > >> an
> > > >> > > ACL is
> > > >> > > > > > > either
> > > >> > > > > > > > defined or not. Could you elaborate a bit
> more
> > on
> > > that?
> > > >> > > > > > > >
> > > >> > > > > > > > Thanks,
> > > >> > > > > > > > David
> > > >> > > > > > > >
> > > >> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco
> Durai
> > <
> > > >> > > bosco@apache.org>
> > > >> > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Hi Rajini
> > > >> > > > > > > > >
> > > >> > > > > > > > > 3.2 - This makes sense. Thanks for
> clarifying.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Rest looks fine. Once the
> implementations are
> > done,
> > > it
> > > >> will be
> > > >> > > more
> > > >> > > > > > > clear
> > > >> > > > > > > > > on the different values RequestType and
> Mode.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks
> > > >> > > > > > > > >
> > > >> > > > > > > > > Bosco
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <
> > > >> rajinisivaram@gmail.com
> > > >> > > >
> > > >> > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > Hi Don,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks for the suggestions. A few
> > responses
> > > below:
> > > >> > > > > > > > >
> > > >> > > > > > > > > 3.1 Can rename and improve docs if
> we keep
> > > this. Let's
> > > >> > > finish the
> > > >> > > > > > > > > discussion on Colin's suggestions
> > regarding this
> > > >> first.
> > > >> > > > > > > > > 3.2 No, I was thinking of some
> requests
> > that
> > > have an
> > > >> old
> > > >> > > way of
> > > >> > > > > > > > > authorizing
> > > >> > > > > > > > > and a new way where we have
> retained the
> > old
> > > way for
> > > >> > > backward
> > > >> > > > > > > > > compatibility. One example is
> > Cluster:Create
> > > >> permission to
> > > >> > > create
> > > >> > > > > > > > > topics.
> > > >> > > > > > > > > We have replaced this with
> fine-grained
> > topic
> > > create
> > > >> > > access using
> > > >> > > > > > > > > Topic:Create
> > > >> > > > > > > > > for topic patterns. But we still
> check if
> > user
> > > has
> > > >> > > Cluster:Create
> > > >> > > > > > > > > first. If
> > > >> > > > > > > > > Denied, the deny is ignored and we
> check
> > > >> Topic:Create. We
> > > >> > > dont
> > > >> > > > > > want
> > > >> > > > > > > > to
> > > >> > > > > > > > > log
> > > >> > > > > > > > > DENY for Cluster:Create at INFO
> level for
> > this,
> > > since
> > > >> this
> > > >> > > is
> > > >> > > > > > not a
> > > >> > > > > > > > > mandatory ACL for creating topics.
> We
> > will get
> > > >> appropriate
> > > >> > > logs
> > > >> > > > > > > from
> > > >> > > > > > > > > the
> > > >> > > > > > > > > subsequent Topic:Create in this
> case.
> > > >> > > > > > > > > 3.3 They are not quite the same.
> FILTER
> > implies
> > > that
> > > >> user
> > > >> > > > > > actually
> > > >> > > > > > > > > requested access to and performed
> some
> > > operation on
> > > >> the
> > > >> > > filtered
> > > >> > > > > > > > > resources.
> > > >> > > > > > > > > LIST_AUTHORZED did not result in any
> > actual
> > > access.
> > > >> User
> > > >> > > simply
> > > >> > > > > > > > wanted
> > > >> > > > > > > > > to
> > > >> > > > > > > > > know what they are allowed to
> access.
> > > >> > > > > > > > > 3.4 Request types are Produce,
> JoinGroup,
> > > OffsetCommit
> > > >> > > etc. So
> > > >> > > > > > that
> > > >> > > > > > > > is
> > > >> > > > > > > > > different from authorization mode,
> > operation
> > > etc.
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Thu, Aug 8, 2019 at 11:36 PM Don
> Bosco
> > Durai
> > > <
> > > >> > > > > > bosco@apache.org>
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > Hi Rajini
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks for clarifying. This is
> very
> > helpful...
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > #1 - On the Ranger side, we
> should be
> > able to
> > > handle
> > > >> > > multiple
> > > >> > > > > > > > > requests at
> > > >> > > > > > > > > > the same time. I was just not
> sure how
> > much
> > > >> processing
> > > >> > > overhead
> > > >> > > > > > > > will
> > > >> > > > > > > > > be
> > > >> > > > > > > > > > there on the Broker side to split
> and
> > then
> > > >> consolidate
> > > >> > > the
> > > >> > > > > > > results.
> > > >> > > > > > > > > If it
> > > >> > > > > > > > > > is negligible, then this is the
> better
> > way.
> > > It will
> > > >> > > make it
> > > >> > > > > > > future
> > > >> > > > > > > > > proof.
> > > >> > > > > > > > > > #2 - I agree, having a single
> "start"
> > call
> > > makes it
> > > >> > > cleaner.
> > > >> > > > > > The
> > > >> > > > > > > > > > Authorization implementation will
> only
> > have
> > > to do
> > > >> the
> > > >> > > > > > > > initialization
> > > >> > > > > > > > > only
> > > >> > > > > > > > > > once.
> > > >> > > > > > > > > > #3.1 - Thanks for the
> clarification. I
> > think
> > > it
> > > >> makes
> > > >> > > sense to
> > > >> > > > > > > have
> > > >> > > > > > > > > this.
> > > >> > > > > > > > > > The term "Mode" might not be
> explicit
> > enough.
> > > >> > > Essentially it
> > > >> > > > > > > seems
> > > >> > > > > > > > > you want
> > > >> > > > > > > > > > the Authorizer to know the
> > Intent/Purpose of
> > > the
> > > >> > > authorize call
> > > >> > > > > > > and
> > > >> > > > > > > > > let the
> > > >> > > > > > > > > > Authorizer decide what to log as
> Audit
> > event.
> > > >> Changing
> > > >> > > the
> > > >> > > > > > > > > class/field name
> > > >> > > > > > > > > > or giving more documentation will
> do.
> > > >> > > > > > > > > > #3.2 - Regarding the option
> "OPTIONAL".
> > Are
> > > you
> > > >> thinking
> > > >> > > from
> > > >> > > > > > > > > chaining
> > > >> > > > > > > > > > multiple Authorizer? If so, I am
> not
> > sure
> > > whether
> > > >> the
> > > >> > > Broker
> > > >> > > > > > > would
> > > >> > > > > > > > > have
> > > >> > > > > > > > > > enough information to make that
> > decision. I
> > > feel the
> > > >> > > Authorizer
> > > >> > > > > > > > will
> > > >> > > > > > > > > be the
> > > >> > > > > > > > > > one who would have that
> knowledge. E.g.
> > in
> > > Ranger
> > > >> we have
> > > >> > > > > > > explicit
> > > >> > > > > > > > > deny,
> > > >> > > > > > > > > > which means no matter what, the
> request
> > > should be
> > > >> denied
> > > >> > > for
> > > >> > > > > > the
> > > >> > > > > > > > > user/group
> > > >> > > > > > > > > > or condition. So if you are
> thinking of
> > > chaining
> > > >> > > Authorizers,
> > > >> > > > > > > then
> > > >> > > > > > > > > > responses should have the third
> state,
> > e.g.
> > > >> > > "DENIED_FINAL", in
> > > >> > > > > > > > which
> > > >> > > > > > > > > case
> > > >> > > > > > > > > > if there is an Authorization
> chain, it
> > will
> > > be stop
> > > >> and
> > > >> > > the
> > > >> > > > > > > request
> > > >> > > > > > > > > will be
> > > >> > > > > > > > > > denied and if it is just denied,
> then
> > you
> > > might fall
> > > >> > > back to
> > > >> > > > > > next
> > > >> > > > > > > > > > authorizer. If we don't have
> chaining of
> > > >> Authorizing in
> > > >> > > mind,
> > > >> > > > > > > then
> > > >> > > > > > > > we
> > > >> > > > > > > > > > should reconsider OPTIONAL for
> now. Or
> > > clarify under
> > > >> > > which
> > > >> > > > > > > scenario
> > > >> > > > > > > > > > OPTIONAL will be called.
> > > >> > > > > > > > > > #3.3 Regarding, FILTER v/s
> > LIST_AUTHORIZED,
> > > isn't
> > > >> > > > > > > LIST_AUTHORIZED a
> > > >> > > > > > > > > > special case for "FILTER"?
> > > >> > > > > > > > > > #3.4 KafkaRequestContext.
> requestType()
> > v/s
> > > Action.
> > > >> > > > > > > > > authorizationMode. I
> > > >> > > > > > > > > > am not sure about the overlap or
> > ambiguity.
> > > >> > > > > > > > > > #4 - Cool. This is good, it will
> be less
> > > stress on
> > > >> the
> > > >> > > > > > > Authorizer.
> > > >> > > > > > > > > Ranger
> > > >> > > > > > > > > > already supports the "count"
> concept
> > and also
> > > has
> > > >> > > batching
> > > >> > > > > > > > > capability to
> > > >> > > > > > > > > > aggregate similar requests to
> reduce the
> > > number of
> > > >> audit
> > > >> > > logs
> > > >> > > > > > to
> > > >> > > > > > > > > write. We
> > > >> > > > > > > > > > should be able to pass this
> through.
> > > >> > > > > > > > > > #5 - Assuming if the object
> instance is
> > going
> > > out of
> > > >> > > scope, we
> > > >> > > > > > > > > should be
> > > >> > > > > > > > > > fine. Not a super important ask.
> Ranger
> > is
> > > already
> > > >> > > catching
> > > >> > > > > > KILL
> > > >> > > > > > > > > signal for
> > > >> > > > > > > > > > clean up.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks again. These are good
> > enhancements.
> > > >> Appreciate
> > > >> > > your
> > > >> > > > > > > efforts
> > > >> > > > > > > > > here.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Bosco
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On 8/8/19, 2:03 AM, "Rajini
> Sivaram" <
> > > >> > > rajinisivaram@gmail.com
> > > >> > > > > > >
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Hi Don,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks for reviewing the KIP.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > 1. I had this originally as a
> single
> > > Action, but
> > > >> > > thought it
> > > >> > > > > > > may
> > > >> > > > > > > > > be
> > > >> > > > > > > > > > useful
> > > >> > > > > > > > > > to support batched authorize
> calls
> > as
> > > well and
> > > >> keep
> > > >> > > it
> > > >> > > > > > > > > consistent with
> > > >> > > > > > > > > > other methods. Single
> requests can
> > contain
> > > >> multiple
> > > >> > > topics.
> > > >> > > > > > > For
> > > >> > > > > > > > > > example a
> > > >> > > > > > > > > > produce request can contain
> records
> > for
> > > several
> > > >> > > partitions
> > > >> > > > > > of
> > > >> > > > > > > > > different
> > > >> > > > > > > > > > topics. Broker could
> potentially
> > > authorize these
> > > >> > > together.
> > > >> > > > > > > For
> > > >> > > > > > > > > > SimpleAclAuthorizer, batched
> > authorize
> > > methods
> > > >> don't
> > > >> > > > > > provide
> > > >> > > > > > > > any
> > > >> > > > > > > > > > optimisation since lookup is
> based
> > on
> > > resources
> > > >> > > followed by
> > > >> > > > > > > the
> > > >> > > > > > > > > > matching
> > > >> > > > > > > > > > logic. But some authorizers
> may
> > manage
> > > ACLs by
> > > >> user
> > > >> > > > > > principal
> > > >> > > > > > > > > rather
> > > >> > > > > > > > > > than
> > > >> > > > > > > > > > resource and may be able to
> optimize
> > > batched
> > > >> > > requests. I am
> > > >> > > > > > > ok
> > > >> > > > > > > > > with
> > > >> > > > > > > > > > using
> > > >> > > > > > > > > > single Action if this is
> likely to
> > cause
> > > issues.
> > > >> > > > > > > > > > 2. If you have two listeners,
> one
> > for
> > > >> inter-broker
> > > >> > > traffic
> > > >> > > > > > > and
> > > >> > > > > > > > > another
> > > >> > > > > > > > > > for
> > > >> > > > > > > > > > external clients, start
> method is
> > invoked
> > > twice,
> > > >> > > once for
> > > >> > > > > > > each
> > > >> > > > > > > > > > listener. On
> > > >> > > > > > > > > > second thought, that may be
> > confusing and
> > > a
> > > >> single
> > > >> > > start()
> > > >> > > > > > > > > invocation
> > > >> > > > > > > > > > that
> > > >> > > > > > > > > > provides all listener
> information
> > and
> > > returns
> > > >> > > multiple
> > > >> > > > > > > futures
> > > >> > > > > > > > > would be
> > > >> > > > > > > > > > better. Will update the KIP.
> > > >> > > > > > > > > > 3. A typical example is a
> consumer
> > > subscribing
> > > >> to a
> > > >> > > regex
> > > >> > > > > > > > > pattern. We
> > > >> > > > > > > > > > request all topic metadata
> from the
> > > broker in
> > > >> order
> > > >> > > to
> > > >> > > > > > decide
> > > >> > > > > > > > > whether
> > > >> > > > > > > > > > the
> > > >> > > > > > > > > > pattern matches, expecting to
> > receive a
> > > list of
> > > >> > > authorised
> > > >> > > > > > > > > topics. The
> > > >> > > > > > > > > > user
> > > >> > > > > > > > > > is not asking to subscribe to
> an
> > > unauthorized
> > > >> topic.
> > > >> > > If
> > > >> > > > > > there
> > > >> > > > > > > > > are 10000
> > > >> > > > > > > > > > topics in the cluster and the
> user
> > has
> > > access
> > > >> to 100
> > > >> > > of
> > > >> > > > > > them,
> > > >> > > > > > > > at
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > moment
> > > >> > > > > > > > > > we log 9900 DENIED log
> entries at
> > INFO
> > > level in
> > > >> > > > > > > > > SimpleAclAuthorizer.
> > > >> > > > > > > > > > The
> > > >> > > > > > > > > > proposal is to authorize this
> > request with
> > > >> > > > > > > > > AuthorizationMode.FILTER, so
> > > >> > > > > > > > > > that authorizers can log
> resources
> > that
> > > are
> > > >> filtered
> > > >> > > out at
> > > >> > > > > > > > > lower level
> > > >> > > > > > > > > > like DEBUG since this is not
> an
> > attempt to
> > > >> access
> > > >> > > > > > > unauthorized
> > > >> > > > > > > > > > resources.
> > > >> > > > > > > > > > Brokers already handle these
> > differently
> > > since
> > > >> no
> > > >> > > > > > > authorization
> > > >> > > > > > > > > error
> > > >> > > > > > > > > > is
> > > >> > > > > > > > > > returned to the client in
> these
> > cases.
> > > Providing
> > > >> > > > > > > authorization
> > > >> > > > > > > > > mode to
> > > >> > > > > > > > > > authorizers enables authorizer
> > > implementations
> > > >> to
> > > >> > > generate
> > > >> > > > > > > > > better audit
> > > >> > > > > > > > > > logs.
> > > >> > > > > > > > > > 4. Each request may contain
> multiple
> > > instances
> > > >> of
> > > >> > > the same
> > > >> > > > > > > > > authorizable
> > > >> > > > > > > > > > resource. For example a
> produce
> > request
> > > may
> > > >> contain
> > > >> > > records
> > > >> > > > > > > for
> > > >> > > > > > > > > 10
> > > >> > > > > > > > > > partitions of the same topic.
> At the
> > > moment, we
> > > >> > > invoke
> > > >> > > > > > > > authorize
> > > >> > > > > > > > > > method 10
> > > >> > > > > > > > > > times. The proposal is to
> invoke it
> > once
> > > with
> > > >> > > count=10. The
> > > >> > > > > > > > > count is
> > > >> > > > > > > > > > provided to authorizer just
> for
> > audit
> > > logging
> > > >> > > purposes.
> > > >> > > > > > > > > > 5. Authorizer implements
> Closeable,
> > so you
> > > >> could use
> > > >> > > > > > close()
> > > >> > > > > > > to
> > > >> > > > > > > > > flush
> > > >> > > > > > > > > > audits?
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Thu, Aug 8, 2019 at 7:01
> AM Don
> > Bosco
> > > Durai <
> > > >> > > > > > > > bosco@apache.org
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Rajini
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Thanks for putting this
> together.
> > It is
> > > >> looking
> > > >> > > good. I
> > > >> > > > > > > have
> > > >> > > > > > > > > few
> > > >> > > > > > > > > > > questions...
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > 1. List<AuthorizationResult>
> > > authorize(...,
> > > >> > > List<Action>
> > > >> > > > > > > > > actions).
> > > >> > > > > > > > > > Do you
> > > >> > > > > > > > > > > see a scenario where the
> broker
> > will
> > > call
> > > >> > > authorize for
> > > >> > > > > > > > > multiple
> > > >> > > > > > > > > > topics at
> > > >> > > > > > > > > > > the same time? I can
> understand
> > that
> > > during
> > > >> > > > > > > creating/deleting
> > > >> > > > > > > > > ACLS,
> > > >> > > > > > > > > > > multiple permissions for
> multiple
> > > resources
> > > >> might
> > > >> > > be
> > > >> > > > > > done.
> > > >> > > > > > > > For
> > > >> > > > > > > > > > authorize
> > > >> > > > > > > > > > > call, would this be a case?
> And
> > does the
> > > >> Authorize
> > > >> > > > > > > > > implementation
> > > >> > > > > > > > > > will be
> > > >> > > > > > > > > > > able to do performance
> > optimization
> > > because of
> > > >> > > this? Or
> > > >> > > > > > > > should
> > > >> > > > > > > > > we
> > > >> > > > > > > > > > just keep
> > > >> > > > > > > > > > > it simple? I don't see it
> as an
> > issue
> > > from
> > > >> Apache
> > > >> > > Ranger
> > > >> > > > > > > > side,
> > > >> > > > > > > > > but
> > > >> > > > > > > > > > just
> > > >> > > > > > > > > > > checking to see whether we
> need
> > to be
> > > aware of
> > > >> > > something.
> > > >> > > > > > > > > > > 2. Should I assume that the
> > > SecurityProtocol
> > > >> passed
> > > >> > > > > > during
> > > >> > > > > > > > > start and
> > > >> > > > > > > > > > the
> > > >> > > > > > > > > > > one return by
> > > >> > > KafkaRequestContext.securityProtocol() will
> > > >> > > > > > > be
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > same?
> > > >> > > > > > > > > > > CompletableFuture<Void>
> > start(String
> > > >> listenerName,
> > > >> > > > > > > > > SecurityProtocol
> > > >> > > > > > > > > > > securityProtocol);
> > > >> > > > > > > > > > >
> > KafkaRequestContext.securityProtocol()
> > > >> > > > > > > > > > > 3. What is the purpose of
> > > AuthorizationMode?
> > > >> How
> > > >> > > does the
> > > >> > > > > > > > > broker
> > > >> > > > > > > > > > decide
> > > >> > > > > > > > > > > what mode to use when the
> > authorize()
> > > method
> > > >> is
> > > >> > > called?
> > > >> > > > > > > > > > > 4. Can we clarify "count" in
> > Action a
> > > bit
> > > >> more?
> > > >> > > How is it
> > > >> > > > > > > > used?
> > > >> > > > > > > > > > > 5. Do you feel having "stop"
> > along with
> > > >> "start" be
> > > >> > > > > > helpful?
> > > >> > > > > > > > > E.g. In
> > > >> > > > > > > > > > Ranger
> > > >> > > > > > > > > > > we try to optimize the Audit
> > writing by
> > > >> caching
> > > >> > > the logs
> > > >> > > > > > > for
> > > >> > > > > > > > a
> > > >> > > > > > > > > fixed
> > > >> > > > > > > > > > > interval. But when the
> Broker
> > > terminates, we
> > > >> do a
> > > >> > > forced
> > > >> > > > > > > > flush.
> > > >> > > > > > > > > > Having an
> > > >> > > > > > > > > > > explicit "stop" might give
> us a
> > formal
> > > way to
> > > >> > > flush our
> > > >> > > > > > > > audits.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Thanks
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Bosco
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On 8/7/19, 3:59 PM, "Rajini
> > Sivaram" <
> > > >> > > > > > > > rajinisivaram@gmail.com
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Hi Ron/Harsha/Satish,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Thanks for reviewing
> the KIP!
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > We should perhaps have
> a wider
> > > discussion
> > > >> > > outside
> > > >> > > > > > this
> > > >> > > > > > > > KIP
> > > >> > > > > > > > > for
> > > >> > > > > > > > > > > refactoring
> > > >> > > > > > > > > > > clients so that others
> who
> > are not
> > > >> following
> > > >> > > this KIP
> > > >> > > > > > > > also
> > > >> > > > > > > > > > notice the
> > > >> > > > > > > > > > > discussion. Satish,
> would you
> > like
> > > to
> > > >> start a
> > > >> > > > > > > discussion
> > > >> > > > > > > > > thread
> > > >> > > > > > > > > > on dev?
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Regards,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Rajini
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Wed, Aug 7, 2019 at
> 6:21 PM
> > > Satish
> > > >> Duggana <
> > > >> > > > > > > > > > > satish.duggana@gmail.com>
> > > >> > > > > > > > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > I felt the same need
> when
> > we want
> > > to
> > > >> add a
> > > >> > > > > > pluggable
> > > >> > > > > > > > API
> > > >> > > > > > > > > for
> > > >> > > > > > > > > > core
> > > >> > > > > > > > > > > > server functionality.
> This
> > does
> > > not
> > > >> need to
> > > >> > > be part
> > > >> > > > > > > of
> > > >> > > > > > > > > this
> > > >> > > > > > > > > > KIP, it
> > > >> > > > > > > > > > > > can be a separate
> KIP. I can
> > > contribute
> > > >> those
> > > >> > > > > > > > refactoring
> > > >> > > > > > > > > > changes if
> > > >> > > > > > > > > > > > others are OK with
> that.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > It is better to have a
> > structure
> > > like
> > > >> below.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > kafka-common:
> > > >> > > > > > > > > > > > common classes which
> can be
> > used
> > > in any
> > > >> of
> > > >> > > the
> > > >> > > > > > other
> > > >> > > > > > > > > modules
> > > >> > > > > > > > > > in Kafka
> > > >> > > > > > > > > > > > like client,
> > Kafka-server-common
> > > and
> > > >> server
> > > >> > > etc.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > kafka-client-common:
> > > >> > > > > > > > > > > > common classes which
> can be
> > used
> > > in the
> > > >> > > client
> > > >> > > > > > > module.
> > > >> > > > > > > > > This
> > > >> > > > > > > > > > can be
> > > >> > > > > > > > > > > > part of client module
> > itself.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > kafka-server-common:
> > > >> > > > > > > > > > > > classes required only
> for
> > > kafka-server.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Thanks.
> > > >> > > > > > > > > > > > Satish.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > On Wed, Aug 7, 2019
> at 9:28
> > PM
> > > Harsha
> > > >> > > Chintalapani
> > > >> > > > > > <
> > > >> > > > > > > > > > kafka@harsha.io>
> > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Thanks for the KIP
> Rajini.
> > > >> > > > > > > > > > > > > Quick thought, it
> would
> > be good
> > > to
> > > >> have a
> > > >> > > common
> > > >> > > > > > > > module
> > > >> > > > > > > > > > outside of
> > > >> > > > > > > > > > > > clients
> > > >> > > > > > > > > > > > > that only applies to
> > server side
> > > >> > > interfaces &
> > > >> > > > > > > > changes.
> > > >> > > > > > > > > It
> > > >> > > > > > > > > > looks
> > > >> > > > > > > > > > > like we
> > > >> > > > > > > > > > > > are
> > > >> > > > > > > > > > > > > increasingly in
> favor of
> > using
> > > Java
> > > >> > > interface for
> > > >> > > > > > > > > pluggable
> > > >> > > > > > > > > > > modules on
> > > >> > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > broker side.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > Harsha
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > On Tue, Aug 06,
> 2019 at
> > 2:31 PM,
> > > >> Rajini
> > > >> > > Sivaram <
> > > >> > > > > > > > > > > rajinisivaram@gmail.com
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Hi all,
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > I have created a
> KIP to
> > > replace the
> > > >> Scala
> > > >> > > > > > > > Authorizer
> > > >> > > > > > > > > API
> > > >> > > > > > > > > > with a
> > > >> > > > > > > > > > > new
> > > >> > > > > > > > > > > > Java
> > > >> > > > > > > > > > > > > > API:
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > -
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > > >> > > > > > > > > > > > > >
> > > >> > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > This is
> replacement for
> > KIP-50
> > > >> which was
> > > >> > > > > > accepted
> > > >> > > > > > > > > but never
> > > >> > > > > > > > > > > merged.
> > > >> > > > > > > > > > > > Apart
> > > >> > > > > > > > > > > > > > from moving to a
> Java
> > API
> > > consistent
> > > >> > > with other
> > > >> > > > > > > > > pluggable
> > > >> > > > > > > > > > > interfaces
> > > >> > > > > > > > > > > > in the
> > > >> > > > > > > > > > > > > > broker, KIP-504
> also
> > attempts
> > > to
> > > >> address
> > > >> > > known
> > > >> > > > > > > > > limitations
> > > >> > > > > > > > > > in the
> > > >> > > > > > > > > > > > > > authorizer. If
> you have
> > come
> > > across
> > > >> other
> > > >> > > > > > > > > limitations that
> > > >> > > > > > > > > > you
> > > >> > > > > > > > > > > would
> > > >> > > > > > > > > > > > like
> > > >> > > > > > > > > > > > > > to see addressed
> in the
> > new
> > > API,
> > > >> please
> > > >> > > raise
> > > >> > > > > > > these
> > > >> > > > > > > > > on the
> > > >> > > > > > > > > > > discussion
> > > >> > > > > > > > > > > > > > thread so that we
> can
> > > consider those
> > > >> > > too. All
> > > >> > > > > > > > > suggestions
> > > >> > > > > > > > > > and
> > > >> > > > > > > > > > > feedback
> > > >> > > > > > > > > > > > are
> > > >> > > > > > > > > > > > > > welcome.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Thank you,
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Rajini
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
>
>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Don Bosco Durai <bo...@apache.org>.
Hi Rajini
Help me understand this a bit more.
1. For all practical purpose, without authorization you can't go to the next step. The calling code needs to block anyway. So should we just let the implementation code do the async part?
2. If you feel management calls need to be async, then we should consider another set of async APIs. I don't feel we should complicate it further (
3. Another concern I have is wrt performance. Kafka has been built to handle 1000s per second requests. Not sure whether making it async will add any unnecessary overhead.
4. How much complication would this add on the calling side? And is it worth it?
Thanks
Bosco
On 9/3/19, 8:50 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
Hi all,
Ismael brought up a point that it will be good to make the Authorizer
interface asynchronous to avoid blocking request threads during remote
operations.
1) Since we want to support different backends for authorization metadata,
making createAcls() and deleteAcls() asynchronous makes sense since these
always involve remote operations. When KIP-500 removes ZooKeeper, we would
want to move ACLs to Kafka and async updates will avoid unnecessary
blocking.
2) For authorize() method, we currently use cached ACLs in the built-in
authorizers, so synchronous authorise operations work well now. But async
authorize() would support this scenario as well as authorizers in large
organisations where an LRU cache would enable a smaller cache even when the
backend holds a large amount of ACLs for infrequently used resources or
users who don't use the system frequently.
For both cases, the built-in authorizer will continue to be synchronous,
using CompletableFuture.completedFuture() to return the actual result. But
async API will make custom authorizer implementations more flexible. I
would like to know if there are any concerns with these changes before
updating the KIP.
*Proposed API:*
public interface Authorizer extends Configurable, Closeable {
Map<Endpoint, CompletionStage<Void>> start(AuthorizerServerInfo serverInfo);
List<CompletionStage<AuthorizationResult>>
authorize(AuthorizableRequestContext requestContext, List<Action>
actions);
List<CompletionStage<AclCreateResult>>
createAcls(AuthorizableRequestContext requestContext, List<AclBinding>
aclBindings);
List<CompletionStage<AclDeleteResult>>
deleteAcls(AuthorizableRequestContext requestContext,
List<AclBindingFilter> aclBindingFilters);
CompletionStage<Collection<AclBinding>> acls(AclBindingFilter filter);
}
Thank you,
Rajini
On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <bo...@apache.org> wrote:
> Hi Rajini
>
> Thanks for clarifying. I am good for now.
>
> Regards
>
> Bosco
>
>
> On 8/16/19, 11:30 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
>
> Hi Don,
>
> That should be fine. I guess Ranger loads policies from the database
> synchronously when the authorizer is configured, similar to
> SimpleAclAuthorizer loading from ZooKeeper. Ranger can continue to load
> synchronously from `configure()` or `start()` and return an empty map
> from
> `start()`. That would retain the existing behaviour.. When the same
> database stores policies for all listeners and the policies are not
> stored
> in Kafka, there is no value in making the load asynchronous.
>
> Regards,
>
> Rajini
>
>
> On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai <bo...@apache.org>
> wrote:
>
> > Hi Rajini
> >
> > Assuming this doesn't affect custom plugins, I don't have any
> concerns
> > regarding this.
> >
> > I do have one question regarding:
> >
> > "For authorizers that don’t store metadata in ZooKeeper, ensure that
> > authorizer metadata for each listener is available before starting
> up the
> > listener. This enables different authorization metadata stores for
> > different listeners."
> >
> > Since Ranger uses its own database for storing policies, do you
> anticipate
> > any issues?
> >
> > Thanks
> >
> > Bosco
> >
> >
> > On 8/16/19, 6:49 AM, "Rajini Sivaram" <ra...@gmail.com>
> wrote:
> >
> > Hi all,
> >
> > I made another change to the KIP. The KIP was originally
> proposing to
> > extend SimpleAclAuthorizer to also implement the new API (in
> addition
> > to
> > the existing API). But since we use the new API when available,
> this
> > can
> > break custom authorizers that extend this class and override
> specific
> > methods of the old API. To avoid breaking any existing custom
> > implementations that extend this class, particularly because it
> is in
> > the
> > public package kafka.security.auth, the KIP now proposes to
> retain the
> > old
> > authorizer as-is. SimpleAclAuthorizer will continue to
> implement the
> > old
> > API, but will be deprecated. A new authorizer implementation
> > kafka.security.authorizer.AclAuthorizer will be added for the
> new API
> > (this
> > will not be in the public package).
> >
> > Please let me know if you have any concerns.
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
> > rajinisivaram@gmail.com>
> > wrote:
> >
> > > Thanks Colin.
> > >
> > > If there are no other concerns, I will start vote later today.
> Many
> > thanks
> > > to every one for the feedback.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <
> cmccabe@apache.org>
> > wrote:
> > >
> > >> Thanks, Rajini. It looks good to me.
> > >>
> > >> best,
> > >> Colin
> > >>
> > >>
> > >> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram wrote:
> > >> > Hi Colin,
> > >> >
> > >> > Thanks for the review. I have updated the KIP to move the
> > interfaces for
> > >> > request context and server info to the authorizer package.
> These
> > are now
> > >> > called AuthorizableRequestContext and AuthorizerServerInfo.
> > Endpoint is
> > >> now
> > >> > a class in org.apache.kafka.common to make it reusable
> since we
> > already
> > >> > have multiple implementations of it. I have removed
> requestName
> > from the
> > >> > request context interface since authorizers can distinguish
> > follower
> > >> fetch
> > >> > and consumer fetch from the operation being authorized. So
> 16-bit
> > >> request
> > >> > type should be sufficient for audit logging. Also replaced
> > AuditFlag
> > >> with
> > >> > two booleans as you suggested.
> > >> >
> > >> > Can you take another look and see if the KIP is ready for
> voting?
> > >> >
> > >> > Thanks for all your help!
> > >> >
> > >> > Regards,
> > >> >
> > >> > Rajini
> > >> >
> > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <
> cmccabe@apache.org>
> > >> wrote:
> > >> >
> > >> > > Hi Rajini,
> > >> > >
> > >> > > I think it would be good to rename KafkaRequestContext to
> > something
> > >> like
> > >> > > AuthorizableRequestContext, and put it in the
> > >> > > org.apache.kafka.server.authorizer namespace. If we put
> it in
> > the
> > >> > > org.apache.kafka.common namespace, then it's not really
> clear
> > that
> > >> it's
> > >> > > part of the Authorizer API. Since this class is purely an
> > interface,
> > >> with
> > >> > > no concrete implementation of anything, there's nothing
> common
> > to
> > >> really
> > >> > > reuse in any case. We definitely don't want someone to
> > accidentally
> > >> add or
> > >> > > remove methods because they think this is just another
> internal
> > class
> > >> used
> > >> > > for requests.
> > >> > >
> > >> > > The BrokerInfo class is a nice improvement. It looks
> like it
> > will be
> > >> > > useful for passing in information about the context we're
> > running
> > >> in. It
> > >> > > would be nice to call this class ServerInfo rather than
> > BrokerInfo,
> > >> since
> > >> > > we will want to run the authorizer on controllers as well
> as on
> > >> brokers,
> > >> > > and the controller may run as a separate process post
> KIP-500.
> > I also
> > >> > > think that this class should be in the
> > >> org.apache.kafka.server.authorizer
> > >> > > namespace. Again, it is an interface, not a concrete
> > implementation,
> > >> and
> > >> > > it's an interface that is very specifically for the
> authorizer.
> > >> > >
> > >> > > I agree that we probably don't have enough information
> > preserved for
> > >> > > requests currently to always know what entity made them.
> So we
> > can
> > >> leave
> > >> > > that out for now (except in the special case of Fetch).
> > Perhaps we
> > >> can add
> > >> > > this later if it's needed.
> > >> > >
> > >> > > I understand the intention behind AuthorizationMode
> (which is
> > now
> > >> called
> > >> > > AuditFlag in the latest revision). But it still feels
> > complex. What
> > >> if we
> > >> > > just had two booleans in Action: logSuccesses and
> logFailures?
> > That
> > >> seems
> > >> > > to cover all the cases here. MANDATORY_AUTHORIZE = true,
> true.
> > >> > > OPTIONAL_AUTHORIZE = true, false. FILTER = true, false.
> > >> LIST_AUTHORIZED =
> > >> > > false, false. Would there be anything lost versus having
> the
> > enum?
> > >> > >
> > >> > > best,
> > >> > > Colin
> > >> > >
> > >> > >
> > >> > > On Wed, Aug 14, 2019, at 06:29, Mickael Maison wrote:
> > >> > > > Hi Rajini,
> > >> > > >
> > >> > > > Thanks for the KIP!
> > >> > > > I really like that authorize() will be able to take a
> batch of
> > >> > > > requests, this will speed up many implementations!
> > >> > > >
> > >> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <
> > >> rajinisivaram@gmail.com>
> > >> > > wrote:
> > >> > > > >
> > >> > > > > Thanks David! I have fixed the typo.
> > >> > > > >
> > >> > > > > Also made a couple of changes to make the context
> > interfaces more
> > >> > > generic.
> > >> > > > > KafkaRequestContext now returns the 16-bit API key as
> Colin
> > >> suggested
> > >> > > as
> > >> > > > > well as the friendly name used in metrics which are
> useful
> > in
> > >> audit
> > >> > > logs.
> > >> > > > > `Authorizer#start` is now provided a generic
> `BrokerInfo`
> > >> interface
> > >> > > that
> > >> > > > > gives cluster id, broker id and endpoint information.
> The
> > generic
> > >> > > interface
> > >> > > > > can potentially be used in other broker plugins in
> future
> > and
> > >> provides
> > >> > > > > dynamically generated configs like broker id and ports
> > which are
> > >> > > currently
> > >> > > > > not available to plugins unless these configs are
> statically
> > >> > > configured.
> > >> > > > > Please let me know if there are any concerns.
> > >> > > > >
> > >> > > > > Regards,
> > >> > > > >
> > >> > > > > Rajini
> > >> > > > >
> > >> > > > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot <
> > djacot@confluent.io>
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > Hi Rajini,
> > >> > > > > >
> > >> > > > > > Thank you for the update! It looks good to me.
> There is a
> > typo
> > >> in the
> > >> > > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` ->
> > >> `MANDATORY_AUTHORIZE`.
> > >> > > > > >
> > >> > > > > > Regards,
> > >> > > > > > David
> > >> > > > > >
> > >> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram <
> > >> > > rajinisivaram@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi David,
> > >> > > > > > >
> > >> > > > > > > Thanks for reviewing the KIP! Since questions
> about
> > >> `authorization
> > >> > > mode`
> > >> > > > > > > and `count` have come up multiple times, I have
> renamed
> > both.
> > >> > > > > > >
> > >> > > > > > > 1) Renamed `count` to `resourceReferenceCount`.
> It is
> > the
> > >> number
> > >> > > of times
> > >> > > > > > > the resource being authorized is referenced
> within the
> > >> request.
> > >> > > > > > >
> > >> > > > > > > 2) Renamed `AuthorizationMode` to `AuditFlag`. It
> is
> > provided
> > >> to
> > >> > > improve
> > >> > > > > > > audit logging in the authorizer. The enum values
> have
> > javadoc
> > >> which
> > >> > > > > > > indicate how the authorization result is used in
> each
> > of the
> > >> modes
> > >> > > to
> > >> > > > > > > enable authorizers to log audit messages at the
> right
> > severity
> > >> > > level.
> > >> > > > > > >
> > >> > > > > > > Regards,
> > >> > > > > > >
> > >> > > > > > > Rajini
> > >> > > > > > >
> > >> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM David Jacot <
> > >> djacot@confluent.io>
> > >> > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi Rajini,
> > >> > > > > > > >
> > >> > > > > > > > Thank you for the KIP. Overall, it looks good
> to me.
> > I have
> > >> few
> > >> > > > > > > > questions/suggestions:
> > >> > > > > > > >
> > >> > > > > > > > 1. It is hard to grasp what `Action#count` is
> for. I
> > guess I
> > >> > > understand
> > >> > > > > > > > where you want to go with it but it took me a
> while to
> > >> figure it
> > >> > > out.
> > >> > > > > > > > Perhaps, we could come up with a better name
> than
> > `count`?
> > >> > > > > > > >
> > >> > > > > > > > 2. I had a hard time trying to understand the
> > >> `AuthorizationMode`
> > >> > > > > > > concept,
> > >> > > > > > > > especially wrt. the OPTIONAL one. My
> understanding is
> > that
> > >> an
> > >> > > ACL is
> > >> > > > > > > either
> > >> > > > > > > > defined or not. Could you elaborate a bit more
> on
> > that?
> > >> > > > > > > >
> > >> > > > > > > > Thanks,
> > >> > > > > > > > David
> > >> > > > > > > >
> > >> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai
> <
> > >> > > bosco@apache.org>
> > >> > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Hi Rajini
> > >> > > > > > > > >
> > >> > > > > > > > > 3.2 - This makes sense. Thanks for clarifying.
> > >> > > > > > > > >
> > >> > > > > > > > > Rest looks fine. Once the implementations are
> done,
> > it
> > >> will be
> > >> > > more
> > >> > > > > > > clear
> > >> > > > > > > > > on the different values RequestType and Mode.
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks
> > >> > > > > > > > >
> > >> > > > > > > > > Bosco
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <
> > >> rajinisivaram@gmail.com
> > >> > > >
> > >> > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > Hi Don,
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks for the suggestions. A few
> responses
> > below:
> > >> > > > > > > > >
> > >> > > > > > > > > 3.1 Can rename and improve docs if we keep
> > this. Let's
> > >> > > finish the
> > >> > > > > > > > > discussion on Colin's suggestions
> regarding this
> > >> first.
> > >> > > > > > > > > 3.2 No, I was thinking of some requests
> that
> > have an
> > >> old
> > >> > > way of
> > >> > > > > > > > > authorizing
> > >> > > > > > > > > and a new way where we have retained the
> old
> > way for
> > >> > > backward
> > >> > > > > > > > > compatibility. One example is
> Cluster:Create
> > >> permission to
> > >> > > create
> > >> > > > > > > > > topics.
> > >> > > > > > > > > We have replaced this with fine-grained
> topic
> > create
> > >> > > access using
> > >> > > > > > > > > Topic:Create
> > >> > > > > > > > > for topic patterns. But we still check if
> user
> > has
> > >> > > Cluster:Create
> > >> > > > > > > > > first. If
> > >> > > > > > > > > Denied, the deny is ignored and we check
> > >> Topic:Create. We
> > >> > > dont
> > >> > > > > > want
> > >> > > > > > > > to
> > >> > > > > > > > > log
> > >> > > > > > > > > DENY for Cluster:Create at INFO level for
> this,
> > since
> > >> this
> > >> > > is
> > >> > > > > > not a
> > >> > > > > > > > > mandatory ACL for creating topics. We
> will get
> > >> appropriate
> > >> > > logs
> > >> > > > > > > from
> > >> > > > > > > > > the
> > >> > > > > > > > > subsequent Topic:Create in this case.
> > >> > > > > > > > > 3.3 They are not quite the same. FILTER
> implies
> > that
> > >> user
> > >> > > > > > actually
> > >> > > > > > > > > requested access to and performed some
> > operation on
> > >> the
> > >> > > filtered
> > >> > > > > > > > > resources.
> > >> > > > > > > > > LIST_AUTHORZED did not result in any
> actual
> > access.
> > >> User
> > >> > > simply
> > >> > > > > > > > wanted
> > >> > > > > > > > > to
> > >> > > > > > > > > know what they are allowed to access.
> > >> > > > > > > > > 3.4 Request types are Produce, JoinGroup,
> > OffsetCommit
> > >> > > etc. So
> > >> > > > > > that
> > >> > > > > > > > is
> > >> > > > > > > > > different from authorization mode,
> operation
> > etc.
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco
> Durai
> > <
> > >> > > > > > bosco@apache.org>
> > >> > > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Hi Rajini
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks for clarifying. This is very
> helpful...
> > >> > > > > > > > > >
> > >> > > > > > > > > > #1 - On the Ranger side, we should be
> able to
> > handle
> > >> > > multiple
> > >> > > > > > > > > requests at
> > >> > > > > > > > > > the same time. I was just not sure how
> much
> > >> processing
> > >> > > overhead
> > >> > > > > > > > will
> > >> > > > > > > > > be
> > >> > > > > > > > > > there on the Broker side to split and
> then
> > >> consolidate
> > >> > > the
> > >> > > > > > > results.
> > >> > > > > > > > > If it
> > >> > > > > > > > > > is negligible, then this is the better
> way.
> > It will
> > >> > > make it
> > >> > > > > > > future
> > >> > > > > > > > > proof.
> > >> > > > > > > > > > #2 - I agree, having a single "start"
> call
> > makes it
> > >> > > cleaner.
> > >> > > > > > The
> > >> > > > > > > > > > Authorization implementation will only
> have
> > to do
> > >> the
> > >> > > > > > > > initialization
> > >> > > > > > > > > only
> > >> > > > > > > > > > once.
> > >> > > > > > > > > > #3.1 - Thanks for the clarification. I
> think
> > it
> > >> makes
> > >> > > sense to
> > >> > > > > > > have
> > >> > > > > > > > > this.
> > >> > > > > > > > > > The term "Mode" might not be explicit
> enough.
> > >> > > Essentially it
> > >> > > > > > > seems
> > >> > > > > > > > > you want
> > >> > > > > > > > > > the Authorizer to know the
> Intent/Purpose of
> > the
> > >> > > authorize call
> > >> > > > > > > and
> > >> > > > > > > > > let the
> > >> > > > > > > > > > Authorizer decide what to log as Audit
> event.
> > >> Changing
> > >> > > the
> > >> > > > > > > > > class/field name
> > >> > > > > > > > > > or giving more documentation will do.
> > >> > > > > > > > > > #3.2 - Regarding the option "OPTIONAL".
> Are
> > you
> > >> thinking
> > >> > > from
> > >> > > > > > > > > chaining
> > >> > > > > > > > > > multiple Authorizer? If so, I am not
> sure
> > whether
> > >> the
> > >> > > Broker
> > >> > > > > > > would
> > >> > > > > > > > > have
> > >> > > > > > > > > > enough information to make that
> decision. I
> > feel the
> > >> > > Authorizer
> > >> > > > > > > > will
> > >> > > > > > > > > be the
> > >> > > > > > > > > > one who would have that knowledge. E.g.
> in
> > Ranger
> > >> we have
> > >> > > > > > > explicit
> > >> > > > > > > > > deny,
> > >> > > > > > > > > > which means no matter what, the request
> > should be
> > >> denied
> > >> > > for
> > >> > > > > > the
> > >> > > > > > > > > user/group
> > >> > > > > > > > > > or condition. So if you are thinking of
> > chaining
> > >> > > Authorizers,
> > >> > > > > > > then
> > >> > > > > > > > > > responses should have the third state,
> e.g.
> > >> > > "DENIED_FINAL", in
> > >> > > > > > > > which
> > >> > > > > > > > > case
> > >> > > > > > > > > > if there is an Authorization chain, it
> will
> > be stop
> > >> and
> > >> > > the
> > >> > > > > > > request
> > >> > > > > > > > > will be
> > >> > > > > > > > > > denied and if it is just denied, then
> you
> > might fall
> > >> > > back to
> > >> > > > > > next
> > >> > > > > > > > > > authorizer. If we don't have chaining of
> > >> Authorizing in
> > >> > > mind,
> > >> > > > > > > then
> > >> > > > > > > > we
> > >> > > > > > > > > > should reconsider OPTIONAL for now. Or
> > clarify under
> > >> > > which
> > >> > > > > > > scenario
> > >> > > > > > > > > > OPTIONAL will be called.
> > >> > > > > > > > > > #3.3 Regarding, FILTER v/s
> LIST_AUTHORIZED,
> > isn't
> > >> > > > > > > LIST_AUTHORIZED a
> > >> > > > > > > > > > special case for "FILTER"?
> > >> > > > > > > > > > #3.4 KafkaRequestContext. requestType()
> v/s
> > Action.
> > >> > > > > > > > > authorizationMode. I
> > >> > > > > > > > > > am not sure about the overlap or
> ambiguity.
> > >> > > > > > > > > > #4 - Cool. This is good, it will be less
> > stress on
> > >> the
> > >> > > > > > > Authorizer.
> > >> > > > > > > > > Ranger
> > >> > > > > > > > > > already supports the "count" concept
> and also
> > has
> > >> > > batching
> > >> > > > > > > > > capability to
> > >> > > > > > > > > > aggregate similar requests to reduce the
> > number of
> > >> audit
> > >> > > logs
> > >> > > > > > to
> > >> > > > > > > > > write. We
> > >> > > > > > > > > > should be able to pass this through.
> > >> > > > > > > > > > #5 - Assuming if the object instance is
> going
> > out of
> > >> > > scope, we
> > >> > > > > > > > > should be
> > >> > > > > > > > > > fine. Not a super important ask. Ranger
> is
> > already
> > >> > > catching
> > >> > > > > > KILL
> > >> > > > > > > > > signal for
> > >> > > > > > > > > > clean up.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks again. These are good
> enhancements.
> > >> Appreciate
> > >> > > your
> > >> > > > > > > efforts
> > >> > > > > > > > > here.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Bosco
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On 8/8/19, 2:03 AM, "Rajini Sivaram" <
> > >> > > rajinisivaram@gmail.com
> > >> > > > > > >
> > >> > > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > Hi Don,
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks for reviewing the KIP.
> > >> > > > > > > > > >
> > >> > > > > > > > > > 1. I had this originally as a single
> > Action, but
> > >> > > thought it
> > >> > > > > > > may
> > >> > > > > > > > > be
> > >> > > > > > > > > > useful
> > >> > > > > > > > > > to support batched authorize calls
> as
> > well and
> > >> keep
> > >> > > it
> > >> > > > > > > > > consistent with
> > >> > > > > > > > > > other methods. Single requests can
> contain
> > >> multiple
> > >> > > topics.
> > >> > > > > > > For
> > >> > > > > > > > > > example a
> > >> > > > > > > > > > produce request can contain records
> for
> > several
> > >> > > partitions
> > >> > > > > > of
> > >> > > > > > > > > different
> > >> > > > > > > > > > topics. Broker could potentially
> > authorize these
> > >> > > together.
> > >> > > > > > > For
> > >> > > > > > > > > > SimpleAclAuthorizer, batched
> authorize
> > methods
> > >> don't
> > >> > > > > > provide
> > >> > > > > > > > any
> > >> > > > > > > > > > optimisation since lookup is based
> on
> > resources
> > >> > > followed by
> > >> > > > > > > the
> > >> > > > > > > > > > matching
> > >> > > > > > > > > > logic. But some authorizers may
> manage
> > ACLs by
> > >> user
> > >> > > > > > principal
> > >> > > > > > > > > rather
> > >> > > > > > > > > > than
> > >> > > > > > > > > > resource and may be able to optimize
> > batched
> > >> > > requests. I am
> > >> > > > > > > ok
> > >> > > > > > > > > with
> > >> > > > > > > > > > using
> > >> > > > > > > > > > single Action if this is likely to
> cause
> > issues.
> > >> > > > > > > > > > 2. If you have two listeners, one
> for
> > >> inter-broker
> > >> > > traffic
> > >> > > > > > > and
> > >> > > > > > > > > another
> > >> > > > > > > > > > for
> > >> > > > > > > > > > external clients, start method is
> invoked
> > twice,
> > >> > > once for
> > >> > > > > > > each
> > >> > > > > > > > > > listener. On
> > >> > > > > > > > > > second thought, that may be
> confusing and
> > a
> > >> single
> > >> > > start()
> > >> > > > > > > > > invocation
> > >> > > > > > > > > > that
> > >> > > > > > > > > > provides all listener information
> and
> > returns
> > >> > > multiple
> > >> > > > > > > futures
> > >> > > > > > > > > would be
> > >> > > > > > > > > > better. Will update the KIP.
> > >> > > > > > > > > > 3. A typical example is a consumer
> > subscribing
> > >> to a
> > >> > > regex
> > >> > > > > > > > > pattern. We
> > >> > > > > > > > > > request all topic metadata from the
> > broker in
> > >> order
> > >> > > to
> > >> > > > > > decide
> > >> > > > > > > > > whether
> > >> > > > > > > > > > the
> > >> > > > > > > > > > pattern matches, expecting to
> receive a
> > list of
> > >> > > authorised
> > >> > > > > > > > > topics. The
> > >> > > > > > > > > > user
> > >> > > > > > > > > > is not asking to subscribe to an
> > unauthorized
> > >> topic.
> > >> > > If
> > >> > > > > > there
> > >> > > > > > > > > are 10000
> > >> > > > > > > > > > topics in the cluster and the user
> has
> > access
> > >> to 100
> > >> > > of
> > >> > > > > > them,
> > >> > > > > > > > at
> > >> > > > > > > > > the
> > >> > > > > > > > > > moment
> > >> > > > > > > > > > we log 9900 DENIED log entries at
> INFO
> > level in
> > >> > > > > > > > > SimpleAclAuthorizer.
> > >> > > > > > > > > > The
> > >> > > > > > > > > > proposal is to authorize this
> request with
> > >> > > > > > > > > AuthorizationMode.FILTER, so
> > >> > > > > > > > > > that authorizers can log resources
> that
> > are
> > >> filtered
> > >> > > out at
> > >> > > > > > > > > lower level
> > >> > > > > > > > > > like DEBUG since this is not an
> attempt to
> > >> access
> > >> > > > > > > unauthorized
> > >> > > > > > > > > > resources.
> > >> > > > > > > > > > Brokers already handle these
> differently
> > since
> > >> no
> > >> > > > > > > authorization
> > >> > > > > > > > > error
> > >> > > > > > > > > > is
> > >> > > > > > > > > > returned to the client in these
> cases.
> > Providing
> > >> > > > > > > authorization
> > >> > > > > > > > > mode to
> > >> > > > > > > > > > authorizers enables authorizer
> > implementations
> > >> to
> > >> > > generate
> > >> > > > > > > > > better audit
> > >> > > > > > > > > > logs.
> > >> > > > > > > > > > 4. Each request may contain multiple
> > instances
> > >> of
> > >> > > the same
> > >> > > > > > > > > authorizable
> > >> > > > > > > > > > resource. For example a produce
> request
> > may
> > >> contain
> > >> > > records
> > >> > > > > > > for
> > >> > > > > > > > > 10
> > >> > > > > > > > > > partitions of the same topic. At the
> > moment, we
> > >> > > invoke
> > >> > > > > > > > authorize
> > >> > > > > > > > > > method 10
> > >> > > > > > > > > > times. The proposal is to invoke it
> once
> > with
> > >> > > count=10. The
> > >> > > > > > > > > count is
> > >> > > > > > > > > > provided to authorizer just for
> audit
> > logging
> > >> > > purposes.
> > >> > > > > > > > > > 5. Authorizer implements Closeable,
> so you
> > >> could use
> > >> > > > > > close()
> > >> > > > > > > to
> > >> > > > > > > > > flush
> > >> > > > > > > > > > audits?
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Thu, Aug 8, 2019 at 7:01 AM Don
> Bosco
> > Durai <
> > >> > > > > > > > bosco@apache.org
> > >> > > > > > > > > >
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Rajini
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Thanks for putting this together.
> It is
> > >> looking
> > >> > > good. I
> > >> > > > > > > have
> > >> > > > > > > > > few
> > >> > > > > > > > > > > questions...
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > 1. List<AuthorizationResult>
> > authorize(...,
> > >> > > List<Action>
> > >> > > > > > > > > actions).
> > >> > > > > > > > > > Do you
> > >> > > > > > > > > > > see a scenario where the broker
> will
> > call
> > >> > > authorize for
> > >> > > > > > > > > multiple
> > >> > > > > > > > > > topics at
> > >> > > > > > > > > > > the same time? I can understand
> that
> > during
> > >> > > > > > > creating/deleting
> > >> > > > > > > > > ACLS,
> > >> > > > > > > > > > > multiple permissions for multiple
> > resources
> > >> might
> > >> > > be
> > >> > > > > > done.
> > >> > > > > > > > For
> > >> > > > > > > > > > authorize
> > >> > > > > > > > > > > call, would this be a case? And
> does the
> > >> Authorize
> > >> > > > > > > > > implementation
> > >> > > > > > > > > > will be
> > >> > > > > > > > > > > able to do performance
> optimization
> > because of
> > >> > > this? Or
> > >> > > > > > > > should
> > >> > > > > > > > > we
> > >> > > > > > > > > > just keep
> > >> > > > > > > > > > > it simple? I don't see it as an
> issue
> > from
> > >> Apache
> > >> > > Ranger
> > >> > > > > > > > side,
> > >> > > > > > > > > but
> > >> > > > > > > > > > just
> > >> > > > > > > > > > > checking to see whether we need
> to be
> > aware of
> > >> > > something.
> > >> > > > > > > > > > > 2. Should I assume that the
> > SecurityProtocol
> > >> passed
> > >> > > > > > during
> > >> > > > > > > > > start and
> > >> > > > > > > > > > the
> > >> > > > > > > > > > > one return by
> > >> > > KafkaRequestContext.securityProtocol() will
> > >> > > > > > > be
> > >> > > > > > > > > the
> > >> > > > > > > > > > same?
> > >> > > > > > > > > > > CompletableFuture<Void>
> start(String
> > >> listenerName,
> > >> > > > > > > > > SecurityProtocol
> > >> > > > > > > > > > > securityProtocol);
> > >> > > > > > > > > > >
> KafkaRequestContext.securityProtocol()
> > >> > > > > > > > > > > 3. What is the purpose of
> > AuthorizationMode?
> > >> How
> > >> > > does the
> > >> > > > > > > > > broker
> > >> > > > > > > > > > decide
> > >> > > > > > > > > > > what mode to use when the
> authorize()
> > method
> > >> is
> > >> > > called?
> > >> > > > > > > > > > > 4. Can we clarify "count" in
> Action a
> > bit
> > >> more?
> > >> > > How is it
> > >> > > > > > > > used?
> > >> > > > > > > > > > > 5. Do you feel having "stop"
> along with
> > >> "start" be
> > >> > > > > > helpful?
> > >> > > > > > > > > E.g. In
> > >> > > > > > > > > > Ranger
> > >> > > > > > > > > > > we try to optimize the Audit
> writing by
> > >> caching
> > >> > > the logs
> > >> > > > > > > for
> > >> > > > > > > > a
> > >> > > > > > > > > fixed
> > >> > > > > > > > > > > interval. But when the Broker
> > terminates, we
> > >> do a
> > >> > > forced
> > >> > > > > > > > flush.
> > >> > > > > > > > > > Having an
> > >> > > > > > > > > > > explicit "stop" might give us a
> formal
> > way to
> > >> > > flush our
> > >> > > > > > > > audits.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Thanks
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Bosco
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On 8/7/19, 3:59 PM, "Rajini
> Sivaram" <
> > >> > > > > > > > rajinisivaram@gmail.com
> > >> > > > > > > > > >
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Hi Ron/Harsha/Satish,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Thanks for reviewing the KIP!
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > We should perhaps have a wider
> > discussion
> > >> > > outside
> > >> > > > > > this
> > >> > > > > > > > KIP
> > >> > > > > > > > > for
> > >> > > > > > > > > > > refactoring
> > >> > > > > > > > > > > clients so that others who
> are not
> > >> following
> > >> > > this KIP
> > >> > > > > > > > also
> > >> > > > > > > > > > notice the
> > >> > > > > > > > > > > discussion. Satish, would you
> like
> > to
> > >> start a
> > >> > > > > > > discussion
> > >> > > > > > > > > thread
> > >> > > > > > > > > > on dev?
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Regards,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Rajini
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Wed, Aug 7, 2019 at 6:21 PM
> > Satish
> > >> Duggana <
> > >> > > > > > > > > > > satish.duggana@gmail.com>
> > >> > > > > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > I felt the same need when
> we want
> > to
> > >> add a
> > >> > > > > > pluggable
> > >> > > > > > > > API
> > >> > > > > > > > > for
> > >> > > > > > > > > > core
> > >> > > > > > > > > > > > server functionality. This
> does
> > not
> > >> need to
> > >> > > be part
> > >> > > > > > > of
> > >> > > > > > > > > this
> > >> > > > > > > > > > KIP, it
> > >> > > > > > > > > > > > can be a separate KIP. I can
> > contribute
> > >> those
> > >> > > > > > > > refactoring
> > >> > > > > > > > > > changes if
> > >> > > > > > > > > > > > others are OK with that.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > It is better to have a
> structure
> > like
> > >> below.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > kafka-common:
> > >> > > > > > > > > > > > common classes which can be
> used
> > in any
> > >> of
> > >> > > the
> > >> > > > > > other
> > >> > > > > > > > > modules
> > >> > > > > > > > > > in Kafka
> > >> > > > > > > > > > > > like client,
> Kafka-server-common
> > and
> > >> server
> > >> > > etc.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > kafka-client-common:
> > >> > > > > > > > > > > > common classes which can be
> used
> > in the
> > >> > > client
> > >> > > > > > > module.
> > >> > > > > > > > > This
> > >> > > > > > > > > > can be
> > >> > > > > > > > > > > > part of client module
> itself.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > kafka-server-common:
> > >> > > > > > > > > > > > classes required only for
> > kafka-server.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Thanks.
> > >> > > > > > > > > > > > Satish.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Wed, Aug 7, 2019 at 9:28
> PM
> > Harsha
> > >> > > Chintalapani
> > >> > > > > > <
> > >> > > > > > > > > > kafka@harsha.io>
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Thanks for the KIP Rajini.
> > >> > > > > > > > > > > > > Quick thought, it would
> be good
> > to
> > >> have a
> > >> > > common
> > >> > > > > > > > module
> > >> > > > > > > > > > outside of
> > >> > > > > > > > > > > > clients
> > >> > > > > > > > > > > > > that only applies to
> server side
> > >> > > interfaces &
> > >> > > > > > > > changes.
> > >> > > > > > > > > It
> > >> > > > > > > > > > looks
> > >> > > > > > > > > > > like we
> > >> > > > > > > > > > > > are
> > >> > > > > > > > > > > > > increasingly in favor of
> using
> > Java
> > >> > > interface for
> > >> > > > > > > > > pluggable
> > >> > > > > > > > > > > modules on
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > > broker side.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > Harsha
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > On Tue, Aug 06, 2019 at
> 2:31 PM,
> > >> Rajini
> > >> > > Sivaram <
> > >> > > > > > > > > > > rajinisivaram@gmail.com
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Hi all,
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > I have created a KIP to
> > replace the
> > >> Scala
> > >> > > > > > > > Authorizer
> > >> > > > > > > > > API
> > >> > > > > > > > > > with a
> > >> > > > > > > > > > > new
> > >> > > > > > > > > > > > Java
> > >> > > > > > > > > > > > > > API:
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > -
> > >> > > > > > > > > > > > > >
> > >> > > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/
> > >> > > > > > > > > > > > > >
> > >> > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > This is replacement for
> KIP-50
> > >> which was
> > >> > > > > > accepted
> > >> > > > > > > > > but never
> > >> > > > > > > > > > > merged.
> > >> > > > > > > > > > > > Apart
> > >> > > > > > > > > > > > > > from moving to a Java
> API
> > consistent
> > >> > > with other
> > >> > > > > > > > > pluggable
> > >> > > > > > > > > > > interfaces
> > >> > > > > > > > > > > > in the
> > >> > > > > > > > > > > > > > broker, KIP-504 also
> attempts
> > to
> > >> address
> > >> > > known
> > >> > > > > > > > > limitations
> > >> > > > > > > > > > in the
> > >> > > > > > > > > > > > > > authorizer. If you have
> come
> > across
> > >> other
> > >> > > > > > > > > limitations that
> > >> > > > > > > > > > you
> > >> > > > > > > > > > > would
> > >> > > > > > > > > > > > like
> > >> > > > > > > > > > > > > > to see addressed in the
> new
> > API,
> > >> please
> > >> > > raise
> > >> > > > > > > these
> > >> > > > > > > > > on the
> > >> > > > > > > > > > > discussion
> > >> > > > > > > > > > > > > > thread so that we can
> > consider those
> > >> > > too. All
> > >> > > > > > > > > suggestions
> > >> > > > > > > > > > and
> > >> > > > > > > > > > > feedback
> > >> > > > > > > > > > > > are
> > >> > > > > > > > > > > > > > welcome.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Thank you,
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Rajini
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Colin McCabe <cm...@apache.org>.
Hi Rajini,
That's an interesting point. I guess I assumed that we would always cache metadata locally, so that a synchronous operation would be OK here. But, I suppose as time goes on, we will eventually want paged authorization metadata.
If the operations are done asynchronously, then that implies background thread(s) at work somewhere. We should specify when that these thread(s) are created, and when they are stopped. Probably they should be created in start, and stopped in close? We should update the JavaDoc.
best,
Colin
On Tue, Sep 3, 2019, at 05:50, Rajini Sivaram wrote:
> Hi all,
>
> Ismael brought up a point that it will be good to make the Authorizer
> interface asynchronous to avoid blocking request threads during remote
> operations.
>
> 1) Since we want to support different backends for authorization metadata,
> making createAcls() and deleteAcls() asynchronous makes sense since these
> always involve remote operations. When KIP-500 removes ZooKeeper, we would
> want to move ACLs to Kafka and async updates will avoid unnecessary
> blocking.
> 2) For authorize() method, we currently use cached ACLs in the built-in
> authorizers, so synchronous authorise operations work well now. But async
> authorize() would support this scenario as well as authorizers in large
> organisations where an LRU cache would enable a smaller cache even when the
> backend holds a large amount of ACLs for infrequently used resources or
> users who don't use the system frequently.
>
> For both cases, the built-in authorizer will continue to be synchronous,
> using CompletableFuture.completedFuture() to return the actual result. But
> async API will make custom authorizer implementations more flexible. I
> would like to know if there are any concerns with these changes before
> updating the KIP.
>
> *Proposed API:*
> public interface Authorizer extends Configurable, Closeable {
>
> Map<Endpoint, CompletionStage<Void>> start(AuthorizerServerInfo serverInfo);
> List<CompletionStage<AuthorizationResult>>
> authorize(AuthorizableRequestContext requestContext, List<Action>
> actions);
> List<CompletionStage<AclCreateResult>>
> createAcls(AuthorizableRequestContext requestContext, List<AclBinding>
> aclBindings);
> List<CompletionStage<AclDeleteResult>>
> deleteAcls(AuthorizableRequestContext requestContext,
> List<AclBindingFilter> aclBindingFilters);
> CompletionStage<Collection<AclBinding>> acls(AclBindingFilter filter);
> }
>
>
> Thank you,
>
> Rajini
>
> On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <bo...@apache.org> wrote:
>
> > Hi Rajini
> >
> > Thanks for clarifying. I am good for now.
> >
> > Regards
> >
> > Bosco
> >
> >
> > On 8/16/19, 11:30 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
> >
> > Hi Don,
> >
> > That should be fine. I guess Ranger loads policies from the database
> > synchronously when the authorizer is configured, similar to
> > SimpleAclAuthorizer loading from ZooKeeper. Ranger can continue to load
> > synchronously from `configure()` or `start()` and return an empty map
> > from
> > `start()`. That would retain the existing behaviour.. When the same
> > database stores policies for all listeners and the policies are not
> > stored
> > in Kafka, there is no value in making the load asynchronous.
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai <bo...@apache.org>
> > wrote:
> >
> > > Hi Rajini
> > >
> > > Assuming this doesn't affect custom plugins, I don't have any
> > concerns
> > > regarding this.
> > >
> > > I do have one question regarding:
> > >
> > > "For authorizers that don’t store metadata in ZooKeeper, ensure that
> > > authorizer metadata for each listener is available before starting
> > up the
> > > listener. This enables different authorization metadata stores for
> > > different listeners."
> > >
> > > Since Ranger uses its own database for storing policies, do you
> > anticipate
> > > any issues?
> > >
> > > Thanks
> > >
> > > Bosco
> > >
> > >
> > > On 8/16/19, 6:49 AM, "Rajini Sivaram" <ra...@gmail.com>
> > wrote:
> > >
> > > Hi all,
> > >
> > > I made another change to the KIP. The KIP was originally
> > proposing to
> > > extend SimpleAclAuthorizer to also implement the new API (in
> > addition
> > > to
> > > the existing API). But since we use the new API when available,
> > this
> > > can
> > > break custom authorizers that extend this class and override
> > specific
> > > methods of the old API. To avoid breaking any existing custom
> > > implementations that extend this class, particularly because it
> > is in
> > > the
> > > public package kafka.security.auth, the KIP now proposes to
> > retain the
> > > old
> > > authorizer as-is. SimpleAclAuthorizer will continue to
> > implement the
> > > old
> > > API, but will be deprecated. A new authorizer implementation
> > > kafka.security.authorizer.AclAuthorizer will be added for the
> > new API
> > > (this
> > > will not be in the public package).
> > >
> > > Please let me know if you have any concerns.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > > On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
> > > rajinisivaram@gmail.com>
> > > wrote:
> > >
> > > > Thanks Colin.
> > > >
> > > > If there are no other concerns, I will start vote later today.
> > Many
> > > thanks
> > > > to every one for the feedback.
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <
> > cmccabe@apache.org>
> > > wrote:
> > > >
> > > >> Thanks, Rajini. It looks good to me.
> > > >>
> > > >> best,
> > > >> Colin
> > > >>
> > > >>
> > > >> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram wrote:
> > > >> > Hi Colin,
> > > >> >
> > > >> > Thanks for the review. I have updated the KIP to move the
> > > interfaces for
> > > >> > request context and server info to the authorizer package.
> > These
> > > are now
> > > >> > called AuthorizableRequestContext and AuthorizerServerInfo.
> > > Endpoint is
> > > >> now
> > > >> > a class in org.apache.kafka.common to make it reusable
> > since we
> > > already
> > > >> > have multiple implementations of it. I have removed
> > requestName
> > > from the
> > > >> > request context interface since authorizers can distinguish
> > > follower
> > > >> fetch
> > > >> > and consumer fetch from the operation being authorized. So
> > 16-bit
> > > >> request
> > > >> > type should be sufficient for audit logging. Also replaced
> > > AuditFlag
> > > >> with
> > > >> > two booleans as you suggested.
> > > >> >
> > > >> > Can you take another look and see if the KIP is ready for
> > voting?
> > > >> >
> > > >> > Thanks for all your help!
> > > >> >
> > > >> > Regards,
> > > >> >
> > > >> > Rajini
> > > >> >
> > > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <
> > cmccabe@apache.org>
> > > >> wrote:
> > > >> >
> > > >> > > Hi Rajini,
> > > >> > >
> > > >> > > I think it would be good to rename KafkaRequestContext to
> > > something
> > > >> like
> > > >> > > AuthorizableRequestContext, and put it in the
> > > >> > > org.apache.kafka.server.authorizer namespace. If we put
> > it in
> > > the
> > > >> > > org.apache.kafka.common namespace, then it's not really
> > clear
> > > that
> > > >> it's
> > > >> > > part of the Authorizer API. Since this class is purely an
> > > interface,
> > > >> with
> > > >> > > no concrete implementation of anything, there's nothing
> > common
> > > to
> > > >> really
> > > >> > > reuse in any case. We definitely don't want someone to
> > > accidentally
> > > >> add or
> > > >> > > remove methods because they think this is just another
> > internal
> > > class
> > > >> used
> > > >> > > for requests.
> > > >> > >
> > > >> > > The BrokerInfo class is a nice improvement. It looks
> > like it
> > > will be
> > > >> > > useful for passing in information about the context we're
> > > running
> > > >> in. It
> > > >> > > would be nice to call this class ServerInfo rather than
> > > BrokerInfo,
> > > >> since
> > > >> > > we will want to run the authorizer on controllers as well
> > as on
> > > >> brokers,
> > > >> > > and the controller may run as a separate process post
> > KIP-500.
> > > I also
> > > >> > > think that this class should be in the
> > > >> org.apache.kafka.server.authorizer
> > > >> > > namespace. Again, it is an interface, not a concrete
> > > implementation,
> > > >> and
> > > >> > > it's an interface that is very specifically for the
> > authorizer.
> > > >> > >
> > > >> > > I agree that we probably don't have enough information
> > > preserved for
> > > >> > > requests currently to always know what entity made them.
> > So we
> > > can
> > > >> leave
> > > >> > > that out for now (except in the special case of Fetch).
> > > Perhaps we
> > > >> can add
> > > >> > > this later if it's needed.
> > > >> > >
> > > >> > > I understand the intention behind AuthorizationMode
> > (which is
> > > now
> > > >> called
> > > >> > > AuditFlag in the latest revision). But it still feels
> > > complex. What
> > > >> if we
> > > >> > > just had two booleans in Action: logSuccesses and
> > logFailures?
> > > That
> > > >> seems
> > > >> > > to cover all the cases here. MANDATORY_AUTHORIZE = true,
> > true.
> > > >> > > OPTIONAL_AUTHORIZE = true, false. FILTER = true, false.
> > > >> LIST_AUTHORIZED =
> > > >> > > false, false. Would there be anything lost versus having
> > the
> > > enum?
> > > >> > >
> > > >> > > best,
> > > >> > > Colin
> > > >> > >
> > > >> > >
> > > >> > > On Wed, Aug 14, 2019, at 06:29, Mickael Maison wrote:
> > > >> > > > Hi Rajini,
> > > >> > > >
> > > >> > > > Thanks for the KIP!
> > > >> > > > I really like that authorize() will be able to take a
> > batch of
> > > >> > > > requests, this will speed up many implementations!
> > > >> > > >
> > > >> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <
> > > >> rajinisivaram@gmail.com>
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > Thanks David! I have fixed the typo.
> > > >> > > > >
> > > >> > > > > Also made a couple of changes to make the context
> > > interfaces more
> > > >> > > generic.
> > > >> > > > > KafkaRequestContext now returns the 16-bit API key as
> > Colin
> > > >> suggested
> > > >> > > as
> > > >> > > > > well as the friendly name used in metrics which are
> > useful
> > > in
> > > >> audit
> > > >> > > logs.
> > > >> > > > > `Authorizer#start` is now provided a generic
> > `BrokerInfo`
> > > >> interface
> > > >> > > that
> > > >> > > > > gives cluster id, broker id and endpoint information.
> > The
> > > generic
> > > >> > > interface
> > > >> > > > > can potentially be used in other broker plugins in
> > future
> > > and
> > > >> provides
> > > >> > > > > dynamically generated configs like broker id and ports
> > > which are
> > > >> > > currently
> > > >> > > > > not available to plugins unless these configs are
> > statically
> > > >> > > configured.
> > > >> > > > > Please let me know if there are any concerns.
> > > >> > > > >
> > > >> > > > > Regards,
> > > >> > > > >
> > > >> > > > > Rajini
> > > >> > > > >
> > > >> > > > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot <
> > > djacot@confluent.io>
> > > >> > > wrote:
> > > >> > > > >
> > > >> > > > > > Hi Rajini,
> > > >> > > > > >
> > > >> > > > > > Thank you for the update! It looks good to me.
> > There is a
> > > typo
> > > >> in the
> > > >> > > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` ->
> > > >> `MANDATORY_AUTHORIZE`.
> > > >> > > > > >
> > > >> > > > > > Regards,
> > > >> > > > > > David
> > > >> > > > > >
> > > >> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram <
> > > >> > > rajinisivaram@gmail.com>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi David,
> > > >> > > > > > >
> > > >> > > > > > > Thanks for reviewing the KIP! Since questions
> > about
> > > >> `authorization
> > > >> > > mode`
> > > >> > > > > > > and `count` have come up multiple times, I have
> > renamed
> > > both.
> > > >> > > > > > >
> > > >> > > > > > > 1) Renamed `count` to `resourceReferenceCount`.
> > It is
> > > the
> > > >> number
> > > >> > > of times
> > > >> > > > > > > the resource being authorized is referenced
> > within the
> > > >> request.
> > > >> > > > > > >
> > > >> > > > > > > 2) Renamed `AuthorizationMode` to `AuditFlag`. It
> > is
> > > provided
> > > >> to
> > > >> > > improve
> > > >> > > > > > > audit logging in the authorizer. The enum values
> > have
> > > javadoc
> > > >> which
> > > >> > > > > > > indicate how the authorization result is used in
> > each
> > > of the
> > > >> modes
> > > >> > > to
> > > >> > > > > > > enable authorizers to log audit messages at the
> > right
> > > severity
> > > >> > > level.
> > > >> > > > > > >
> > > >> > > > > > > Regards,
> > > >> > > > > > >
> > > >> > > > > > > Rajini
> > > >> > > > > > >
> > > >> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM David Jacot <
> > > >> djacot@confluent.io>
> > > >> > > wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Hi Rajini,
> > > >> > > > > > > >
> > > >> > > > > > > > Thank you for the KIP. Overall, it looks good
> > to me.
> > > I have
> > > >> few
> > > >> > > > > > > > questions/suggestions:
> > > >> > > > > > > >
> > > >> > > > > > > > 1. It is hard to grasp what `Action#count` is
> > for. I
> > > guess I
> > > >> > > understand
> > > >> > > > > > > > where you want to go with it but it took me a
> > while to
> > > >> figure it
> > > >> > > out.
> > > >> > > > > > > > Perhaps, we could come up with a better name
> > than
> > > `count`?
> > > >> > > > > > > >
> > > >> > > > > > > > 2. I had a hard time trying to understand the
> > > >> `AuthorizationMode`
> > > >> > > > > > > concept,
> > > >> > > > > > > > especially wrt. the OPTIONAL one. My
> > understanding is
> > > that
> > > >> an
> > > >> > > ACL is
> > > >> > > > > > > either
> > > >> > > > > > > > defined or not. Could you elaborate a bit more
> > on
> > > that?
> > > >> > > > > > > >
> > > >> > > > > > > > Thanks,
> > > >> > > > > > > > David
> > > >> > > > > > > >
> > > >> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai
> > <
> > > >> > > bosco@apache.org>
> > > >> > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Hi Rajini
> > > >> > > > > > > > >
> > > >> > > > > > > > > 3.2 - This makes sense. Thanks for clarifying.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Rest looks fine. Once the implementations are
> > done,
> > > it
> > > >> will be
> > > >> > > more
> > > >> > > > > > > clear
> > > >> > > > > > > > > on the different values RequestType and Mode.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks
> > > >> > > > > > > > >
> > > >> > > > > > > > > Bosco
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <
> > > >> rajinisivaram@gmail.com
> > > >> > > >
> > > >> > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > Hi Don,
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thanks for the suggestions. A few
> > responses
> > > below:
> > > >> > > > > > > > >
> > > >> > > > > > > > > 3.1 Can rename and improve docs if we keep
> > > this. Let's
> > > >> > > finish the
> > > >> > > > > > > > > discussion on Colin's suggestions
> > regarding this
> > > >> first.
> > > >> > > > > > > > > 3.2 No, I was thinking of some requests
> > that
> > > have an
> > > >> old
> > > >> > > way of
> > > >> > > > > > > > > authorizing
> > > >> > > > > > > > > and a new way where we have retained the
> > old
> > > way for
> > > >> > > backward
> > > >> > > > > > > > > compatibility. One example is
> > Cluster:Create
> > > >> permission to
> > > >> > > create
> > > >> > > > > > > > > topics.
> > > >> > > > > > > > > We have replaced this with fine-grained
> > topic
> > > create
> > > >> > > access using
> > > >> > > > > > > > > Topic:Create
> > > >> > > > > > > > > for topic patterns. But we still check if
> > user
> > > has
> > > >> > > Cluster:Create
> > > >> > > > > > > > > first. If
> > > >> > > > > > > > > Denied, the deny is ignored and we check
> > > >> Topic:Create. We
> > > >> > > dont
> > > >> > > > > > want
> > > >> > > > > > > > to
> > > >> > > > > > > > > log
> > > >> > > > > > > > > DENY for Cluster:Create at INFO level for
> > this,
> > > since
> > > >> this
> > > >> > > is
> > > >> > > > > > not a
> > > >> > > > > > > > > mandatory ACL for creating topics. We
> > will get
> > > >> appropriate
> > > >> > > logs
> > > >> > > > > > > from
> > > >> > > > > > > > > the
> > > >> > > > > > > > > subsequent Topic:Create in this case.
> > > >> > > > > > > > > 3.3 They are not quite the same. FILTER
> > implies
> > > that
> > > >> user
> > > >> > > > > > actually
> > > >> > > > > > > > > requested access to and performed some
> > > operation on
> > > >> the
> > > >> > > filtered
> > > >> > > > > > > > > resources.
> > > >> > > > > > > > > LIST_AUTHORZED did not result in any
> > actual
> > > access.
> > > >> User
> > > >> > > simply
> > > >> > > > > > > > wanted
> > > >> > > > > > > > > to
> > > >> > > > > > > > > know what they are allowed to access.
> > > >> > > > > > > > > 3.4 Request types are Produce, JoinGroup,
> > > OffsetCommit
> > > >> > > etc. So
> > > >> > > > > > that
> > > >> > > > > > > > is
> > > >> > > > > > > > > different from authorization mode,
> > operation
> > > etc.
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco
> > Durai
> > > <
> > > >> > > > > > bosco@apache.org>
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > Hi Rajini
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks for clarifying. This is very
> > helpful...
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > #1 - On the Ranger side, we should be
> > able to
> > > handle
> > > >> > > multiple
> > > >> > > > > > > > > requests at
> > > >> > > > > > > > > > the same time. I was just not sure how
> > much
> > > >> processing
> > > >> > > overhead
> > > >> > > > > > > > will
> > > >> > > > > > > > > be
> > > >> > > > > > > > > > there on the Broker side to split and
> > then
> > > >> consolidate
> > > >> > > the
> > > >> > > > > > > results.
> > > >> > > > > > > > > If it
> > > >> > > > > > > > > > is negligible, then this is the better
> > way.
> > > It will
> > > >> > > make it
> > > >> > > > > > > future
> > > >> > > > > > > > > proof.
> > > >> > > > > > > > > > #2 - I agree, having a single "start"
> > call
> > > makes it
> > > >> > > cleaner.
> > > >> > > > > > The
> > > >> > > > > > > > > > Authorization implementation will only
> > have
> > > to do
> > > >> the
> > > >> > > > > > > > initialization
> > > >> > > > > > > > > only
> > > >> > > > > > > > > > once.
> > > >> > > > > > > > > > #3.1 - Thanks for the clarification. I
> > think
> > > it
> > > >> makes
> > > >> > > sense to
> > > >> > > > > > > have
> > > >> > > > > > > > > this.
> > > >> > > > > > > > > > The term "Mode" might not be explicit
> > enough.
> > > >> > > Essentially it
> > > >> > > > > > > seems
> > > >> > > > > > > > > you want
> > > >> > > > > > > > > > the Authorizer to know the
> > Intent/Purpose of
> > > the
> > > >> > > authorize call
> > > >> > > > > > > and
> > > >> > > > > > > > > let the
> > > >> > > > > > > > > > Authorizer decide what to log as Audit
> > event.
> > > >> Changing
> > > >> > > the
> > > >> > > > > > > > > class/field name
> > > >> > > > > > > > > > or giving more documentation will do.
> > > >> > > > > > > > > > #3.2 - Regarding the option "OPTIONAL".
> > Are
> > > you
> > > >> thinking
> > > >> > > from
> > > >> > > > > > > > > chaining
> > > >> > > > > > > > > > multiple Authorizer? If so, I am not
> > sure
> > > whether
> > > >> the
> > > >> > > Broker
> > > >> > > > > > > would
> > > >> > > > > > > > > have
> > > >> > > > > > > > > > enough information to make that
> > decision. I
> > > feel the
> > > >> > > Authorizer
> > > >> > > > > > > > will
> > > >> > > > > > > > > be the
> > > >> > > > > > > > > > one who would have that knowledge. E.g.
> > in
> > > Ranger
> > > >> we have
> > > >> > > > > > > explicit
> > > >> > > > > > > > > deny,
> > > >> > > > > > > > > > which means no matter what, the request
> > > should be
> > > >> denied
> > > >> > > for
> > > >> > > > > > the
> > > >> > > > > > > > > user/group
> > > >> > > > > > > > > > or condition. So if you are thinking of
> > > chaining
> > > >> > > Authorizers,
> > > >> > > > > > > then
> > > >> > > > > > > > > > responses should have the third state,
> > e.g.
> > > >> > > "DENIED_FINAL", in
> > > >> > > > > > > > which
> > > >> > > > > > > > > case
> > > >> > > > > > > > > > if there is an Authorization chain, it
> > will
> > > be stop
> > > >> and
> > > >> > > the
> > > >> > > > > > > request
> > > >> > > > > > > > > will be
> > > >> > > > > > > > > > denied and if it is just denied, then
> > you
> > > might fall
> > > >> > > back to
> > > >> > > > > > next
> > > >> > > > > > > > > > authorizer. If we don't have chaining of
> > > >> Authorizing in
> > > >> > > mind,
> > > >> > > > > > > then
> > > >> > > > > > > > we
> > > >> > > > > > > > > > should reconsider OPTIONAL for now. Or
> > > clarify under
> > > >> > > which
> > > >> > > > > > > scenario
> > > >> > > > > > > > > > OPTIONAL will be called.
> > > >> > > > > > > > > > #3.3 Regarding, FILTER v/s
> > LIST_AUTHORIZED,
> > > isn't
> > > >> > > > > > > LIST_AUTHORIZED a
> > > >> > > > > > > > > > special case for "FILTER"?
> > > >> > > > > > > > > > #3.4 KafkaRequestContext. requestType()
> > v/s
> > > Action.
> > > >> > > > > > > > > authorizationMode. I
> > > >> > > > > > > > > > am not sure about the overlap or
> > ambiguity.
> > > >> > > > > > > > > > #4 - Cool. This is good, it will be less
> > > stress on
> > > >> the
> > > >> > > > > > > Authorizer.
> > > >> > > > > > > > > Ranger
> > > >> > > > > > > > > > already supports the "count" concept
> > and also
> > > has
> > > >> > > batching
> > > >> > > > > > > > > capability to
> > > >> > > > > > > > > > aggregate similar requests to reduce the
> > > number of
> > > >> audit
> > > >> > > logs
> > > >> > > > > > to
> > > >> > > > > > > > > write. We
> > > >> > > > > > > > > > should be able to pass this through.
> > > >> > > > > > > > > > #5 - Assuming if the object instance is
> > going
> > > out of
> > > >> > > scope, we
> > > >> > > > > > > > > should be
> > > >> > > > > > > > > > fine. Not a super important ask. Ranger
> > is
> > > already
> > > >> > > catching
> > > >> > > > > > KILL
> > > >> > > > > > > > > signal for
> > > >> > > > > > > > > > clean up.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks again. These are good
> > enhancements.
> > > >> Appreciate
> > > >> > > your
> > > >> > > > > > > efforts
> > > >> > > > > > > > > here.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Bosco
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On 8/8/19, 2:03 AM, "Rajini Sivaram" <
> > > >> > > rajinisivaram@gmail.com
> > > >> > > > > > >
> > > >> > > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Hi Don,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Thanks for reviewing the KIP.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > 1. I had this originally as a single
> > > Action, but
> > > >> > > thought it
> > > >> > > > > > > may
> > > >> > > > > > > > > be
> > > >> > > > > > > > > > useful
> > > >> > > > > > > > > > to support batched authorize calls
> > as
> > > well and
> > > >> keep
> > > >> > > it
> > > >> > > > > > > > > consistent with
> > > >> > > > > > > > > > other methods. Single requests can
> > contain
> > > >> multiple
> > > >> > > topics.
> > > >> > > > > > > For
> > > >> > > > > > > > > > example a
> > > >> > > > > > > > > > produce request can contain records
> > for
> > > several
> > > >> > > partitions
> > > >> > > > > > of
> > > >> > > > > > > > > different
> > > >> > > > > > > > > > topics. Broker could potentially
> > > authorize these
> > > >> > > together.
> > > >> > > > > > > For
> > > >> > > > > > > > > > SimpleAclAuthorizer, batched
> > authorize
> > > methods
> > > >> don't
> > > >> > > > > > provide
> > > >> > > > > > > > any
> > > >> > > > > > > > > > optimisation since lookup is based
> > on
> > > resources
> > > >> > > followed by
> > > >> > > > > > > the
> > > >> > > > > > > > > > matching
> > > >> > > > > > > > > > logic. But some authorizers may
> > manage
> > > ACLs by
> > > >> user
> > > >> > > > > > principal
> > > >> > > > > > > > > rather
> > > >> > > > > > > > > > than
> > > >> > > > > > > > > > resource and may be able to optimize
> > > batched
> > > >> > > requests. I am
> > > >> > > > > > > ok
> > > >> > > > > > > > > with
> > > >> > > > > > > > > > using
> > > >> > > > > > > > > > single Action if this is likely to
> > cause
> > > issues.
> > > >> > > > > > > > > > 2. If you have two listeners, one
> > for
> > > >> inter-broker
> > > >> > > traffic
> > > >> > > > > > > and
> > > >> > > > > > > > > another
> > > >> > > > > > > > > > for
> > > >> > > > > > > > > > external clients, start method is
> > invoked
> > > twice,
> > > >> > > once for
> > > >> > > > > > > each
> > > >> > > > > > > > > > listener. On
> > > >> > > > > > > > > > second thought, that may be
> > confusing and
> > > a
> > > >> single
> > > >> > > start()
> > > >> > > > > > > > > invocation
> > > >> > > > > > > > > > that
> > > >> > > > > > > > > > provides all listener information
> > and
> > > returns
> > > >> > > multiple
> > > >> > > > > > > futures
> > > >> > > > > > > > > would be
> > > >> > > > > > > > > > better. Will update the KIP.
> > > >> > > > > > > > > > 3. A typical example is a consumer
> > > subscribing
> > > >> to a
> > > >> > > regex
> > > >> > > > > > > > > pattern. We
> > > >> > > > > > > > > > request all topic metadata from the
> > > broker in
> > > >> order
> > > >> > > to
> > > >> > > > > > decide
> > > >> > > > > > > > > whether
> > > >> > > > > > > > > > the
> > > >> > > > > > > > > > pattern matches, expecting to
> > receive a
> > > list of
> > > >> > > authorised
> > > >> > > > > > > > > topics. The
> > > >> > > > > > > > > > user
> > > >> > > > > > > > > > is not asking to subscribe to an
> > > unauthorized
> > > >> topic.
> > > >> > > If
> > > >> > > > > > there
> > > >> > > > > > > > > are 10000
> > > >> > > > > > > > > > topics in the cluster and the user
> > has
> > > access
> > > >> to 100
> > > >> > > of
> > > >> > > > > > them,
> > > >> > > > > > > > at
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > moment
> > > >> > > > > > > > > > we log 9900 DENIED log entries at
> > INFO
> > > level in
> > > >> > > > > > > > > SimpleAclAuthorizer.
> > > >> > > > > > > > > > The
> > > >> > > > > > > > > > proposal is to authorize this
> > request with
> > > >> > > > > > > > > AuthorizationMode.FILTER, so
> > > >> > > > > > > > > > that authorizers can log resources
> > that
> > > are
> > > >> filtered
> > > >> > > out at
> > > >> > > > > > > > > lower level
> > > >> > > > > > > > > > like DEBUG since this is not an
> > attempt to
> > > >> access
> > > >> > > > > > > unauthorized
> > > >> > > > > > > > > > resources.
> > > >> > > > > > > > > > Brokers already handle these
> > differently
> > > since
> > > >> no
> > > >> > > > > > > authorization
> > > >> > > > > > > > > error
> > > >> > > > > > > > > > is
> > > >> > > > > > > > > > returned to the client in these
> > cases.
> > > Providing
> > > >> > > > > > > authorization
> > > >> > > > > > > > > mode to
> > > >> > > > > > > > > > authorizers enables authorizer
> > > implementations
> > > >> to
> > > >> > > generate
> > > >> > > > > > > > > better audit
> > > >> > > > > > > > > > logs.
> > > >> > > > > > > > > > 4. Each request may contain multiple
> > > instances
> > > >> of
> > > >> > > the same
> > > >> > > > > > > > > authorizable
> > > >> > > > > > > > > > resource. For example a produce
> > request
> > > may
> > > >> contain
> > > >> > > records
> > > >> > > > > > > for
> > > >> > > > > > > > > 10
> > > >> > > > > > > > > > partitions of the same topic. At the
> > > moment, we
> > > >> > > invoke
> > > >> > > > > > > > authorize
> > > >> > > > > > > > > > method 10
> > > >> > > > > > > > > > times. The proposal is to invoke it
> > once
> > > with
> > > >> > > count=10. The
> > > >> > > > > > > > > count is
> > > >> > > > > > > > > > provided to authorizer just for
> > audit
> > > logging
> > > >> > > purposes.
> > > >> > > > > > > > > > 5. Authorizer implements Closeable,
> > so you
> > > >> could use
> > > >> > > > > > close()
> > > >> > > > > > > to
> > > >> > > > > > > > > flush
> > > >> > > > > > > > > > audits?
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > On Thu, Aug 8, 2019 at 7:01 AM Don
> > Bosco
> > > Durai <
> > > >> > > > > > > > bosco@apache.org
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > Rajini
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Thanks for putting this together.
> > It is
> > > >> looking
> > > >> > > good. I
> > > >> > > > > > > have
> > > >> > > > > > > > > few
> > > >> > > > > > > > > > > questions...
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > 1. List<AuthorizationResult>
> > > authorize(...,
> > > >> > > List<Action>
> > > >> > > > > > > > > actions).
> > > >> > > > > > > > > > Do you
> > > >> > > > > > > > > > > see a scenario where the broker
> > will
> > > call
> > > >> > > authorize for
> > > >> > > > > > > > > multiple
> > > >> > > > > > > > > > topics at
> > > >> > > > > > > > > > > the same time? I can understand
> > that
> > > during
> > > >> > > > > > > creating/deleting
> > > >> > > > > > > > > ACLS,
> > > >> > > > > > > > > > > multiple permissions for multiple
> > > resources
> > > >> might
> > > >> > > be
> > > >> > > > > > done.
> > > >> > > > > > > > For
> > > >> > > > > > > > > > authorize
> > > >> > > > > > > > > > > call, would this be a case? And
> > does the
> > > >> Authorize
> > > >> > > > > > > > > implementation
> > > >> > > > > > > > > > will be
> > > >> > > > > > > > > > > able to do performance
> > optimization
> > > because of
> > > >> > > this? Or
> > > >> > > > > > > > should
> > > >> > > > > > > > > we
> > > >> > > > > > > > > > just keep
> > > >> > > > > > > > > > > it simple? I don't see it as an
> > issue
> > > from
> > > >> Apache
> > > >> > > Ranger
> > > >> > > > > > > > side,
> > > >> > > > > > > > > but
> > > >> > > > > > > > > > just
> > > >> > > > > > > > > > > checking to see whether we need
> > to be
> > > aware of
> > > >> > > something.
> > > >> > > > > > > > > > > 2. Should I assume that the
> > > SecurityProtocol
> > > >> passed
> > > >> > > > > > during
> > > >> > > > > > > > > start and
> > > >> > > > > > > > > > the
> > > >> > > > > > > > > > > one return by
> > > >> > > KafkaRequestContext.securityProtocol() will
> > > >> > > > > > > be
> > > >> > > > > > > > > the
> > > >> > > > > > > > > > same?
> > > >> > > > > > > > > > > CompletableFuture<Void>
> > start(String
> > > >> listenerName,
> > > >> > > > > > > > > SecurityProtocol
> > > >> > > > > > > > > > > securityProtocol);
> > > >> > > > > > > > > > >
> > KafkaRequestContext.securityProtocol()
> > > >> > > > > > > > > > > 3. What is the purpose of
> > > AuthorizationMode?
> > > >> How
> > > >> > > does the
> > > >> > > > > > > > > broker
> > > >> > > > > > > > > > decide
> > > >> > > > > > > > > > > what mode to use when the
> > authorize()
> > > method
> > > >> is
> > > >> > > called?
> > > >> > > > > > > > > > > 4. Can we clarify "count" in
> > Action a
> > > bit
> > > >> more?
> > > >> > > How is it
> > > >> > > > > > > > used?
> > > >> > > > > > > > > > > 5. Do you feel having "stop"
> > along with
> > > >> "start" be
> > > >> > > > > > helpful?
> > > >> > > > > > > > > E.g. In
> > > >> > > > > > > > > > Ranger
> > > >> > > > > > > > > > > we try to optimize the Audit
> > writing by
> > > >> caching
> > > >> > > the logs
> > > >> > > > > > > for
> > > >> > > > > > > > a
> > > >> > > > > > > > > fixed
> > > >> > > > > > > > > > > interval. But when the Broker
> > > terminates, we
> > > >> do a
> > > >> > > forced
> > > >> > > > > > > > flush.
> > > >> > > > > > > > > > Having an
> > > >> > > > > > > > > > > explicit "stop" might give us a
> > formal
> > > way to
> > > >> > > flush our
> > > >> > > > > > > > audits.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Thanks
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Bosco
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On 8/7/19, 3:59 PM, "Rajini
> > Sivaram" <
> > > >> > > > > > > > rajinisivaram@gmail.com
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Hi Ron/Harsha/Satish,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Thanks for reviewing the KIP!
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > We should perhaps have a wider
> > > discussion
> > > >> > > outside
> > > >> > > > > > this
> > > >> > > > > > > > KIP
> > > >> > > > > > > > > for
> > > >> > > > > > > > > > > refactoring
> > > >> > > > > > > > > > > clients so that others who
> > are not
> > > >> following
> > > >> > > this KIP
> > > >> > > > > > > > also
> > > >> > > > > > > > > > notice the
> > > >> > > > > > > > > > > discussion. Satish, would you
> > like
> > > to
> > > >> start a
> > > >> > > > > > > discussion
> > > >> > > > > > > > > thread
> > > >> > > > > > > > > > on dev?
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Regards,
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Rajini
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > On Wed, Aug 7, 2019 at 6:21 PM
> > > Satish
> > > >> Duggana <
> > > >> > > > > > > > > > > satish.duggana@gmail.com>
> > > >> > > > > > > > > > > wrote:
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > > I felt the same need when
> > we want
> > > to
> > > >> add a
> > > >> > > > > > pluggable
> > > >> > > > > > > > API
> > > >> > > > > > > > > for
> > > >> > > > > > > > > > core
> > > >> > > > > > > > > > > > server functionality. This
> > does
> > > not
> > > >> need to
> > > >> > > be part
> > > >> > > > > > > of
> > > >> > > > > > > > > this
> > > >> > > > > > > > > > KIP, it
> > > >> > > > > > > > > > > > can be a separate KIP. I can
> > > contribute
> > > >> those
> > > >> > > > > > > > refactoring
> > > >> > > > > > > > > > changes if
> > > >> > > > > > > > > > > > others are OK with that.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > It is better to have a
> > structure
> > > like
> > > >> below.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > kafka-common:
> > > >> > > > > > > > > > > > common classes which can be
> > used
> > > in any
> > > >> of
> > > >> > > the
> > > >> > > > > > other
> > > >> > > > > > > > > modules
> > > >> > > > > > > > > > in Kafka
> > > >> > > > > > > > > > > > like client,
> > Kafka-server-common
> > > and
> > > >> server
> > > >> > > etc.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > kafka-client-common:
> > > >> > > > > > > > > > > > common classes which can be
> > used
> > > in the
> > > >> > > client
> > > >> > > > > > > module.
> > > >> > > > > > > > > This
> > > >> > > > > > > > > > can be
> > > >> > > > > > > > > > > > part of client module
> > itself.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > kafka-server-common:
> > > >> > > > > > > > > > > > classes required only for
> > > kafka-server.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Thanks.
> > > >> > > > > > > > > > > > Satish.
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > On Wed, Aug 7, 2019 at 9:28
> > PM
> > > Harsha
> > > >> > > Chintalapani
> > > >> > > > > > <
> > > >> > > > > > > > > > kafka@harsha.io>
> > > >> > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Thanks for the KIP Rajini.
> > > >> > > > > > > > > > > > > Quick thought, it would
> > be good
> > > to
> > > >> have a
> > > >> > > common
> > > >> > > > > > > > module
> > > >> > > > > > > > > > outside of
> > > >> > > > > > > > > > > > clients
> > > >> > > > > > > > > > > > > that only applies to
> > server side
> > > >> > > interfaces &
> > > >> > > > > > > > changes.
> > > >> > > > > > > > > It
> > > >> > > > > > > > > > looks
> > > >> > > > > > > > > > > like we
> > > >> > > > > > > > > > > > are
> > > >> > > > > > > > > > > > > increasingly in favor of
> > using
> > > Java
> > > >> > > interface for
> > > >> > > > > > > > > pluggable
> > > >> > > > > > > > > > > modules on
> > > >> > > > > > > > > > > > the
> > > >> > > > > > > > > > > > > broker side.
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > Thanks,
> > > >> > > > > > > > > > > > > Harsha
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > On Tue, Aug 06, 2019 at
> > 2:31 PM,
> > > >> Rajini
> > > >> > > Sivaram <
> > > >> > > > > > > > > > > rajinisivaram@gmail.com
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Hi all,
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > I have created a KIP to
> > > replace the
> > > >> Scala
> > > >> > > > > > > > Authorizer
> > > >> > > > > > > > > API
> > > >> > > > > > > > > > with a
> > > >> > > > > > > > > > > new
> > > >> > > > > > > > > > > > Java
> > > >> > > > > > > > > > > > > > API:
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > -
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > > >> > > > > > > > > > > > > >
> > > >> > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > This is replacement for
> > KIP-50
> > > >> which was
> > > >> > > > > > accepted
> > > >> > > > > > > > > but never
> > > >> > > > > > > > > > > merged.
> > > >> > > > > > > > > > > > Apart
> > > >> > > > > > > > > > > > > > from moving to a Java
> > API
> > > consistent
> > > >> > > with other
> > > >> > > > > > > > > pluggable
> > > >> > > > > > > > > > > interfaces
> > > >> > > > > > > > > > > > in the
> > > >> > > > > > > > > > > > > > broker, KIP-504 also
> > attempts
> > > to
> > > >> address
> > > >> > > known
> > > >> > > > > > > > > limitations
> > > >> > > > > > > > > > in the
> > > >> > > > > > > > > > > > > > authorizer. If you have
> > come
> > > across
> > > >> other
> > > >> > > > > > > > > limitations that
> > > >> > > > > > > > > > you
> > > >> > > > > > > > > > > would
> > > >> > > > > > > > > > > > like
> > > >> > > > > > > > > > > > > > to see addressed in the
> > new
> > > API,
> > > >> please
> > > >> > > raise
> > > >> > > > > > > these
> > > >> > > > > > > > > on the
> > > >> > > > > > > > > > > discussion
> > > >> > > > > > > > > > > > > > thread so that we can
> > > consider those
> > > >> > > too. All
> > > >> > > > > > > > > suggestions
> > > >> > > > > > > > > > and
> > > >> > > > > > > > > > > feedback
> > > >> > > > > > > > > > > > are
> > > >> > > > > > > > > > > > > > welcome.
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Thank you,
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > > > > Rajini
> > > >> > > > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi all,
Ismael brought up a point that it will be good to make the Authorizer
interface asynchronous to avoid blocking request threads during remote
operations.
1) Since we want to support different backends for authorization metadata,
making createAcls() and deleteAcls() asynchronous makes sense since these
always involve remote operations. When KIP-500 removes ZooKeeper, we would
want to move ACLs to Kafka and async updates will avoid unnecessary
blocking.
2) For authorize() method, we currently use cached ACLs in the built-in
authorizers, so synchronous authorise operations work well now. But async
authorize() would support this scenario as well as authorizers in large
organisations where an LRU cache would enable a smaller cache even when the
backend holds a large amount of ACLs for infrequently used resources or
users who don't use the system frequently.
For both cases, the built-in authorizer will continue to be synchronous,
using CompletableFuture.completedFuture() to return the actual result. But
async API will make custom authorizer implementations more flexible. I
would like to know if there are any concerns with these changes before
updating the KIP.
*Proposed API:*
public interface Authorizer extends Configurable, Closeable {
Map<Endpoint, CompletionStage<Void>> start(AuthorizerServerInfo serverInfo);
List<CompletionStage<AuthorizationResult>>
authorize(AuthorizableRequestContext requestContext, List<Action>
actions);
List<CompletionStage<AclCreateResult>>
createAcls(AuthorizableRequestContext requestContext, List<AclBinding>
aclBindings);
List<CompletionStage<AclDeleteResult>>
deleteAcls(AuthorizableRequestContext requestContext,
List<AclBindingFilter> aclBindingFilters);
CompletionStage<Collection<AclBinding>> acls(AclBindingFilter filter);
}
Thank you,
Rajini
On Sun, Aug 18, 2019 at 6:25 PM Don Bosco Durai <bo...@apache.org> wrote:
> Hi Rajini
>
> Thanks for clarifying. I am good for now.
>
> Regards
>
> Bosco
>
>
> On 8/16/19, 11:30 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
>
> Hi Don,
>
> That should be fine. I guess Ranger loads policies from the database
> synchronously when the authorizer is configured, similar to
> SimpleAclAuthorizer loading from ZooKeeper. Ranger can continue to load
> synchronously from `configure()` or `start()` and return an empty map
> from
> `start()`. That would retain the existing behaviour.. When the same
> database stores policies for all listeners and the policies are not
> stored
> in Kafka, there is no value in making the load asynchronous.
>
> Regards,
>
> Rajini
>
>
> On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai <bo...@apache.org>
> wrote:
>
> > Hi Rajini
> >
> > Assuming this doesn't affect custom plugins, I don't have any
> concerns
> > regarding this.
> >
> > I do have one question regarding:
> >
> > "For authorizers that don’t store metadata in ZooKeeper, ensure that
> > authorizer metadata for each listener is available before starting
> up the
> > listener. This enables different authorization metadata stores for
> > different listeners."
> >
> > Since Ranger uses its own database for storing policies, do you
> anticipate
> > any issues?
> >
> > Thanks
> >
> > Bosco
> >
> >
> > On 8/16/19, 6:49 AM, "Rajini Sivaram" <ra...@gmail.com>
> wrote:
> >
> > Hi all,
> >
> > I made another change to the KIP. The KIP was originally
> proposing to
> > extend SimpleAclAuthorizer to also implement the new API (in
> addition
> > to
> > the existing API). But since we use the new API when available,
> this
> > can
> > break custom authorizers that extend this class and override
> specific
> > methods of the old API. To avoid breaking any existing custom
> > implementations that extend this class, particularly because it
> is in
> > the
> > public package kafka.security.auth, the KIP now proposes to
> retain the
> > old
> > authorizer as-is. SimpleAclAuthorizer will continue to
> implement the
> > old
> > API, but will be deprecated. A new authorizer implementation
> > kafka.security.authorizer.AclAuthorizer will be added for the
> new API
> > (this
> > will not be in the public package).
> >
> > Please let me know if you have any concerns.
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
> > rajinisivaram@gmail.com>
> > wrote:
> >
> > > Thanks Colin.
> > >
> > > If there are no other concerns, I will start vote later today.
> Many
> > thanks
> > > to every one for the feedback.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <
> cmccabe@apache.org>
> > wrote:
> > >
> > >> Thanks, Rajini. It looks good to me.
> > >>
> > >> best,
> > >> Colin
> > >>
> > >>
> > >> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram wrote:
> > >> > Hi Colin,
> > >> >
> > >> > Thanks for the review. I have updated the KIP to move the
> > interfaces for
> > >> > request context and server info to the authorizer package.
> These
> > are now
> > >> > called AuthorizableRequestContext and AuthorizerServerInfo.
> > Endpoint is
> > >> now
> > >> > a class in org.apache.kafka.common to make it reusable
> since we
> > already
> > >> > have multiple implementations of it. I have removed
> requestName
> > from the
> > >> > request context interface since authorizers can distinguish
> > follower
> > >> fetch
> > >> > and consumer fetch from the operation being authorized. So
> 16-bit
> > >> request
> > >> > type should be sufficient for audit logging. Also replaced
> > AuditFlag
> > >> with
> > >> > two booleans as you suggested.
> > >> >
> > >> > Can you take another look and see if the KIP is ready for
> voting?
> > >> >
> > >> > Thanks for all your help!
> > >> >
> > >> > Regards,
> > >> >
> > >> > Rajini
> > >> >
> > >> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <
> cmccabe@apache.org>
> > >> wrote:
> > >> >
> > >> > > Hi Rajini,
> > >> > >
> > >> > > I think it would be good to rename KafkaRequestContext to
> > something
> > >> like
> > >> > > AuthorizableRequestContext, and put it in the
> > >> > > org.apache.kafka.server.authorizer namespace. If we put
> it in
> > the
> > >> > > org.apache.kafka.common namespace, then it's not really
> clear
> > that
> > >> it's
> > >> > > part of the Authorizer API. Since this class is purely an
> > interface,
> > >> with
> > >> > > no concrete implementation of anything, there's nothing
> common
> > to
> > >> really
> > >> > > reuse in any case. We definitely don't want someone to
> > accidentally
> > >> add or
> > >> > > remove methods because they think this is just another
> internal
> > class
> > >> used
> > >> > > for requests.
> > >> > >
> > >> > > The BrokerInfo class is a nice improvement. It looks
> like it
> > will be
> > >> > > useful for passing in information about the context we're
> > running
> > >> in. It
> > >> > > would be nice to call this class ServerInfo rather than
> > BrokerInfo,
> > >> since
> > >> > > we will want to run the authorizer on controllers as well
> as on
> > >> brokers,
> > >> > > and the controller may run as a separate process post
> KIP-500.
> > I also
> > >> > > think that this class should be in the
> > >> org.apache.kafka.server.authorizer
> > >> > > namespace. Again, it is an interface, not a concrete
> > implementation,
> > >> and
> > >> > > it's an interface that is very specifically for the
> authorizer.
> > >> > >
> > >> > > I agree that we probably don't have enough information
> > preserved for
> > >> > > requests currently to always know what entity made them.
> So we
> > can
> > >> leave
> > >> > > that out for now (except in the special case of Fetch).
> > Perhaps we
> > >> can add
> > >> > > this later if it's needed.
> > >> > >
> > >> > > I understand the intention behind AuthorizationMode
> (which is
> > now
> > >> called
> > >> > > AuditFlag in the latest revision). But it still feels
> > complex. What
> > >> if we
> > >> > > just had two booleans in Action: logSuccesses and
> logFailures?
> > That
> > >> seems
> > >> > > to cover all the cases here. MANDATORY_AUTHORIZE = true,
> true.
> > >> > > OPTIONAL_AUTHORIZE = true, false. FILTER = true, false.
> > >> LIST_AUTHORIZED =
> > >> > > false, false. Would there be anything lost versus having
> the
> > enum?
> > >> > >
> > >> > > best,
> > >> > > Colin
> > >> > >
> > >> > >
> > >> > > On Wed, Aug 14, 2019, at 06:29, Mickael Maison wrote:
> > >> > > > Hi Rajini,
> > >> > > >
> > >> > > > Thanks for the KIP!
> > >> > > > I really like that authorize() will be able to take a
> batch of
> > >> > > > requests, this will speed up many implementations!
> > >> > > >
> > >> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <
> > >> rajinisivaram@gmail.com>
> > >> > > wrote:
> > >> > > > >
> > >> > > > > Thanks David! I have fixed the typo.
> > >> > > > >
> > >> > > > > Also made a couple of changes to make the context
> > interfaces more
> > >> > > generic.
> > >> > > > > KafkaRequestContext now returns the 16-bit API key as
> Colin
> > >> suggested
> > >> > > as
> > >> > > > > well as the friendly name used in metrics which are
> useful
> > in
> > >> audit
> > >> > > logs.
> > >> > > > > `Authorizer#start` is now provided a generic
> `BrokerInfo`
> > >> interface
> > >> > > that
> > >> > > > > gives cluster id, broker id and endpoint information.
> The
> > generic
> > >> > > interface
> > >> > > > > can potentially be used in other broker plugins in
> future
> > and
> > >> provides
> > >> > > > > dynamically generated configs like broker id and ports
> > which are
> > >> > > currently
> > >> > > > > not available to plugins unless these configs are
> statically
> > >> > > configured.
> > >> > > > > Please let me know if there are any concerns.
> > >> > > > >
> > >> > > > > Regards,
> > >> > > > >
> > >> > > > > Rajini
> > >> > > > >
> > >> > > > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot <
> > djacot@confluent.io>
> > >> > > wrote:
> > >> > > > >
> > >> > > > > > Hi Rajini,
> > >> > > > > >
> > >> > > > > > Thank you for the update! It looks good to me.
> There is a
> > typo
> > >> in the
> > >> > > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` ->
> > >> `MANDATORY_AUTHORIZE`.
> > >> > > > > >
> > >> > > > > > Regards,
> > >> > > > > > David
> > >> > > > > >
> > >> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram <
> > >> > > rajinisivaram@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi David,
> > >> > > > > > >
> > >> > > > > > > Thanks for reviewing the KIP! Since questions
> about
> > >> `authorization
> > >> > > mode`
> > >> > > > > > > and `count` have come up multiple times, I have
> renamed
> > both.
> > >> > > > > > >
> > >> > > > > > > 1) Renamed `count` to `resourceReferenceCount`.
> It is
> > the
> > >> number
> > >> > > of times
> > >> > > > > > > the resource being authorized is referenced
> within the
> > >> request.
> > >> > > > > > >
> > >> > > > > > > 2) Renamed `AuthorizationMode` to `AuditFlag`. It
> is
> > provided
> > >> to
> > >> > > improve
> > >> > > > > > > audit logging in the authorizer. The enum values
> have
> > javadoc
> > >> which
> > >> > > > > > > indicate how the authorization result is used in
> each
> > of the
> > >> modes
> > >> > > to
> > >> > > > > > > enable authorizers to log audit messages at the
> right
> > severity
> > >> > > level.
> > >> > > > > > >
> > >> > > > > > > Regards,
> > >> > > > > > >
> > >> > > > > > > Rajini
> > >> > > > > > >
> > >> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM David Jacot <
> > >> djacot@confluent.io>
> > >> > > wrote:
> > >> > > > > > >
> > >> > > > > > > > Hi Rajini,
> > >> > > > > > > >
> > >> > > > > > > > Thank you for the KIP. Overall, it looks good
> to me.
> > I have
> > >> few
> > >> > > > > > > > questions/suggestions:
> > >> > > > > > > >
> > >> > > > > > > > 1. It is hard to grasp what `Action#count` is
> for. I
> > guess I
> > >> > > understand
> > >> > > > > > > > where you want to go with it but it took me a
> while to
> > >> figure it
> > >> > > out.
> > >> > > > > > > > Perhaps, we could come up with a better name
> than
> > `count`?
> > >> > > > > > > >
> > >> > > > > > > > 2. I had a hard time trying to understand the
> > >> `AuthorizationMode`
> > >> > > > > > > concept,
> > >> > > > > > > > especially wrt. the OPTIONAL one. My
> understanding is
> > that
> > >> an
> > >> > > ACL is
> > >> > > > > > > either
> > >> > > > > > > > defined or not. Could you elaborate a bit more
> on
> > that?
> > >> > > > > > > >
> > >> > > > > > > > Thanks,
> > >> > > > > > > > David
> > >> > > > > > > >
> > >> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai
> <
> > >> > > bosco@apache.org>
> > >> > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Hi Rajini
> > >> > > > > > > > >
> > >> > > > > > > > > 3.2 - This makes sense. Thanks for clarifying.
> > >> > > > > > > > >
> > >> > > > > > > > > Rest looks fine. Once the implementations are
> done,
> > it
> > >> will be
> > >> > > more
> > >> > > > > > > clear
> > >> > > > > > > > > on the different values RequestType and Mode.
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks
> > >> > > > > > > > >
> > >> > > > > > > > > Bosco
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <
> > >> rajinisivaram@gmail.com
> > >> > > >
> > >> > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > Hi Don,
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks for the suggestions. A few
> responses
> > below:
> > >> > > > > > > > >
> > >> > > > > > > > > 3.1 Can rename and improve docs if we keep
> > this. Let's
> > >> > > finish the
> > >> > > > > > > > > discussion on Colin's suggestions
> regarding this
> > >> first.
> > >> > > > > > > > > 3.2 No, I was thinking of some requests
> that
> > have an
> > >> old
> > >> > > way of
> > >> > > > > > > > > authorizing
> > >> > > > > > > > > and a new way where we have retained the
> old
> > way for
> > >> > > backward
> > >> > > > > > > > > compatibility. One example is
> Cluster:Create
> > >> permission to
> > >> > > create
> > >> > > > > > > > > topics.
> > >> > > > > > > > > We have replaced this with fine-grained
> topic
> > create
> > >> > > access using
> > >> > > > > > > > > Topic:Create
> > >> > > > > > > > > for topic patterns. But we still check if
> user
> > has
> > >> > > Cluster:Create
> > >> > > > > > > > > first. If
> > >> > > > > > > > > Denied, the deny is ignored and we check
> > >> Topic:Create. We
> > >> > > dont
> > >> > > > > > want
> > >> > > > > > > > to
> > >> > > > > > > > > log
> > >> > > > > > > > > DENY for Cluster:Create at INFO level for
> this,
> > since
> > >> this
> > >> > > is
> > >> > > > > > not a
> > >> > > > > > > > > mandatory ACL for creating topics. We
> will get
> > >> appropriate
> > >> > > logs
> > >> > > > > > > from
> > >> > > > > > > > > the
> > >> > > > > > > > > subsequent Topic:Create in this case.
> > >> > > > > > > > > 3.3 They are not quite the same. FILTER
> implies
> > that
> > >> user
> > >> > > > > > actually
> > >> > > > > > > > > requested access to and performed some
> > operation on
> > >> the
> > >> > > filtered
> > >> > > > > > > > > resources.
> > >> > > > > > > > > LIST_AUTHORZED did not result in any
> actual
> > access.
> > >> User
> > >> > > simply
> > >> > > > > > > > wanted
> > >> > > > > > > > > to
> > >> > > > > > > > > know what they are allowed to access.
> > >> > > > > > > > > 3.4 Request types are Produce, JoinGroup,
> > OffsetCommit
> > >> > > etc. So
> > >> > > > > > that
> > >> > > > > > > > is
> > >> > > > > > > > > different from authorization mode,
> operation
> > etc.
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco
> Durai
> > <
> > >> > > > > > bosco@apache.org>
> > >> > > > > > > > > wrote:
> > >> > > > > > > > >
> > >> > > > > > > > > > Hi Rajini
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks for clarifying. This is very
> helpful...
> > >> > > > > > > > > >
> > >> > > > > > > > > > #1 - On the Ranger side, we should be
> able to
> > handle
> > >> > > multiple
> > >> > > > > > > > > requests at
> > >> > > > > > > > > > the same time. I was just not sure how
> much
> > >> processing
> > >> > > overhead
> > >> > > > > > > > will
> > >> > > > > > > > > be
> > >> > > > > > > > > > there on the Broker side to split and
> then
> > >> consolidate
> > >> > > the
> > >> > > > > > > results.
> > >> > > > > > > > > If it
> > >> > > > > > > > > > is negligible, then this is the better
> way.
> > It will
> > >> > > make it
> > >> > > > > > > future
> > >> > > > > > > > > proof.
> > >> > > > > > > > > > #2 - I agree, having a single "start"
> call
> > makes it
> > >> > > cleaner.
> > >> > > > > > The
> > >> > > > > > > > > > Authorization implementation will only
> have
> > to do
> > >> the
> > >> > > > > > > > initialization
> > >> > > > > > > > > only
> > >> > > > > > > > > > once.
> > >> > > > > > > > > > #3.1 - Thanks for the clarification. I
> think
> > it
> > >> makes
> > >> > > sense to
> > >> > > > > > > have
> > >> > > > > > > > > this.
> > >> > > > > > > > > > The term "Mode" might not be explicit
> enough.
> > >> > > Essentially it
> > >> > > > > > > seems
> > >> > > > > > > > > you want
> > >> > > > > > > > > > the Authorizer to know the
> Intent/Purpose of
> > the
> > >> > > authorize call
> > >> > > > > > > and
> > >> > > > > > > > > let the
> > >> > > > > > > > > > Authorizer decide what to log as Audit
> event.
> > >> Changing
> > >> > > the
> > >> > > > > > > > > class/field name
> > >> > > > > > > > > > or giving more documentation will do.
> > >> > > > > > > > > > #3.2 - Regarding the option "OPTIONAL".
> Are
> > you
> > >> thinking
> > >> > > from
> > >> > > > > > > > > chaining
> > >> > > > > > > > > > multiple Authorizer? If so, I am not
> sure
> > whether
> > >> the
> > >> > > Broker
> > >> > > > > > > would
> > >> > > > > > > > > have
> > >> > > > > > > > > > enough information to make that
> decision. I
> > feel the
> > >> > > Authorizer
> > >> > > > > > > > will
> > >> > > > > > > > > be the
> > >> > > > > > > > > > one who would have that knowledge. E.g.
> in
> > Ranger
> > >> we have
> > >> > > > > > > explicit
> > >> > > > > > > > > deny,
> > >> > > > > > > > > > which means no matter what, the request
> > should be
> > >> denied
> > >> > > for
> > >> > > > > > the
> > >> > > > > > > > > user/group
> > >> > > > > > > > > > or condition. So if you are thinking of
> > chaining
> > >> > > Authorizers,
> > >> > > > > > > then
> > >> > > > > > > > > > responses should have the third state,
> e.g.
> > >> > > "DENIED_FINAL", in
> > >> > > > > > > > which
> > >> > > > > > > > > case
> > >> > > > > > > > > > if there is an Authorization chain, it
> will
> > be stop
> > >> and
> > >> > > the
> > >> > > > > > > request
> > >> > > > > > > > > will be
> > >> > > > > > > > > > denied and if it is just denied, then
> you
> > might fall
> > >> > > back to
> > >> > > > > > next
> > >> > > > > > > > > > authorizer. If we don't have chaining of
> > >> Authorizing in
> > >> > > mind,
> > >> > > > > > > then
> > >> > > > > > > > we
> > >> > > > > > > > > > should reconsider OPTIONAL for now. Or
> > clarify under
> > >> > > which
> > >> > > > > > > scenario
> > >> > > > > > > > > > OPTIONAL will be called.
> > >> > > > > > > > > > #3.3 Regarding, FILTER v/s
> LIST_AUTHORIZED,
> > isn't
> > >> > > > > > > LIST_AUTHORIZED a
> > >> > > > > > > > > > special case for "FILTER"?
> > >> > > > > > > > > > #3.4 KafkaRequestContext. requestType()
> v/s
> > Action.
> > >> > > > > > > > > authorizationMode. I
> > >> > > > > > > > > > am not sure about the overlap or
> ambiguity.
> > >> > > > > > > > > > #4 - Cool. This is good, it will be less
> > stress on
> > >> the
> > >> > > > > > > Authorizer.
> > >> > > > > > > > > Ranger
> > >> > > > > > > > > > already supports the "count" concept
> and also
> > has
> > >> > > batching
> > >> > > > > > > > > capability to
> > >> > > > > > > > > > aggregate similar requests to reduce the
> > number of
> > >> audit
> > >> > > logs
> > >> > > > > > to
> > >> > > > > > > > > write. We
> > >> > > > > > > > > > should be able to pass this through.
> > >> > > > > > > > > > #5 - Assuming if the object instance is
> going
> > out of
> > >> > > scope, we
> > >> > > > > > > > > should be
> > >> > > > > > > > > > fine. Not a super important ask. Ranger
> is
> > already
> > >> > > catching
> > >> > > > > > KILL
> > >> > > > > > > > > signal for
> > >> > > > > > > > > > clean up.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks again. These are good
> enhancements.
> > >> Appreciate
> > >> > > your
> > >> > > > > > > efforts
> > >> > > > > > > > > here.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Bosco
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > > On 8/8/19, 2:03 AM, "Rajini Sivaram" <
> > >> > > rajinisivaram@gmail.com
> > >> > > > > > >
> > >> > > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > Hi Don,
> > >> > > > > > > > > >
> > >> > > > > > > > > > Thanks for reviewing the KIP.
> > >> > > > > > > > > >
> > >> > > > > > > > > > 1. I had this originally as a single
> > Action, but
> > >> > > thought it
> > >> > > > > > > may
> > >> > > > > > > > > be
> > >> > > > > > > > > > useful
> > >> > > > > > > > > > to support batched authorize calls
> as
> > well and
> > >> keep
> > >> > > it
> > >> > > > > > > > > consistent with
> > >> > > > > > > > > > other methods. Single requests can
> contain
> > >> multiple
> > >> > > topics.
> > >> > > > > > > For
> > >> > > > > > > > > > example a
> > >> > > > > > > > > > produce request can contain records
> for
> > several
> > >> > > partitions
> > >> > > > > > of
> > >> > > > > > > > > different
> > >> > > > > > > > > > topics. Broker could potentially
> > authorize these
> > >> > > together.
> > >> > > > > > > For
> > >> > > > > > > > > > SimpleAclAuthorizer, batched
> authorize
> > methods
> > >> don't
> > >> > > > > > provide
> > >> > > > > > > > any
> > >> > > > > > > > > > optimisation since lookup is based
> on
> > resources
> > >> > > followed by
> > >> > > > > > > the
> > >> > > > > > > > > > matching
> > >> > > > > > > > > > logic. But some authorizers may
> manage
> > ACLs by
> > >> user
> > >> > > > > > principal
> > >> > > > > > > > > rather
> > >> > > > > > > > > > than
> > >> > > > > > > > > > resource and may be able to optimize
> > batched
> > >> > > requests. I am
> > >> > > > > > > ok
> > >> > > > > > > > > with
> > >> > > > > > > > > > using
> > >> > > > > > > > > > single Action if this is likely to
> cause
> > issues.
> > >> > > > > > > > > > 2. If you have two listeners, one
> for
> > >> inter-broker
> > >> > > traffic
> > >> > > > > > > and
> > >> > > > > > > > > another
> > >> > > > > > > > > > for
> > >> > > > > > > > > > external clients, start method is
> invoked
> > twice,
> > >> > > once for
> > >> > > > > > > each
> > >> > > > > > > > > > listener. On
> > >> > > > > > > > > > second thought, that may be
> confusing and
> > a
> > >> single
> > >> > > start()
> > >> > > > > > > > > invocation
> > >> > > > > > > > > > that
> > >> > > > > > > > > > provides all listener information
> and
> > returns
> > >> > > multiple
> > >> > > > > > > futures
> > >> > > > > > > > > would be
> > >> > > > > > > > > > better. Will update the KIP.
> > >> > > > > > > > > > 3. A typical example is a consumer
> > subscribing
> > >> to a
> > >> > > regex
> > >> > > > > > > > > pattern. We
> > >> > > > > > > > > > request all topic metadata from the
> > broker in
> > >> order
> > >> > > to
> > >> > > > > > decide
> > >> > > > > > > > > whether
> > >> > > > > > > > > > the
> > >> > > > > > > > > > pattern matches, expecting to
> receive a
> > list of
> > >> > > authorised
> > >> > > > > > > > > topics. The
> > >> > > > > > > > > > user
> > >> > > > > > > > > > is not asking to subscribe to an
> > unauthorized
> > >> topic.
> > >> > > If
> > >> > > > > > there
> > >> > > > > > > > > are 10000
> > >> > > > > > > > > > topics in the cluster and the user
> has
> > access
> > >> to 100
> > >> > > of
> > >> > > > > > them,
> > >> > > > > > > > at
> > >> > > > > > > > > the
> > >> > > > > > > > > > moment
> > >> > > > > > > > > > we log 9900 DENIED log entries at
> INFO
> > level in
> > >> > > > > > > > > SimpleAclAuthorizer.
> > >> > > > > > > > > > The
> > >> > > > > > > > > > proposal is to authorize this
> request with
> > >> > > > > > > > > AuthorizationMode.FILTER, so
> > >> > > > > > > > > > that authorizers can log resources
> that
> > are
> > >> filtered
> > >> > > out at
> > >> > > > > > > > > lower level
> > >> > > > > > > > > > like DEBUG since this is not an
> attempt to
> > >> access
> > >> > > > > > > unauthorized
> > >> > > > > > > > > > resources.
> > >> > > > > > > > > > Brokers already handle these
> differently
> > since
> > >> no
> > >> > > > > > > authorization
> > >> > > > > > > > > error
> > >> > > > > > > > > > is
> > >> > > > > > > > > > returned to the client in these
> cases.
> > Providing
> > >> > > > > > > authorization
> > >> > > > > > > > > mode to
> > >> > > > > > > > > > authorizers enables authorizer
> > implementations
> > >> to
> > >> > > generate
> > >> > > > > > > > > better audit
> > >> > > > > > > > > > logs.
> > >> > > > > > > > > > 4. Each request may contain multiple
> > instances
> > >> of
> > >> > > the same
> > >> > > > > > > > > authorizable
> > >> > > > > > > > > > resource. For example a produce
> request
> > may
> > >> contain
> > >> > > records
> > >> > > > > > > for
> > >> > > > > > > > > 10
> > >> > > > > > > > > > partitions of the same topic. At the
> > moment, we
> > >> > > invoke
> > >> > > > > > > > authorize
> > >> > > > > > > > > > method 10
> > >> > > > > > > > > > times. The proposal is to invoke it
> once
> > with
> > >> > > count=10. The
> > >> > > > > > > > > count is
> > >> > > > > > > > > > provided to authorizer just for
> audit
> > logging
> > >> > > purposes.
> > >> > > > > > > > > > 5. Authorizer implements Closeable,
> so you
> > >> could use
> > >> > > > > > close()
> > >> > > > > > > to
> > >> > > > > > > > > flush
> > >> > > > > > > > > > audits?
> > >> > > > > > > > > >
> > >> > > > > > > > > > On Thu, Aug 8, 2019 at 7:01 AM Don
> Bosco
> > Durai <
> > >> > > > > > > > bosco@apache.org
> > >> > > > > > > > > >
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > Rajini
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Thanks for putting this together.
> It is
> > >> looking
> > >> > > good. I
> > >> > > > > > > have
> > >> > > > > > > > > few
> > >> > > > > > > > > > > questions...
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > 1. List<AuthorizationResult>
> > authorize(...,
> > >> > > List<Action>
> > >> > > > > > > > > actions).
> > >> > > > > > > > > > Do you
> > >> > > > > > > > > > > see a scenario where the broker
> will
> > call
> > >> > > authorize for
> > >> > > > > > > > > multiple
> > >> > > > > > > > > > topics at
> > >> > > > > > > > > > > the same time? I can understand
> that
> > during
> > >> > > > > > > creating/deleting
> > >> > > > > > > > > ACLS,
> > >> > > > > > > > > > > multiple permissions for multiple
> > resources
> > >> might
> > >> > > be
> > >> > > > > > done.
> > >> > > > > > > > For
> > >> > > > > > > > > > authorize
> > >> > > > > > > > > > > call, would this be a case? And
> does the
> > >> Authorize
> > >> > > > > > > > > implementation
> > >> > > > > > > > > > will be
> > >> > > > > > > > > > > able to do performance
> optimization
> > because of
> > >> > > this? Or
> > >> > > > > > > > should
> > >> > > > > > > > > we
> > >> > > > > > > > > > just keep
> > >> > > > > > > > > > > it simple? I don't see it as an
> issue
> > from
> > >> Apache
> > >> > > Ranger
> > >> > > > > > > > side,
> > >> > > > > > > > > but
> > >> > > > > > > > > > just
> > >> > > > > > > > > > > checking to see whether we need
> to be
> > aware of
> > >> > > something.
> > >> > > > > > > > > > > 2. Should I assume that the
> > SecurityProtocol
> > >> passed
> > >> > > > > > during
> > >> > > > > > > > > start and
> > >> > > > > > > > > > the
> > >> > > > > > > > > > > one return by
> > >> > > KafkaRequestContext.securityProtocol() will
> > >> > > > > > > be
> > >> > > > > > > > > the
> > >> > > > > > > > > > same?
> > >> > > > > > > > > > > CompletableFuture<Void>
> start(String
> > >> listenerName,
> > >> > > > > > > > > SecurityProtocol
> > >> > > > > > > > > > > securityProtocol);
> > >> > > > > > > > > > >
> KafkaRequestContext.securityProtocol()
> > >> > > > > > > > > > > 3. What is the purpose of
> > AuthorizationMode?
> > >> How
> > >> > > does the
> > >> > > > > > > > > broker
> > >> > > > > > > > > > decide
> > >> > > > > > > > > > > what mode to use when the
> authorize()
> > method
> > >> is
> > >> > > called?
> > >> > > > > > > > > > > 4. Can we clarify "count" in
> Action a
> > bit
> > >> more?
> > >> > > How is it
> > >> > > > > > > > used?
> > >> > > > > > > > > > > 5. Do you feel having "stop"
> along with
> > >> "start" be
> > >> > > > > > helpful?
> > >> > > > > > > > > E.g. In
> > >> > > > > > > > > > Ranger
> > >> > > > > > > > > > > we try to optimize the Audit
> writing by
> > >> caching
> > >> > > the logs
> > >> > > > > > > for
> > >> > > > > > > > a
> > >> > > > > > > > > fixed
> > >> > > > > > > > > > > interval. But when the Broker
> > terminates, we
> > >> do a
> > >> > > forced
> > >> > > > > > > > flush.
> > >> > > > > > > > > > Having an
> > >> > > > > > > > > > > explicit "stop" might give us a
> formal
> > way to
> > >> > > flush our
> > >> > > > > > > > audits.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Thanks
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Bosco
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On 8/7/19, 3:59 PM, "Rajini
> Sivaram" <
> > >> > > > > > > > rajinisivaram@gmail.com
> > >> > > > > > > > > >
> > >> > > > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Hi Ron/Harsha/Satish,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Thanks for reviewing the KIP!
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > We should perhaps have a wider
> > discussion
> > >> > > outside
> > >> > > > > > this
> > >> > > > > > > > KIP
> > >> > > > > > > > > for
> > >> > > > > > > > > > > refactoring
> > >> > > > > > > > > > > clients so that others who
> are not
> > >> following
> > >> > > this KIP
> > >> > > > > > > > also
> > >> > > > > > > > > > notice the
> > >> > > > > > > > > > > discussion. Satish, would you
> like
> > to
> > >> start a
> > >> > > > > > > discussion
> > >> > > > > > > > > thread
> > >> > > > > > > > > > on dev?
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Regards,
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Rajini
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > On Wed, Aug 7, 2019 at 6:21 PM
> > Satish
> > >> Duggana <
> > >> > > > > > > > > > > satish.duggana@gmail.com>
> > >> > > > > > > > > > > wrote:
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > > I felt the same need when
> we want
> > to
> > >> add a
> > >> > > > > > pluggable
> > >> > > > > > > > API
> > >> > > > > > > > > for
> > >> > > > > > > > > > core
> > >> > > > > > > > > > > > server functionality. This
> does
> > not
> > >> need to
> > >> > > be part
> > >> > > > > > > of
> > >> > > > > > > > > this
> > >> > > > > > > > > > KIP, it
> > >> > > > > > > > > > > > can be a separate KIP. I can
> > contribute
> > >> those
> > >> > > > > > > > refactoring
> > >> > > > > > > > > > changes if
> > >> > > > > > > > > > > > others are OK with that.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > It is better to have a
> structure
> > like
> > >> below.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > kafka-common:
> > >> > > > > > > > > > > > common classes which can be
> used
> > in any
> > >> of
> > >> > > the
> > >> > > > > > other
> > >> > > > > > > > > modules
> > >> > > > > > > > > > in Kafka
> > >> > > > > > > > > > > > like client,
> Kafka-server-common
> > and
> > >> server
> > >> > > etc.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > kafka-client-common:
> > >> > > > > > > > > > > > common classes which can be
> used
> > in the
> > >> > > client
> > >> > > > > > > module.
> > >> > > > > > > > > This
> > >> > > > > > > > > > can be
> > >> > > > > > > > > > > > part of client module
> itself.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > kafka-server-common:
> > >> > > > > > > > > > > > classes required only for
> > kafka-server.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Thanks.
> > >> > > > > > > > > > > > Satish.
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > On Wed, Aug 7, 2019 at 9:28
> PM
> > Harsha
> > >> > > Chintalapani
> > >> > > > > > <
> > >> > > > > > > > > > kafka@harsha.io>
> > >> > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Thanks for the KIP Rajini.
> > >> > > > > > > > > > > > > Quick thought, it would
> be good
> > to
> > >> have a
> > >> > > common
> > >> > > > > > > > module
> > >> > > > > > > > > > outside of
> > >> > > > > > > > > > > > clients
> > >> > > > > > > > > > > > > that only applies to
> server side
> > >> > > interfaces &
> > >> > > > > > > > changes.
> > >> > > > > > > > > It
> > >> > > > > > > > > > looks
> > >> > > > > > > > > > > like we
> > >> > > > > > > > > > > > are
> > >> > > > > > > > > > > > > increasingly in favor of
> using
> > Java
> > >> > > interface for
> > >> > > > > > > > > pluggable
> > >> > > > > > > > > > > modules on
> > >> > > > > > > > > > > > the
> > >> > > > > > > > > > > > > broker side.
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > Thanks,
> > >> > > > > > > > > > > > > Harsha
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > On Tue, Aug 06, 2019 at
> 2:31 PM,
> > >> Rajini
> > >> > > Sivaram <
> > >> > > > > > > > > > > rajinisivaram@gmail.com
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > wrote:
> > >> > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Hi all,
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > I have created a KIP to
> > replace the
> > >> Scala
> > >> > > > > > > > Authorizer
> > >> > > > > > > > > API
> > >> > > > > > > > > > with a
> > >> > > > > > > > > > > new
> > >> > > > > > > > > > > > Java
> > >> > > > > > > > > > > > > > API:
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > -
> > >> > > > > > > > > > > > > >
> > >> > > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/
> > >> > > > > > > > > > > > > >
> > >> > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > This is replacement for
> KIP-50
> > >> which was
> > >> > > > > > accepted
> > >> > > > > > > > > but never
> > >> > > > > > > > > > > merged.
> > >> > > > > > > > > > > > Apart
> > >> > > > > > > > > > > > > > from moving to a Java
> API
> > consistent
> > >> > > with other
> > >> > > > > > > > > pluggable
> > >> > > > > > > > > > > interfaces
> > >> > > > > > > > > > > > in the
> > >> > > > > > > > > > > > > > broker, KIP-504 also
> attempts
> > to
> > >> address
> > >> > > known
> > >> > > > > > > > > limitations
> > >> > > > > > > > > > in the
> > >> > > > > > > > > > > > > > authorizer. If you have
> come
> > across
> > >> other
> > >> > > > > > > > > limitations that
> > >> > > > > > > > > > you
> > >> > > > > > > > > > > would
> > >> > > > > > > > > > > > like
> > >> > > > > > > > > > > > > > to see addressed in the
> new
> > API,
> > >> please
> > >> > > raise
> > >> > > > > > > these
> > >> > > > > > > > > on the
> > >> > > > > > > > > > > discussion
> > >> > > > > > > > > > > > > > thread so that we can
> > consider those
> > >> > > too. All
> > >> > > > > > > > > suggestions
> > >> > > > > > > > > > and
> > >> > > > > > > > > > > feedback
> > >> > > > > > > > > > > > are
> > >> > > > > > > > > > > > > > welcome.
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Thank you,
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > > > > Rajini
> > >> > > > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
> >
> >
> >
> >
>
>
>
>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Don Bosco Durai <bo...@apache.org>.
Hi Rajini
Thanks for clarifying. I am good for now.
Regards
Bosco
On 8/16/19, 11:30 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
Hi Don,
That should be fine. I guess Ranger loads policies from the database
synchronously when the authorizer is configured, similar to
SimpleAclAuthorizer loading from ZooKeeper. Ranger can continue to load
synchronously from `configure()` or `start()` and return an empty map from
`start()`. That would retain the existing behaviour.. When the same
database stores policies for all listeners and the policies are not stored
in Kafka, there is no value in making the load asynchronous.
Regards,
Rajini
On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai <bo...@apache.org> wrote:
> Hi Rajini
>
> Assuming this doesn't affect custom plugins, I don't have any concerns
> regarding this.
>
> I do have one question regarding:
>
> "For authorizers that don’t store metadata in ZooKeeper, ensure that
> authorizer metadata for each listener is available before starting up the
> listener. This enables different authorization metadata stores for
> different listeners."
>
> Since Ranger uses its own database for storing policies, do you anticipate
> any issues?
>
> Thanks
>
> Bosco
>
>
> On 8/16/19, 6:49 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
>
> Hi all,
>
> I made another change to the KIP. The KIP was originally proposing to
> extend SimpleAclAuthorizer to also implement the new API (in addition
> to
> the existing API). But since we use the new API when available, this
> can
> break custom authorizers that extend this class and override specific
> methods of the old API. To avoid breaking any existing custom
> implementations that extend this class, particularly because it is in
> the
> public package kafka.security.auth, the KIP now proposes to retain the
> old
> authorizer as-is. SimpleAclAuthorizer will continue to implement the
> old
> API, but will be deprecated. A new authorizer implementation
> kafka.security.authorizer.AclAuthorizer will be added for the new API
> (this
> will not be in the public package).
>
> Please let me know if you have any concerns.
>
> Regards,
>
> Rajini
>
>
> On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
> rajinisivaram@gmail.com>
> wrote:
>
> > Thanks Colin.
> >
> > If there are no other concerns, I will start vote later today. Many
> thanks
> > to every one for the feedback.
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <cm...@apache.org>
> wrote:
> >
> >> Thanks, Rajini. It looks good to me.
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram wrote:
> >> > Hi Colin,
> >> >
> >> > Thanks for the review. I have updated the KIP to move the
> interfaces for
> >> > request context and server info to the authorizer package. These
> are now
> >> > called AuthorizableRequestContext and AuthorizerServerInfo.
> Endpoint is
> >> now
> >> > a class in org.apache.kafka.common to make it reusable since we
> already
> >> > have multiple implementations of it. I have removed requestName
> from the
> >> > request context interface since authorizers can distinguish
> follower
> >> fetch
> >> > and consumer fetch from the operation being authorized. So 16-bit
> >> request
> >> > type should be sufficient for audit logging. Also replaced
> AuditFlag
> >> with
> >> > two booleans as you suggested.
> >> >
> >> > Can you take another look and see if the KIP is ready for voting?
> >> >
> >> > Thanks for all your help!
> >> >
> >> > Regards,
> >> >
> >> > Rajini
> >> >
> >> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <cm...@apache.org>
> >> wrote:
> >> >
> >> > > Hi Rajini,
> >> > >
> >> > > I think it would be good to rename KafkaRequestContext to
> something
> >> like
> >> > > AuthorizableRequestContext, and put it in the
> >> > > org.apache.kafka.server.authorizer namespace. If we put it in
> the
> >> > > org.apache.kafka.common namespace, then it's not really clear
> that
> >> it's
> >> > > part of the Authorizer API. Since this class is purely an
> interface,
> >> with
> >> > > no concrete implementation of anything, there's nothing common
> to
> >> really
> >> > > reuse in any case. We definitely don't want someone to
> accidentally
> >> add or
> >> > > remove methods because they think this is just another internal
> class
> >> used
> >> > > for requests.
> >> > >
> >> > > The BrokerInfo class is a nice improvement. It looks like it
> will be
> >> > > useful for passing in information about the context we're
> running
> >> in. It
> >> > > would be nice to call this class ServerInfo rather than
> BrokerInfo,
> >> since
> >> > > we will want to run the authorizer on controllers as well as on
> >> brokers,
> >> > > and the controller may run as a separate process post KIP-500.
> I also
> >> > > think that this class should be in the
> >> org.apache.kafka.server.authorizer
> >> > > namespace. Again, it is an interface, not a concrete
> implementation,
> >> and
> >> > > it's an interface that is very specifically for the authorizer.
> >> > >
> >> > > I agree that we probably don't have enough information
> preserved for
> >> > > requests currently to always know what entity made them. So we
> can
> >> leave
> >> > > that out for now (except in the special case of Fetch).
> Perhaps we
> >> can add
> >> > > this later if it's needed.
> >> > >
> >> > > I understand the intention behind AuthorizationMode (which is
> now
> >> called
> >> > > AuditFlag in the latest revision). But it still feels
> complex. What
> >> if we
> >> > > just had two booleans in Action: logSuccesses and logFailures?
> That
> >> seems
> >> > > to cover all the cases here. MANDATORY_AUTHORIZE = true, true.
> >> > > OPTIONAL_AUTHORIZE = true, false. FILTER = true, false.
> >> LIST_AUTHORIZED =
> >> > > false, false. Would there be anything lost versus having the
> enum?
> >> > >
> >> > > best,
> >> > > Colin
> >> > >
> >> > >
> >> > > On Wed, Aug 14, 2019, at 06:29, Mickael Maison wrote:
> >> > > > Hi Rajini,
> >> > > >
> >> > > > Thanks for the KIP!
> >> > > > I really like that authorize() will be able to take a batch of
> >> > > > requests, this will speed up many implementations!
> >> > > >
> >> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <
> >> rajinisivaram@gmail.com>
> >> > > wrote:
> >> > > > >
> >> > > > > Thanks David! I have fixed the typo.
> >> > > > >
> >> > > > > Also made a couple of changes to make the context
> interfaces more
> >> > > generic.
> >> > > > > KafkaRequestContext now returns the 16-bit API key as Colin
> >> suggested
> >> > > as
> >> > > > > well as the friendly name used in metrics which are useful
> in
> >> audit
> >> > > logs.
> >> > > > > `Authorizer#start` is now provided a generic `BrokerInfo`
> >> interface
> >> > > that
> >> > > > > gives cluster id, broker id and endpoint information. The
> generic
> >> > > interface
> >> > > > > can potentially be used in other broker plugins in future
> and
> >> provides
> >> > > > > dynamically generated configs like broker id and ports
> which are
> >> > > currently
> >> > > > > not available to plugins unless these configs are statically
> >> > > configured.
> >> > > > > Please let me know if there are any concerns.
> >> > > > >
> >> > > > > Regards,
> >> > > > >
> >> > > > > Rajini
> >> > > > >
> >> > > > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot <
> djacot@confluent.io>
> >> > > wrote:
> >> > > > >
> >> > > > > > Hi Rajini,
> >> > > > > >
> >> > > > > > Thank you for the update! It looks good to me. There is a
> typo
> >> in the
> >> > > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` ->
> >> `MANDATORY_AUTHORIZE`.
> >> > > > > >
> >> > > > > > Regards,
> >> > > > > > David
> >> > > > > >
> >> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram <
> >> > > rajinisivaram@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi David,
> >> > > > > > >
> >> > > > > > > Thanks for reviewing the KIP! Since questions about
> >> `authorization
> >> > > mode`
> >> > > > > > > and `count` have come up multiple times, I have renamed
> both.
> >> > > > > > >
> >> > > > > > > 1) Renamed `count` to `resourceReferenceCount`. It is
> the
> >> number
> >> > > of times
> >> > > > > > > the resource being authorized is referenced within the
> >> request.
> >> > > > > > >
> >> > > > > > > 2) Renamed `AuthorizationMode` to `AuditFlag`. It is
> provided
> >> to
> >> > > improve
> >> > > > > > > audit logging in the authorizer. The enum values have
> javadoc
> >> which
> >> > > > > > > indicate how the authorization result is used in each
> of the
> >> modes
> >> > > to
> >> > > > > > > enable authorizers to log audit messages at the right
> severity
> >> > > level.
> >> > > > > > >
> >> > > > > > > Regards,
> >> > > > > > >
> >> > > > > > > Rajini
> >> > > > > > >
> >> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM David Jacot <
> >> djacot@confluent.io>
> >> > > wrote:
> >> > > > > > >
> >> > > > > > > > Hi Rajini,
> >> > > > > > > >
> >> > > > > > > > Thank you for the KIP. Overall, it looks good to me.
> I have
> >> few
> >> > > > > > > > questions/suggestions:
> >> > > > > > > >
> >> > > > > > > > 1. It is hard to grasp what `Action#count` is for. I
> guess I
> >> > > understand
> >> > > > > > > > where you want to go with it but it took me a while to
> >> figure it
> >> > > out.
> >> > > > > > > > Perhaps, we could come up with a better name than
> `count`?
> >> > > > > > > >
> >> > > > > > > > 2. I had a hard time trying to understand the
> >> `AuthorizationMode`
> >> > > > > > > concept,
> >> > > > > > > > especially wrt. the OPTIONAL one. My understanding is
> that
> >> an
> >> > > ACL is
> >> > > > > > > either
> >> > > > > > > > defined or not. Could you elaborate a bit more on
> that?
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > > David
> >> > > > > > > >
> >> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai <
> >> > > bosco@apache.org>
> >> > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi Rajini
> >> > > > > > > > >
> >> > > > > > > > > 3.2 - This makes sense. Thanks for clarifying.
> >> > > > > > > > >
> >> > > > > > > > > Rest looks fine. Once the implementations are done,
> it
> >> will be
> >> > > more
> >> > > > > > > clear
> >> > > > > > > > > on the different values RequestType and Mode.
> >> > > > > > > > >
> >> > > > > > > > > Thanks
> >> > > > > > > > >
> >> > > > > > > > > Bosco
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <
> >> rajinisivaram@gmail.com
> >> > > >
> >> > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > Hi Don,
> >> > > > > > > > >
> >> > > > > > > > > Thanks for the suggestions. A few responses
> below:
> >> > > > > > > > >
> >> > > > > > > > > 3.1 Can rename and improve docs if we keep
> this. Let's
> >> > > finish the
> >> > > > > > > > > discussion on Colin's suggestions regarding this
> >> first.
> >> > > > > > > > > 3.2 No, I was thinking of some requests that
> have an
> >> old
> >> > > way of
> >> > > > > > > > > authorizing
> >> > > > > > > > > and a new way where we have retained the old
> way for
> >> > > backward
> >> > > > > > > > > compatibility. One example is Cluster:Create
> >> permission to
> >> > > create
> >> > > > > > > > > topics.
> >> > > > > > > > > We have replaced this with fine-grained topic
> create
> >> > > access using
> >> > > > > > > > > Topic:Create
> >> > > > > > > > > for topic patterns. But we still check if user
> has
> >> > > Cluster:Create
> >> > > > > > > > > first. If
> >> > > > > > > > > Denied, the deny is ignored and we check
> >> Topic:Create. We
> >> > > dont
> >> > > > > > want
> >> > > > > > > > to
> >> > > > > > > > > log
> >> > > > > > > > > DENY for Cluster:Create at INFO level for this,
> since
> >> this
> >> > > is
> >> > > > > > not a
> >> > > > > > > > > mandatory ACL for creating topics. We will get
> >> appropriate
> >> > > logs
> >> > > > > > > from
> >> > > > > > > > > the
> >> > > > > > > > > subsequent Topic:Create in this case.
> >> > > > > > > > > 3.3 They are not quite the same. FILTER implies
> that
> >> user
> >> > > > > > actually
> >> > > > > > > > > requested access to and performed some
> operation on
> >> the
> >> > > filtered
> >> > > > > > > > > resources.
> >> > > > > > > > > LIST_AUTHORZED did not result in any actual
> access.
> >> User
> >> > > simply
> >> > > > > > > > wanted
> >> > > > > > > > > to
> >> > > > > > > > > know what they are allowed to access.
> >> > > > > > > > > 3.4 Request types are Produce, JoinGroup,
> OffsetCommit
> >> > > etc. So
> >> > > > > > that
> >> > > > > > > > is
> >> > > > > > > > > different from authorization mode, operation
> etc.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai
> <
> >> > > > > > bosco@apache.org>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi Rajini
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks for clarifying. This is very helpful...
> >> > > > > > > > > >
> >> > > > > > > > > > #1 - On the Ranger side, we should be able to
> handle
> >> > > multiple
> >> > > > > > > > > requests at
> >> > > > > > > > > > the same time. I was just not sure how much
> >> processing
> >> > > overhead
> >> > > > > > > > will
> >> > > > > > > > > be
> >> > > > > > > > > > there on the Broker side to split and then
> >> consolidate
> >> > > the
> >> > > > > > > results.
> >> > > > > > > > > If it
> >> > > > > > > > > > is negligible, then this is the better way.
> It will
> >> > > make it
> >> > > > > > > future
> >> > > > > > > > > proof.
> >> > > > > > > > > > #2 - I agree, having a single "start" call
> makes it
> >> > > cleaner.
> >> > > > > > The
> >> > > > > > > > > > Authorization implementation will only have
> to do
> >> the
> >> > > > > > > > initialization
> >> > > > > > > > > only
> >> > > > > > > > > > once.
> >> > > > > > > > > > #3.1 - Thanks for the clarification. I think
> it
> >> makes
> >> > > sense to
> >> > > > > > > have
> >> > > > > > > > > this.
> >> > > > > > > > > > The term "Mode" might not be explicit enough.
> >> > > Essentially it
> >> > > > > > > seems
> >> > > > > > > > > you want
> >> > > > > > > > > > the Authorizer to know the Intent/Purpose of
> the
> >> > > authorize call
> >> > > > > > > and
> >> > > > > > > > > let the
> >> > > > > > > > > > Authorizer decide what to log as Audit event.
> >> Changing
> >> > > the
> >> > > > > > > > > class/field name
> >> > > > > > > > > > or giving more documentation will do.
> >> > > > > > > > > > #3.2 - Regarding the option "OPTIONAL". Are
> you
> >> thinking
> >> > > from
> >> > > > > > > > > chaining
> >> > > > > > > > > > multiple Authorizer? If so, I am not sure
> whether
> >> the
> >> > > Broker
> >> > > > > > > would
> >> > > > > > > > > have
> >> > > > > > > > > > enough information to make that decision. I
> feel the
> >> > > Authorizer
> >> > > > > > > > will
> >> > > > > > > > > be the
> >> > > > > > > > > > one who would have that knowledge. E.g. in
> Ranger
> >> we have
> >> > > > > > > explicit
> >> > > > > > > > > deny,
> >> > > > > > > > > > which means no matter what, the request
> should be
> >> denied
> >> > > for
> >> > > > > > the
> >> > > > > > > > > user/group
> >> > > > > > > > > > or condition. So if you are thinking of
> chaining
> >> > > Authorizers,
> >> > > > > > > then
> >> > > > > > > > > > responses should have the third state, e.g.
> >> > > "DENIED_FINAL", in
> >> > > > > > > > which
> >> > > > > > > > > case
> >> > > > > > > > > > if there is an Authorization chain, it will
> be stop
> >> and
> >> > > the
> >> > > > > > > request
> >> > > > > > > > > will be
> >> > > > > > > > > > denied and if it is just denied, then you
> might fall
> >> > > back to
> >> > > > > > next
> >> > > > > > > > > > authorizer. If we don't have chaining of
> >> Authorizing in
> >> > > mind,
> >> > > > > > > then
> >> > > > > > > > we
> >> > > > > > > > > > should reconsider OPTIONAL for now. Or
> clarify under
> >> > > which
> >> > > > > > > scenario
> >> > > > > > > > > > OPTIONAL will be called.
> >> > > > > > > > > > #3.3 Regarding, FILTER v/s LIST_AUTHORIZED,
> isn't
> >> > > > > > > LIST_AUTHORIZED a
> >> > > > > > > > > > special case for "FILTER"?
> >> > > > > > > > > > #3.4 KafkaRequestContext. requestType() v/s
> Action.
> >> > > > > > > > > authorizationMode. I
> >> > > > > > > > > > am not sure about the overlap or ambiguity.
> >> > > > > > > > > > #4 - Cool. This is good, it will be less
> stress on
> >> the
> >> > > > > > > Authorizer.
> >> > > > > > > > > Ranger
> >> > > > > > > > > > already supports the "count" concept and also
> has
> >> > > batching
> >> > > > > > > > > capability to
> >> > > > > > > > > > aggregate similar requests to reduce the
> number of
> >> audit
> >> > > logs
> >> > > > > > to
> >> > > > > > > > > write. We
> >> > > > > > > > > > should be able to pass this through.
> >> > > > > > > > > > #5 - Assuming if the object instance is going
> out of
> >> > > scope, we
> >> > > > > > > > > should be
> >> > > > > > > > > > fine. Not a super important ask. Ranger is
> already
> >> > > catching
> >> > > > > > KILL
> >> > > > > > > > > signal for
> >> > > > > > > > > > clean up.
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks again. These are good enhancements.
> >> Appreciate
> >> > > your
> >> > > > > > > efforts
> >> > > > > > > > > here.
> >> > > > > > > > > >
> >> > > > > > > > > > Bosco
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On 8/8/19, 2:03 AM, "Rajini Sivaram" <
> >> > > rajinisivaram@gmail.com
> >> > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > Hi Don,
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks for reviewing the KIP.
> >> > > > > > > > > >
> >> > > > > > > > > > 1. I had this originally as a single
> Action, but
> >> > > thought it
> >> > > > > > > may
> >> > > > > > > > > be
> >> > > > > > > > > > useful
> >> > > > > > > > > > to support batched authorize calls as
> well and
> >> keep
> >> > > it
> >> > > > > > > > > consistent with
> >> > > > > > > > > > other methods. Single requests can contain
> >> multiple
> >> > > topics.
> >> > > > > > > For
> >> > > > > > > > > > example a
> >> > > > > > > > > > produce request can contain records for
> several
> >> > > partitions
> >> > > > > > of
> >> > > > > > > > > different
> >> > > > > > > > > > topics. Broker could potentially
> authorize these
> >> > > together.
> >> > > > > > > For
> >> > > > > > > > > > SimpleAclAuthorizer, batched authorize
> methods
> >> don't
> >> > > > > > provide
> >> > > > > > > > any
> >> > > > > > > > > > optimisation since lookup is based on
> resources
> >> > > followed by
> >> > > > > > > the
> >> > > > > > > > > > matching
> >> > > > > > > > > > logic. But some authorizers may manage
> ACLs by
> >> user
> >> > > > > > principal
> >> > > > > > > > > rather
> >> > > > > > > > > > than
> >> > > > > > > > > > resource and may be able to optimize
> batched
> >> > > requests. I am
> >> > > > > > > ok
> >> > > > > > > > > with
> >> > > > > > > > > > using
> >> > > > > > > > > > single Action if this is likely to cause
> issues.
> >> > > > > > > > > > 2. If you have two listeners, one for
> >> inter-broker
> >> > > traffic
> >> > > > > > > and
> >> > > > > > > > > another
> >> > > > > > > > > > for
> >> > > > > > > > > > external clients, start method is invoked
> twice,
> >> > > once for
> >> > > > > > > each
> >> > > > > > > > > > listener. On
> >> > > > > > > > > > second thought, that may be confusing and
> a
> >> single
> >> > > start()
> >> > > > > > > > > invocation
> >> > > > > > > > > > that
> >> > > > > > > > > > provides all listener information and
> returns
> >> > > multiple
> >> > > > > > > futures
> >> > > > > > > > > would be
> >> > > > > > > > > > better. Will update the KIP.
> >> > > > > > > > > > 3. A typical example is a consumer
> subscribing
> >> to a
> >> > > regex
> >> > > > > > > > > pattern. We
> >> > > > > > > > > > request all topic metadata from the
> broker in
> >> order
> >> > > to
> >> > > > > > decide
> >> > > > > > > > > whether
> >> > > > > > > > > > the
> >> > > > > > > > > > pattern matches, expecting to receive a
> list of
> >> > > authorised
> >> > > > > > > > > topics. The
> >> > > > > > > > > > user
> >> > > > > > > > > > is not asking to subscribe to an
> unauthorized
> >> topic.
> >> > > If
> >> > > > > > there
> >> > > > > > > > > are 10000
> >> > > > > > > > > > topics in the cluster and the user has
> access
> >> to 100
> >> > > of
> >> > > > > > them,
> >> > > > > > > > at
> >> > > > > > > > > the
> >> > > > > > > > > > moment
> >> > > > > > > > > > we log 9900 DENIED log entries at INFO
> level in
> >> > > > > > > > > SimpleAclAuthorizer.
> >> > > > > > > > > > The
> >> > > > > > > > > > proposal is to authorize this request with
> >> > > > > > > > > AuthorizationMode.FILTER, so
> >> > > > > > > > > > that authorizers can log resources that
> are
> >> filtered
> >> > > out at
> >> > > > > > > > > lower level
> >> > > > > > > > > > like DEBUG since this is not an attempt to
> >> access
> >> > > > > > > unauthorized
> >> > > > > > > > > > resources.
> >> > > > > > > > > > Brokers already handle these differently
> since
> >> no
> >> > > > > > > authorization
> >> > > > > > > > > error
> >> > > > > > > > > > is
> >> > > > > > > > > > returned to the client in these cases.
> Providing
> >> > > > > > > authorization
> >> > > > > > > > > mode to
> >> > > > > > > > > > authorizers enables authorizer
> implementations
> >> to
> >> > > generate
> >> > > > > > > > > better audit
> >> > > > > > > > > > logs.
> >> > > > > > > > > > 4. Each request may contain multiple
> instances
> >> of
> >> > > the same
> >> > > > > > > > > authorizable
> >> > > > > > > > > > resource. For example a produce request
> may
> >> contain
> >> > > records
> >> > > > > > > for
> >> > > > > > > > > 10
> >> > > > > > > > > > partitions of the same topic. At the
> moment, we
> >> > > invoke
> >> > > > > > > > authorize
> >> > > > > > > > > > method 10
> >> > > > > > > > > > times. The proposal is to invoke it once
> with
> >> > > count=10. The
> >> > > > > > > > > count is
> >> > > > > > > > > > provided to authorizer just for audit
> logging
> >> > > purposes.
> >> > > > > > > > > > 5. Authorizer implements Closeable, so you
> >> could use
> >> > > > > > close()
> >> > > > > > > to
> >> > > > > > > > > flush
> >> > > > > > > > > > audits?
> >> > > > > > > > > >
> >> > > > > > > > > > On Thu, Aug 8, 2019 at 7:01 AM Don Bosco
> Durai <
> >> > > > > > > > bosco@apache.org
> >> > > > > > > > > >
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Rajini
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks for putting this together. It is
> >> looking
> >> > > good. I
> >> > > > > > > have
> >> > > > > > > > > few
> >> > > > > > > > > > > questions...
> >> > > > > > > > > > >
> >> > > > > > > > > > > 1. List<AuthorizationResult>
> authorize(...,
> >> > > List<Action>
> >> > > > > > > > > actions).
> >> > > > > > > > > > Do you
> >> > > > > > > > > > > see a scenario where the broker will
> call
> >> > > authorize for
> >> > > > > > > > > multiple
> >> > > > > > > > > > topics at
> >> > > > > > > > > > > the same time? I can understand that
> during
> >> > > > > > > creating/deleting
> >> > > > > > > > > ACLS,
> >> > > > > > > > > > > multiple permissions for multiple
> resources
> >> might
> >> > > be
> >> > > > > > done.
> >> > > > > > > > For
> >> > > > > > > > > > authorize
> >> > > > > > > > > > > call, would this be a case? And does the
> >> Authorize
> >> > > > > > > > > implementation
> >> > > > > > > > > > will be
> >> > > > > > > > > > > able to do performance optimization
> because of
> >> > > this? Or
> >> > > > > > > > should
> >> > > > > > > > > we
> >> > > > > > > > > > just keep
> >> > > > > > > > > > > it simple? I don't see it as an issue
> from
> >> Apache
> >> > > Ranger
> >> > > > > > > > side,
> >> > > > > > > > > but
> >> > > > > > > > > > just
> >> > > > > > > > > > > checking to see whether we need to be
> aware of
> >> > > something.
> >> > > > > > > > > > > 2. Should I assume that the
> SecurityProtocol
> >> passed
> >> > > > > > during
> >> > > > > > > > > start and
> >> > > > > > > > > > the
> >> > > > > > > > > > > one return by
> >> > > KafkaRequestContext.securityProtocol() will
> >> > > > > > > be
> >> > > > > > > > > the
> >> > > > > > > > > > same?
> >> > > > > > > > > > > CompletableFuture<Void> start(String
> >> listenerName,
> >> > > > > > > > > SecurityProtocol
> >> > > > > > > > > > > securityProtocol);
> >> > > > > > > > > > > KafkaRequestContext.securityProtocol()
> >> > > > > > > > > > > 3. What is the purpose of
> AuthorizationMode?
> >> How
> >> > > does the
> >> > > > > > > > > broker
> >> > > > > > > > > > decide
> >> > > > > > > > > > > what mode to use when the authorize()
> method
> >> is
> >> > > called?
> >> > > > > > > > > > > 4. Can we clarify "count" in Action a
> bit
> >> more?
> >> > > How is it
> >> > > > > > > > used?
> >> > > > > > > > > > > 5. Do you feel having "stop" along with
> >> "start" be
> >> > > > > > helpful?
> >> > > > > > > > > E.g. In
> >> > > > > > > > > > Ranger
> >> > > > > > > > > > > we try to optimize the Audit writing by
> >> caching
> >> > > the logs
> >> > > > > > > for
> >> > > > > > > > a
> >> > > > > > > > > fixed
> >> > > > > > > > > > > interval. But when the Broker
> terminates, we
> >> do a
> >> > > forced
> >> > > > > > > > flush.
> >> > > > > > > > > > Having an
> >> > > > > > > > > > > explicit "stop" might give us a formal
> way to
> >> > > flush our
> >> > > > > > > > audits.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks
> >> > > > > > > > > > >
> >> > > > > > > > > > > Bosco
> >> > > > > > > > > > >
> >> > > > > > > > > > > On 8/7/19, 3:59 PM, "Rajini Sivaram" <
> >> > > > > > > > rajinisivaram@gmail.com
> >> > > > > > > > > >
> >> > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > Hi Ron/Harsha/Satish,
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks for reviewing the KIP!
> >> > > > > > > > > > >
> >> > > > > > > > > > > We should perhaps have a wider
> discussion
> >> > > outside
> >> > > > > > this
> >> > > > > > > > KIP
> >> > > > > > > > > for
> >> > > > > > > > > > > refactoring
> >> > > > > > > > > > > clients so that others who are not
> >> following
> >> > > this KIP
> >> > > > > > > > also
> >> > > > > > > > > > notice the
> >> > > > > > > > > > > discussion. Satish, would you like
> to
> >> start a
> >> > > > > > > discussion
> >> > > > > > > > > thread
> >> > > > > > > > > > on dev?
> >> > > > > > > > > > >
> >> > > > > > > > > > > Regards,
> >> > > > > > > > > > >
> >> > > > > > > > > > > Rajini
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Wed, Aug 7, 2019 at 6:21 PM
> Satish
> >> Duggana <
> >> > > > > > > > > > > satish.duggana@gmail.com>
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > I felt the same need when we want
> to
> >> add a
> >> > > > > > pluggable
> >> > > > > > > > API
> >> > > > > > > > > for
> >> > > > > > > > > > core
> >> > > > > > > > > > > > server functionality. This does
> not
> >> need to
> >> > > be part
> >> > > > > > > of
> >> > > > > > > > > this
> >> > > > > > > > > > KIP, it
> >> > > > > > > > > > > > can be a separate KIP. I can
> contribute
> >> those
> >> > > > > > > > refactoring
> >> > > > > > > > > > changes if
> >> > > > > > > > > > > > others are OK with that.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > It is better to have a structure
> like
> >> below.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > kafka-common:
> >> > > > > > > > > > > > common classes which can be used
> in any
> >> of
> >> > > the
> >> > > > > > other
> >> > > > > > > > > modules
> >> > > > > > > > > > in Kafka
> >> > > > > > > > > > > > like client, Kafka-server-common
> and
> >> server
> >> > > etc.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > kafka-client-common:
> >> > > > > > > > > > > > common classes which can be used
> in the
> >> > > client
> >> > > > > > > module.
> >> > > > > > > > > This
> >> > > > > > > > > > can be
> >> > > > > > > > > > > > part of client module itself.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > kafka-server-common:
> >> > > > > > > > > > > > classes required only for
> kafka-server.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks.
> >> > > > > > > > > > > > Satish.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Wed, Aug 7, 2019 at 9:28 PM
> Harsha
> >> > > Chintalapani
> >> > > > > > <
> >> > > > > > > > > > kafka@harsha.io>
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Thanks for the KIP Rajini.
> >> > > > > > > > > > > > > Quick thought, it would be good
> to
> >> have a
> >> > > common
> >> > > > > > > > module
> >> > > > > > > > > > outside of
> >> > > > > > > > > > > > clients
> >> > > > > > > > > > > > > that only applies to server side
> >> > > interfaces &
> >> > > > > > > > changes.
> >> > > > > > > > > It
> >> > > > > > > > > > looks
> >> > > > > > > > > > > like we
> >> > > > > > > > > > > > are
> >> > > > > > > > > > > > > increasingly in favor of using
> Java
> >> > > interface for
> >> > > > > > > > > pluggable
> >> > > > > > > > > > > modules on
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > > broker side.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > Harsha
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Tue, Aug 06, 2019 at 2:31 PM,
> >> Rajini
> >> > > Sivaram <
> >> > > > > > > > > > > rajinisivaram@gmail.com
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Hi all,
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > I have created a KIP to
> replace the
> >> Scala
> >> > > > > > > > Authorizer
> >> > > > > > > > > API
> >> > > > > > > > > > with a
> >> > > > > > > > > > > new
> >> > > > > > > > > > > > Java
> >> > > > > > > > > > > > > > API:
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > -
> >> > > > > > > > > > > > > >
> >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> >> > > > > > > > > > > > > >
> >> > > KIP-504+-+Add+new+Java+Authorizer+Interface
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > This is replacement for KIP-50
> >> which was
> >> > > > > > accepted
> >> > > > > > > > > but never
> >> > > > > > > > > > > merged.
> >> > > > > > > > > > > > Apart
> >> > > > > > > > > > > > > > from moving to a Java API
> consistent
> >> > > with other
> >> > > > > > > > > pluggable
> >> > > > > > > > > > > interfaces
> >> > > > > > > > > > > > in the
> >> > > > > > > > > > > > > > broker, KIP-504 also attempts
> to
> >> address
> >> > > known
> >> > > > > > > > > limitations
> >> > > > > > > > > > in the
> >> > > > > > > > > > > > > > authorizer. If you have come
> across
> >> other
> >> > > > > > > > > limitations that
> >> > > > > > > > > > you
> >> > > > > > > > > > > would
> >> > > > > > > > > > > > like
> >> > > > > > > > > > > > > > to see addressed in the new
> API,
> >> please
> >> > > raise
> >> > > > > > > these
> >> > > > > > > > > on the
> >> > > > > > > > > > > discussion
> >> > > > > > > > > > > > > > thread so that we can
> consider those
> >> > > too. All
> >> > > > > > > > > suggestions
> >> > > > > > > > > > and
> >> > > > > > > > > > > feedback
> >> > > > > > > > > > > > are
> >> > > > > > > > > > > > > > welcome.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Thank you,
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Rajini
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
>
>
>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Don,
That should be fine. I guess Ranger loads policies from the database
synchronously when the authorizer is configured, similar to
SimpleAclAuthorizer loading from ZooKeeper. Ranger can continue to load
synchronously from `configure()` or `start()` and return an empty map from
`start()`. That would retain the existing behaviour.. When the same
database stores policies for all listeners and the policies are not stored
in Kafka, there is no value in making the load asynchronous.
Regards,
Rajini
On Fri, Aug 16, 2019 at 6:43 PM Don Bosco Durai <bo...@apache.org> wrote:
> Hi Rajini
>
> Assuming this doesn't affect custom plugins, I don't have any concerns
> regarding this.
>
> I do have one question regarding:
>
> "For authorizers that don’t store metadata in ZooKeeper, ensure that
> authorizer metadata for each listener is available before starting up the
> listener. This enables different authorization metadata stores for
> different listeners."
>
> Since Ranger uses its own database for storing policies, do you anticipate
> any issues?
>
> Thanks
>
> Bosco
>
>
> On 8/16/19, 6:49 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
>
> Hi all,
>
> I made another change to the KIP. The KIP was originally proposing to
> extend SimpleAclAuthorizer to also implement the new API (in addition
> to
> the existing API). But since we use the new API when available, this
> can
> break custom authorizers that extend this class and override specific
> methods of the old API. To avoid breaking any existing custom
> implementations that extend this class, particularly because it is in
> the
> public package kafka.security.auth, the KIP now proposes to retain the
> old
> authorizer as-is. SimpleAclAuthorizer will continue to implement the
> old
> API, but will be deprecated. A new authorizer implementation
> kafka.security.authorizer.AclAuthorizer will be added for the new API
> (this
> will not be in the public package).
>
> Please let me know if you have any concerns.
>
> Regards,
>
> Rajini
>
>
> On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <
> rajinisivaram@gmail.com>
> wrote:
>
> > Thanks Colin.
> >
> > If there are no other concerns, I will start vote later today. Many
> thanks
> > to every one for the feedback.
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <cm...@apache.org>
> wrote:
> >
> >> Thanks, Rajini. It looks good to me.
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram wrote:
> >> > Hi Colin,
> >> >
> >> > Thanks for the review. I have updated the KIP to move the
> interfaces for
> >> > request context and server info to the authorizer package. These
> are now
> >> > called AuthorizableRequestContext and AuthorizerServerInfo.
> Endpoint is
> >> now
> >> > a class in org.apache.kafka.common to make it reusable since we
> already
> >> > have multiple implementations of it. I have removed requestName
> from the
> >> > request context interface since authorizers can distinguish
> follower
> >> fetch
> >> > and consumer fetch from the operation being authorized. So 16-bit
> >> request
> >> > type should be sufficient for audit logging. Also replaced
> AuditFlag
> >> with
> >> > two booleans as you suggested.
> >> >
> >> > Can you take another look and see if the KIP is ready for voting?
> >> >
> >> > Thanks for all your help!
> >> >
> >> > Regards,
> >> >
> >> > Rajini
> >> >
> >> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <cm...@apache.org>
> >> wrote:
> >> >
> >> > > Hi Rajini,
> >> > >
> >> > > I think it would be good to rename KafkaRequestContext to
> something
> >> like
> >> > > AuthorizableRequestContext, and put it in the
> >> > > org.apache.kafka.server.authorizer namespace. If we put it in
> the
> >> > > org.apache.kafka.common namespace, then it's not really clear
> that
> >> it's
> >> > > part of the Authorizer API. Since this class is purely an
> interface,
> >> with
> >> > > no concrete implementation of anything, there's nothing common
> to
> >> really
> >> > > reuse in any case. We definitely don't want someone to
> accidentally
> >> add or
> >> > > remove methods because they think this is just another internal
> class
> >> used
> >> > > for requests.
> >> > >
> >> > > The BrokerInfo class is a nice improvement. It looks like it
> will be
> >> > > useful for passing in information about the context we're
> running
> >> in. It
> >> > > would be nice to call this class ServerInfo rather than
> BrokerInfo,
> >> since
> >> > > we will want to run the authorizer on controllers as well as on
> >> brokers,
> >> > > and the controller may run as a separate process post KIP-500.
> I also
> >> > > think that this class should be in the
> >> org.apache.kafka.server.authorizer
> >> > > namespace. Again, it is an interface, not a concrete
> implementation,
> >> and
> >> > > it's an interface that is very specifically for the authorizer.
> >> > >
> >> > > I agree that we probably don't have enough information
> preserved for
> >> > > requests currently to always know what entity made them. So we
> can
> >> leave
> >> > > that out for now (except in the special case of Fetch).
> Perhaps we
> >> can add
> >> > > this later if it's needed.
> >> > >
> >> > > I understand the intention behind AuthorizationMode (which is
> now
> >> called
> >> > > AuditFlag in the latest revision). But it still feels
> complex. What
> >> if we
> >> > > just had two booleans in Action: logSuccesses and logFailures?
> That
> >> seems
> >> > > to cover all the cases here. MANDATORY_AUTHORIZE = true, true.
> >> > > OPTIONAL_AUTHORIZE = true, false. FILTER = true, false.
> >> LIST_AUTHORIZED =
> >> > > false, false. Would there be anything lost versus having the
> enum?
> >> > >
> >> > > best,
> >> > > Colin
> >> > >
> >> > >
> >> > > On Wed, Aug 14, 2019, at 06:29, Mickael Maison wrote:
> >> > > > Hi Rajini,
> >> > > >
> >> > > > Thanks for the KIP!
> >> > > > I really like that authorize() will be able to take a batch of
> >> > > > requests, this will speed up many implementations!
> >> > > >
> >> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <
> >> rajinisivaram@gmail.com>
> >> > > wrote:
> >> > > > >
> >> > > > > Thanks David! I have fixed the typo.
> >> > > > >
> >> > > > > Also made a couple of changes to make the context
> interfaces more
> >> > > generic.
> >> > > > > KafkaRequestContext now returns the 16-bit API key as Colin
> >> suggested
> >> > > as
> >> > > > > well as the friendly name used in metrics which are useful
> in
> >> audit
> >> > > logs.
> >> > > > > `Authorizer#start` is now provided a generic `BrokerInfo`
> >> interface
> >> > > that
> >> > > > > gives cluster id, broker id and endpoint information. The
> generic
> >> > > interface
> >> > > > > can potentially be used in other broker plugins in future
> and
> >> provides
> >> > > > > dynamically generated configs like broker id and ports
> which are
> >> > > currently
> >> > > > > not available to plugins unless these configs are statically
> >> > > configured.
> >> > > > > Please let me know if there are any concerns.
> >> > > > >
> >> > > > > Regards,
> >> > > > >
> >> > > > > Rajini
> >> > > > >
> >> > > > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot <
> djacot@confluent.io>
> >> > > wrote:
> >> > > > >
> >> > > > > > Hi Rajini,
> >> > > > > >
> >> > > > > > Thank you for the update! It looks good to me. There is a
> typo
> >> in the
> >> > > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` ->
> >> `MANDATORY_AUTHORIZE`.
> >> > > > > >
> >> > > > > > Regards,
> >> > > > > > David
> >> > > > > >
> >> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram <
> >> > > rajinisivaram@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi David,
> >> > > > > > >
> >> > > > > > > Thanks for reviewing the KIP! Since questions about
> >> `authorization
> >> > > mode`
> >> > > > > > > and `count` have come up multiple times, I have renamed
> both.
> >> > > > > > >
> >> > > > > > > 1) Renamed `count` to `resourceReferenceCount`. It is
> the
> >> number
> >> > > of times
> >> > > > > > > the resource being authorized is referenced within the
> >> request.
> >> > > > > > >
> >> > > > > > > 2) Renamed `AuthorizationMode` to `AuditFlag`. It is
> provided
> >> to
> >> > > improve
> >> > > > > > > audit logging in the authorizer. The enum values have
> javadoc
> >> which
> >> > > > > > > indicate how the authorization result is used in each
> of the
> >> modes
> >> > > to
> >> > > > > > > enable authorizers to log audit messages at the right
> severity
> >> > > level.
> >> > > > > > >
> >> > > > > > > Regards,
> >> > > > > > >
> >> > > > > > > Rajini
> >> > > > > > >
> >> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM David Jacot <
> >> djacot@confluent.io>
> >> > > wrote:
> >> > > > > > >
> >> > > > > > > > Hi Rajini,
> >> > > > > > > >
> >> > > > > > > > Thank you for the KIP. Overall, it looks good to me.
> I have
> >> few
> >> > > > > > > > questions/suggestions:
> >> > > > > > > >
> >> > > > > > > > 1. It is hard to grasp what `Action#count` is for. I
> guess I
> >> > > understand
> >> > > > > > > > where you want to go with it but it took me a while to
> >> figure it
> >> > > out.
> >> > > > > > > > Perhaps, we could come up with a better name than
> `count`?
> >> > > > > > > >
> >> > > > > > > > 2. I had a hard time trying to understand the
> >> `AuthorizationMode`
> >> > > > > > > concept,
> >> > > > > > > > especially wrt. the OPTIONAL one. My understanding is
> that
> >> an
> >> > > ACL is
> >> > > > > > > either
> >> > > > > > > > defined or not. Could you elaborate a bit more on
> that?
> >> > > > > > > >
> >> > > > > > > > Thanks,
> >> > > > > > > > David
> >> > > > > > > >
> >> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai <
> >> > > bosco@apache.org>
> >> > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Hi Rajini
> >> > > > > > > > >
> >> > > > > > > > > 3.2 - This makes sense. Thanks for clarifying.
> >> > > > > > > > >
> >> > > > > > > > > Rest looks fine. Once the implementations are done,
> it
> >> will be
> >> > > more
> >> > > > > > > clear
> >> > > > > > > > > on the different values RequestType and Mode.
> >> > > > > > > > >
> >> > > > > > > > > Thanks
> >> > > > > > > > >
> >> > > > > > > > > Bosco
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <
> >> rajinisivaram@gmail.com
> >> > > >
> >> > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > Hi Don,
> >> > > > > > > > >
> >> > > > > > > > > Thanks for the suggestions. A few responses
> below:
> >> > > > > > > > >
> >> > > > > > > > > 3.1 Can rename and improve docs if we keep
> this. Let's
> >> > > finish the
> >> > > > > > > > > discussion on Colin's suggestions regarding this
> >> first.
> >> > > > > > > > > 3.2 No, I was thinking of some requests that
> have an
> >> old
> >> > > way of
> >> > > > > > > > > authorizing
> >> > > > > > > > > and a new way where we have retained the old
> way for
> >> > > backward
> >> > > > > > > > > compatibility. One example is Cluster:Create
> >> permission to
> >> > > create
> >> > > > > > > > > topics.
> >> > > > > > > > > We have replaced this with fine-grained topic
> create
> >> > > access using
> >> > > > > > > > > Topic:Create
> >> > > > > > > > > for topic patterns. But we still check if user
> has
> >> > > Cluster:Create
> >> > > > > > > > > first. If
> >> > > > > > > > > Denied, the deny is ignored and we check
> >> Topic:Create. We
> >> > > dont
> >> > > > > > want
> >> > > > > > > > to
> >> > > > > > > > > log
> >> > > > > > > > > DENY for Cluster:Create at INFO level for this,
> since
> >> this
> >> > > is
> >> > > > > > not a
> >> > > > > > > > > mandatory ACL for creating topics. We will get
> >> appropriate
> >> > > logs
> >> > > > > > > from
> >> > > > > > > > > the
> >> > > > > > > > > subsequent Topic:Create in this case.
> >> > > > > > > > > 3.3 They are not quite the same. FILTER implies
> that
> >> user
> >> > > > > > actually
> >> > > > > > > > > requested access to and performed some
> operation on
> >> the
> >> > > filtered
> >> > > > > > > > > resources.
> >> > > > > > > > > LIST_AUTHORZED did not result in any actual
> access.
> >> User
> >> > > simply
> >> > > > > > > > wanted
> >> > > > > > > > > to
> >> > > > > > > > > know what they are allowed to access.
> >> > > > > > > > > 3.4 Request types are Produce, JoinGroup,
> OffsetCommit
> >> > > etc. So
> >> > > > > > that
> >> > > > > > > > is
> >> > > > > > > > > different from authorization mode, operation
> etc.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai
> <
> >> > > > > > bosco@apache.org>
> >> > > > > > > > > wrote:
> >> > > > > > > > >
> >> > > > > > > > > > Hi Rajini
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks for clarifying. This is very helpful...
> >> > > > > > > > > >
> >> > > > > > > > > > #1 - On the Ranger side, we should be able to
> handle
> >> > > multiple
> >> > > > > > > > > requests at
> >> > > > > > > > > > the same time. I was just not sure how much
> >> processing
> >> > > overhead
> >> > > > > > > > will
> >> > > > > > > > > be
> >> > > > > > > > > > there on the Broker side to split and then
> >> consolidate
> >> > > the
> >> > > > > > > results.
> >> > > > > > > > > If it
> >> > > > > > > > > > is negligible, then this is the better way.
> It will
> >> > > make it
> >> > > > > > > future
> >> > > > > > > > > proof.
> >> > > > > > > > > > #2 - I agree, having a single "start" call
> makes it
> >> > > cleaner.
> >> > > > > > The
> >> > > > > > > > > > Authorization implementation will only have
> to do
> >> the
> >> > > > > > > > initialization
> >> > > > > > > > > only
> >> > > > > > > > > > once.
> >> > > > > > > > > > #3.1 - Thanks for the clarification. I think
> it
> >> makes
> >> > > sense to
> >> > > > > > > have
> >> > > > > > > > > this.
> >> > > > > > > > > > The term "Mode" might not be explicit enough.
> >> > > Essentially it
> >> > > > > > > seems
> >> > > > > > > > > you want
> >> > > > > > > > > > the Authorizer to know the Intent/Purpose of
> the
> >> > > authorize call
> >> > > > > > > and
> >> > > > > > > > > let the
> >> > > > > > > > > > Authorizer decide what to log as Audit event.
> >> Changing
> >> > > the
> >> > > > > > > > > class/field name
> >> > > > > > > > > > or giving more documentation will do.
> >> > > > > > > > > > #3.2 - Regarding the option "OPTIONAL". Are
> you
> >> thinking
> >> > > from
> >> > > > > > > > > chaining
> >> > > > > > > > > > multiple Authorizer? If so, I am not sure
> whether
> >> the
> >> > > Broker
> >> > > > > > > would
> >> > > > > > > > > have
> >> > > > > > > > > > enough information to make that decision. I
> feel the
> >> > > Authorizer
> >> > > > > > > > will
> >> > > > > > > > > be the
> >> > > > > > > > > > one who would have that knowledge. E.g. in
> Ranger
> >> we have
> >> > > > > > > explicit
> >> > > > > > > > > deny,
> >> > > > > > > > > > which means no matter what, the request
> should be
> >> denied
> >> > > for
> >> > > > > > the
> >> > > > > > > > > user/group
> >> > > > > > > > > > or condition. So if you are thinking of
> chaining
> >> > > Authorizers,
> >> > > > > > > then
> >> > > > > > > > > > responses should have the third state, e.g.
> >> > > "DENIED_FINAL", in
> >> > > > > > > > which
> >> > > > > > > > > case
> >> > > > > > > > > > if there is an Authorization chain, it will
> be stop
> >> and
> >> > > the
> >> > > > > > > request
> >> > > > > > > > > will be
> >> > > > > > > > > > denied and if it is just denied, then you
> might fall
> >> > > back to
> >> > > > > > next
> >> > > > > > > > > > authorizer. If we don't have chaining of
> >> Authorizing in
> >> > > mind,
> >> > > > > > > then
> >> > > > > > > > we
> >> > > > > > > > > > should reconsider OPTIONAL for now. Or
> clarify under
> >> > > which
> >> > > > > > > scenario
> >> > > > > > > > > > OPTIONAL will be called.
> >> > > > > > > > > > #3.3 Regarding, FILTER v/s LIST_AUTHORIZED,
> isn't
> >> > > > > > > LIST_AUTHORIZED a
> >> > > > > > > > > > special case for "FILTER"?
> >> > > > > > > > > > #3.4 KafkaRequestContext. requestType() v/s
> Action.
> >> > > > > > > > > authorizationMode. I
> >> > > > > > > > > > am not sure about the overlap or ambiguity.
> >> > > > > > > > > > #4 - Cool. This is good, it will be less
> stress on
> >> the
> >> > > > > > > Authorizer.
> >> > > > > > > > > Ranger
> >> > > > > > > > > > already supports the "count" concept and also
> has
> >> > > batching
> >> > > > > > > > > capability to
> >> > > > > > > > > > aggregate similar requests to reduce the
> number of
> >> audit
> >> > > logs
> >> > > > > > to
> >> > > > > > > > > write. We
> >> > > > > > > > > > should be able to pass this through.
> >> > > > > > > > > > #5 - Assuming if the object instance is going
> out of
> >> > > scope, we
> >> > > > > > > > > should be
> >> > > > > > > > > > fine. Not a super important ask. Ranger is
> already
> >> > > catching
> >> > > > > > KILL
> >> > > > > > > > > signal for
> >> > > > > > > > > > clean up.
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks again. These are good enhancements.
> >> Appreciate
> >> > > your
> >> > > > > > > efforts
> >> > > > > > > > > here.
> >> > > > > > > > > >
> >> > > > > > > > > > Bosco
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > > On 8/8/19, 2:03 AM, "Rajini Sivaram" <
> >> > > rajinisivaram@gmail.com
> >> > > > > > >
> >> > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > Hi Don,
> >> > > > > > > > > >
> >> > > > > > > > > > Thanks for reviewing the KIP.
> >> > > > > > > > > >
> >> > > > > > > > > > 1. I had this originally as a single
> Action, but
> >> > > thought it
> >> > > > > > > may
> >> > > > > > > > > be
> >> > > > > > > > > > useful
> >> > > > > > > > > > to support batched authorize calls as
> well and
> >> keep
> >> > > it
> >> > > > > > > > > consistent with
> >> > > > > > > > > > other methods. Single requests can contain
> >> multiple
> >> > > topics.
> >> > > > > > > For
> >> > > > > > > > > > example a
> >> > > > > > > > > > produce request can contain records for
> several
> >> > > partitions
> >> > > > > > of
> >> > > > > > > > > different
> >> > > > > > > > > > topics. Broker could potentially
> authorize these
> >> > > together.
> >> > > > > > > For
> >> > > > > > > > > > SimpleAclAuthorizer, batched authorize
> methods
> >> don't
> >> > > > > > provide
> >> > > > > > > > any
> >> > > > > > > > > > optimisation since lookup is based on
> resources
> >> > > followed by
> >> > > > > > > the
> >> > > > > > > > > > matching
> >> > > > > > > > > > logic. But some authorizers may manage
> ACLs by
> >> user
> >> > > > > > principal
> >> > > > > > > > > rather
> >> > > > > > > > > > than
> >> > > > > > > > > > resource and may be able to optimize
> batched
> >> > > requests. I am
> >> > > > > > > ok
> >> > > > > > > > > with
> >> > > > > > > > > > using
> >> > > > > > > > > > single Action if this is likely to cause
> issues.
> >> > > > > > > > > > 2. If you have two listeners, one for
> >> inter-broker
> >> > > traffic
> >> > > > > > > and
> >> > > > > > > > > another
> >> > > > > > > > > > for
> >> > > > > > > > > > external clients, start method is invoked
> twice,
> >> > > once for
> >> > > > > > > each
> >> > > > > > > > > > listener. On
> >> > > > > > > > > > second thought, that may be confusing and
> a
> >> single
> >> > > start()
> >> > > > > > > > > invocation
> >> > > > > > > > > > that
> >> > > > > > > > > > provides all listener information and
> returns
> >> > > multiple
> >> > > > > > > futures
> >> > > > > > > > > would be
> >> > > > > > > > > > better. Will update the KIP.
> >> > > > > > > > > > 3. A typical example is a consumer
> subscribing
> >> to a
> >> > > regex
> >> > > > > > > > > pattern. We
> >> > > > > > > > > > request all topic metadata from the
> broker in
> >> order
> >> > > to
> >> > > > > > decide
> >> > > > > > > > > whether
> >> > > > > > > > > > the
> >> > > > > > > > > > pattern matches, expecting to receive a
> list of
> >> > > authorised
> >> > > > > > > > > topics. The
> >> > > > > > > > > > user
> >> > > > > > > > > > is not asking to subscribe to an
> unauthorized
> >> topic.
> >> > > If
> >> > > > > > there
> >> > > > > > > > > are 10000
> >> > > > > > > > > > topics in the cluster and the user has
> access
> >> to 100
> >> > > of
> >> > > > > > them,
> >> > > > > > > > at
> >> > > > > > > > > the
> >> > > > > > > > > > moment
> >> > > > > > > > > > we log 9900 DENIED log entries at INFO
> level in
> >> > > > > > > > > SimpleAclAuthorizer.
> >> > > > > > > > > > The
> >> > > > > > > > > > proposal is to authorize this request with
> >> > > > > > > > > AuthorizationMode.FILTER, so
> >> > > > > > > > > > that authorizers can log resources that
> are
> >> filtered
> >> > > out at
> >> > > > > > > > > lower level
> >> > > > > > > > > > like DEBUG since this is not an attempt to
> >> access
> >> > > > > > > unauthorized
> >> > > > > > > > > > resources.
> >> > > > > > > > > > Brokers already handle these differently
> since
> >> no
> >> > > > > > > authorization
> >> > > > > > > > > error
> >> > > > > > > > > > is
> >> > > > > > > > > > returned to the client in these cases.
> Providing
> >> > > > > > > authorization
> >> > > > > > > > > mode to
> >> > > > > > > > > > authorizers enables authorizer
> implementations
> >> to
> >> > > generate
> >> > > > > > > > > better audit
> >> > > > > > > > > > logs.
> >> > > > > > > > > > 4. Each request may contain multiple
> instances
> >> of
> >> > > the same
> >> > > > > > > > > authorizable
> >> > > > > > > > > > resource. For example a produce request
> may
> >> contain
> >> > > records
> >> > > > > > > for
> >> > > > > > > > > 10
> >> > > > > > > > > > partitions of the same topic. At the
> moment, we
> >> > > invoke
> >> > > > > > > > authorize
> >> > > > > > > > > > method 10
> >> > > > > > > > > > times. The proposal is to invoke it once
> with
> >> > > count=10. The
> >> > > > > > > > > count is
> >> > > > > > > > > > provided to authorizer just for audit
> logging
> >> > > purposes.
> >> > > > > > > > > > 5. Authorizer implements Closeable, so you
> >> could use
> >> > > > > > close()
> >> > > > > > > to
> >> > > > > > > > > flush
> >> > > > > > > > > > audits?
> >> > > > > > > > > >
> >> > > > > > > > > > On Thu, Aug 8, 2019 at 7:01 AM Don Bosco
> Durai <
> >> > > > > > > > bosco@apache.org
> >> > > > > > > > > >
> >> > > > > > > > > > wrote:
> >> > > > > > > > > >
> >> > > > > > > > > > > Rajini
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks for putting this together. It is
> >> looking
> >> > > good. I
> >> > > > > > > have
> >> > > > > > > > > few
> >> > > > > > > > > > > questions...
> >> > > > > > > > > > >
> >> > > > > > > > > > > 1. List<AuthorizationResult>
> authorize(...,
> >> > > List<Action>
> >> > > > > > > > > actions).
> >> > > > > > > > > > Do you
> >> > > > > > > > > > > see a scenario where the broker will
> call
> >> > > authorize for
> >> > > > > > > > > multiple
> >> > > > > > > > > > topics at
> >> > > > > > > > > > > the same time? I can understand that
> during
> >> > > > > > > creating/deleting
> >> > > > > > > > > ACLS,
> >> > > > > > > > > > > multiple permissions for multiple
> resources
> >> might
> >> > > be
> >> > > > > > done.
> >> > > > > > > > For
> >> > > > > > > > > > authorize
> >> > > > > > > > > > > call, would this be a case? And does the
> >> Authorize
> >> > > > > > > > > implementation
> >> > > > > > > > > > will be
> >> > > > > > > > > > > able to do performance optimization
> because of
> >> > > this? Or
> >> > > > > > > > should
> >> > > > > > > > > we
> >> > > > > > > > > > just keep
> >> > > > > > > > > > > it simple? I don't see it as an issue
> from
> >> Apache
> >> > > Ranger
> >> > > > > > > > side,
> >> > > > > > > > > but
> >> > > > > > > > > > just
> >> > > > > > > > > > > checking to see whether we need to be
> aware of
> >> > > something.
> >> > > > > > > > > > > 2. Should I assume that the
> SecurityProtocol
> >> passed
> >> > > > > > during
> >> > > > > > > > > start and
> >> > > > > > > > > > the
> >> > > > > > > > > > > one return by
> >> > > KafkaRequestContext.securityProtocol() will
> >> > > > > > > be
> >> > > > > > > > > the
> >> > > > > > > > > > same?
> >> > > > > > > > > > > CompletableFuture<Void> start(String
> >> listenerName,
> >> > > > > > > > > SecurityProtocol
> >> > > > > > > > > > > securityProtocol);
> >> > > > > > > > > > > KafkaRequestContext.securityProtocol()
> >> > > > > > > > > > > 3. What is the purpose of
> AuthorizationMode?
> >> How
> >> > > does the
> >> > > > > > > > > broker
> >> > > > > > > > > > decide
> >> > > > > > > > > > > what mode to use when the authorize()
> method
> >> is
> >> > > called?
> >> > > > > > > > > > > 4. Can we clarify "count" in Action a
> bit
> >> more?
> >> > > How is it
> >> > > > > > > > used?
> >> > > > > > > > > > > 5. Do you feel having "stop" along with
> >> "start" be
> >> > > > > > helpful?
> >> > > > > > > > > E.g. In
> >> > > > > > > > > > Ranger
> >> > > > > > > > > > > we try to optimize the Audit writing by
> >> caching
> >> > > the logs
> >> > > > > > > for
> >> > > > > > > > a
> >> > > > > > > > > fixed
> >> > > > > > > > > > > interval. But when the Broker
> terminates, we
> >> do a
> >> > > forced
> >> > > > > > > > flush.
> >> > > > > > > > > > Having an
> >> > > > > > > > > > > explicit "stop" might give us a formal
> way to
> >> > > flush our
> >> > > > > > > > audits.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks
> >> > > > > > > > > > >
> >> > > > > > > > > > > Bosco
> >> > > > > > > > > > >
> >> > > > > > > > > > > On 8/7/19, 3:59 PM, "Rajini Sivaram" <
> >> > > > > > > > rajinisivaram@gmail.com
> >> > > > > > > > > >
> >> > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > Hi Ron/Harsha/Satish,
> >> > > > > > > > > > >
> >> > > > > > > > > > > Thanks for reviewing the KIP!
> >> > > > > > > > > > >
> >> > > > > > > > > > > We should perhaps have a wider
> discussion
> >> > > outside
> >> > > > > > this
> >> > > > > > > > KIP
> >> > > > > > > > > for
> >> > > > > > > > > > > refactoring
> >> > > > > > > > > > > clients so that others who are not
> >> following
> >> > > this KIP
> >> > > > > > > > also
> >> > > > > > > > > > notice the
> >> > > > > > > > > > > discussion. Satish, would you like
> to
> >> start a
> >> > > > > > > discussion
> >> > > > > > > > > thread
> >> > > > > > > > > > on dev?
> >> > > > > > > > > > >
> >> > > > > > > > > > > Regards,
> >> > > > > > > > > > >
> >> > > > > > > > > > > Rajini
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > On Wed, Aug 7, 2019 at 6:21 PM
> Satish
> >> Duggana <
> >> > > > > > > > > > > satish.duggana@gmail.com>
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > >
> >> > > > > > > > > > > > I felt the same need when we want
> to
> >> add a
> >> > > > > > pluggable
> >> > > > > > > > API
> >> > > > > > > > > for
> >> > > > > > > > > > core
> >> > > > > > > > > > > > server functionality. This does
> not
> >> need to
> >> > > be part
> >> > > > > > > of
> >> > > > > > > > > this
> >> > > > > > > > > > KIP, it
> >> > > > > > > > > > > > can be a separate KIP. I can
> contribute
> >> those
> >> > > > > > > > refactoring
> >> > > > > > > > > > changes if
> >> > > > > > > > > > > > others are OK with that.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > It is better to have a structure
> like
> >> below.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > kafka-common:
> >> > > > > > > > > > > > common classes which can be used
> in any
> >> of
> >> > > the
> >> > > > > > other
> >> > > > > > > > > modules
> >> > > > > > > > > > in Kafka
> >> > > > > > > > > > > > like client, Kafka-server-common
> and
> >> server
> >> > > etc.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > kafka-client-common:
> >> > > > > > > > > > > > common classes which can be used
> in the
> >> > > client
> >> > > > > > > module.
> >> > > > > > > > > This
> >> > > > > > > > > > can be
> >> > > > > > > > > > > > part of client module itself.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > kafka-server-common:
> >> > > > > > > > > > > > classes required only for
> kafka-server.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks.
> >> > > > > > > > > > > > Satish.
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > On Wed, Aug 7, 2019 at 9:28 PM
> Harsha
> >> > > Chintalapani
> >> > > > > > <
> >> > > > > > > > > > kafka@harsha.io>
> >> > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Thanks for the KIP Rajini.
> >> > > > > > > > > > > > > Quick thought, it would be good
> to
> >> have a
> >> > > common
> >> > > > > > > > module
> >> > > > > > > > > > outside of
> >> > > > > > > > > > > > clients
> >> > > > > > > > > > > > > that only applies to server side
> >> > > interfaces &
> >> > > > > > > > changes.
> >> > > > > > > > > It
> >> > > > > > > > > > looks
> >> > > > > > > > > > > like we
> >> > > > > > > > > > > > are
> >> > > > > > > > > > > > > increasingly in favor of using
> Java
> >> > > interface for
> >> > > > > > > > > pluggable
> >> > > > > > > > > > > modules on
> >> > > > > > > > > > > > the
> >> > > > > > > > > > > > > broker side.
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > Thanks,
> >> > > > > > > > > > > > > Harsha
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > On Tue, Aug 06, 2019 at 2:31 PM,
> >> Rajini
> >> > > Sivaram <
> >> > > > > > > > > > > rajinisivaram@gmail.com
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > wrote:
> >> > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Hi all,
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > I have created a KIP to
> replace the
> >> Scala
> >> > > > > > > > Authorizer
> >> > > > > > > > > API
> >> > > > > > > > > > with a
> >> > > > > > > > > > > new
> >> > > > > > > > > > > > Java
> >> > > > > > > > > > > > > > API:
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > -
> >> > > > > > > > > > > > > >
> >> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> >> > > > > > > > > > > > > >
> >> > > KIP-504+-+Add+new+Java+Authorizer+Interface
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > This is replacement for KIP-50
> >> which was
> >> > > > > > accepted
> >> > > > > > > > > but never
> >> > > > > > > > > > > merged.
> >> > > > > > > > > > > > Apart
> >> > > > > > > > > > > > > > from moving to a Java API
> consistent
> >> > > with other
> >> > > > > > > > > pluggable
> >> > > > > > > > > > > interfaces
> >> > > > > > > > > > > > in the
> >> > > > > > > > > > > > > > broker, KIP-504 also attempts
> to
> >> address
> >> > > known
> >> > > > > > > > > limitations
> >> > > > > > > > > > in the
> >> > > > > > > > > > > > > > authorizer. If you have come
> across
> >> other
> >> > > > > > > > > limitations that
> >> > > > > > > > > > you
> >> > > > > > > > > > > would
> >> > > > > > > > > > > > like
> >> > > > > > > > > > > > > > to see addressed in the new
> API,
> >> please
> >> > > raise
> >> > > > > > > these
> >> > > > > > > > > on the
> >> > > > > > > > > > > discussion
> >> > > > > > > > > > > > > > thread so that we can
> consider those
> >> > > too. All
> >> > > > > > > > > suggestions
> >> > > > > > > > > > and
> >> > > > > > > > > > > feedback
> >> > > > > > > > > > > > are
> >> > > > > > > > > > > > > > welcome.
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Thank you,
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > > > > Rajini
> >> > > > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
>
>
>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Don Bosco Durai <bo...@apache.org>.
Hi Rajini
Assuming this doesn't affect custom plugins, I don't have any concerns regarding this.
I do have one question regarding:
"For authorizers that don’t store metadata in ZooKeeper, ensure that authorizer metadata for each listener is available before starting up the listener. This enables different authorization metadata stores for different listeners."
Since Ranger uses its own database for storing policies, do you anticipate any issues?
Thanks
Bosco
On 8/16/19, 6:49 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
Hi all,
I made another change to the KIP. The KIP was originally proposing to
extend SimpleAclAuthorizer to also implement the new API (in addition to
the existing API). But since we use the new API when available, this can
break custom authorizers that extend this class and override specific
methods of the old API. To avoid breaking any existing custom
implementations that extend this class, particularly because it is in the
public package kafka.security.auth, the KIP now proposes to retain the old
authorizer as-is. SimpleAclAuthorizer will continue to implement the old
API, but will be deprecated. A new authorizer implementation
kafka.security.authorizer.AclAuthorizer will be added for the new API (this
will not be in the public package).
Please let me know if you have any concerns.
Regards,
Rajini
On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <ra...@gmail.com>
wrote:
> Thanks Colin.
>
> If there are no other concerns, I will start vote later today. Many thanks
> to every one for the feedback.
>
> Regards,
>
> Rajini
>
>
> On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <cm...@apache.org> wrote:
>
>> Thanks, Rajini. It looks good to me.
>>
>> best,
>> Colin
>>
>>
>> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram wrote:
>> > Hi Colin,
>> >
>> > Thanks for the review. I have updated the KIP to move the interfaces for
>> > request context and server info to the authorizer package. These are now
>> > called AuthorizableRequestContext and AuthorizerServerInfo. Endpoint is
>> now
>> > a class in org.apache.kafka.common to make it reusable since we already
>> > have multiple implementations of it. I have removed requestName from the
>> > request context interface since authorizers can distinguish follower
>> fetch
>> > and consumer fetch from the operation being authorized. So 16-bit
>> request
>> > type should be sufficient for audit logging. Also replaced AuditFlag
>> with
>> > two booleans as you suggested.
>> >
>> > Can you take another look and see if the KIP is ready for voting?
>> >
>> > Thanks for all your help!
>> >
>> > Regards,
>> >
>> > Rajini
>> >
>> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <cm...@apache.org>
>> wrote:
>> >
>> > > Hi Rajini,
>> > >
>> > > I think it would be good to rename KafkaRequestContext to something
>> like
>> > > AuthorizableRequestContext, and put it in the
>> > > org.apache.kafka.server.authorizer namespace. If we put it in the
>> > > org.apache.kafka.common namespace, then it's not really clear that
>> it's
>> > > part of the Authorizer API. Since this class is purely an interface,
>> with
>> > > no concrete implementation of anything, there's nothing common to
>> really
>> > > reuse in any case. We definitely don't want someone to accidentally
>> add or
>> > > remove methods because they think this is just another internal class
>> used
>> > > for requests.
>> > >
>> > > The BrokerInfo class is a nice improvement. It looks like it will be
>> > > useful for passing in information about the context we're running
>> in. It
>> > > would be nice to call this class ServerInfo rather than BrokerInfo,
>> since
>> > > we will want to run the authorizer on controllers as well as on
>> brokers,
>> > > and the controller may run as a separate process post KIP-500. I also
>> > > think that this class should be in the
>> org.apache.kafka.server.authorizer
>> > > namespace. Again, it is an interface, not a concrete implementation,
>> and
>> > > it's an interface that is very specifically for the authorizer.
>> > >
>> > > I agree that we probably don't have enough information preserved for
>> > > requests currently to always know what entity made them. So we can
>> leave
>> > > that out for now (except in the special case of Fetch). Perhaps we
>> can add
>> > > this later if it's needed.
>> > >
>> > > I understand the intention behind AuthorizationMode (which is now
>> called
>> > > AuditFlag in the latest revision). But it still feels complex. What
>> if we
>> > > just had two booleans in Action: logSuccesses and logFailures? That
>> seems
>> > > to cover all the cases here. MANDATORY_AUTHORIZE = true, true.
>> > > OPTIONAL_AUTHORIZE = true, false. FILTER = true, false.
>> LIST_AUTHORIZED =
>> > > false, false. Would there be anything lost versus having the enum?
>> > >
>> > > best,
>> > > Colin
>> > >
>> > >
>> > > On Wed, Aug 14, 2019, at 06:29, Mickael Maison wrote:
>> > > > Hi Rajini,
>> > > >
>> > > > Thanks for the KIP!
>> > > > I really like that authorize() will be able to take a batch of
>> > > > requests, this will speed up many implementations!
>> > > >
>> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <
>> rajinisivaram@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > Thanks David! I have fixed the typo.
>> > > > >
>> > > > > Also made a couple of changes to make the context interfaces more
>> > > generic.
>> > > > > KafkaRequestContext now returns the 16-bit API key as Colin
>> suggested
>> > > as
>> > > > > well as the friendly name used in metrics which are useful in
>> audit
>> > > logs.
>> > > > > `Authorizer#start` is now provided a generic `BrokerInfo`
>> interface
>> > > that
>> > > > > gives cluster id, broker id and endpoint information. The generic
>> > > interface
>> > > > > can potentially be used in other broker plugins in future and
>> provides
>> > > > > dynamically generated configs like broker id and ports which are
>> > > currently
>> > > > > not available to plugins unless these configs are statically
>> > > configured.
>> > > > > Please let me know if there are any concerns.
>> > > > >
>> > > > > Regards,
>> > > > >
>> > > > > Rajini
>> > > > >
>> > > > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot <dj...@confluent.io>
>> > > wrote:
>> > > > >
>> > > > > > Hi Rajini,
>> > > > > >
>> > > > > > Thank you for the update! It looks good to me. There is a typo
>> in the
>> > > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` ->
>> `MANDATORY_AUTHORIZE`.
>> > > > > >
>> > > > > > Regards,
>> > > > > > David
>> > > > > >
>> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram <
>> > > rajinisivaram@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi David,
>> > > > > > >
>> > > > > > > Thanks for reviewing the KIP! Since questions about
>> `authorization
>> > > mode`
>> > > > > > > and `count` have come up multiple times, I have renamed both.
>> > > > > > >
>> > > > > > > 1) Renamed `count` to `resourceReferenceCount`. It is the
>> number
>> > > of times
>> > > > > > > the resource being authorized is referenced within the
>> request.
>> > > > > > >
>> > > > > > > 2) Renamed `AuthorizationMode` to `AuditFlag`. It is provided
>> to
>> > > improve
>> > > > > > > audit logging in the authorizer. The enum values have javadoc
>> which
>> > > > > > > indicate how the authorization result is used in each of the
>> modes
>> > > to
>> > > > > > > enable authorizers to log audit messages at the right severity
>> > > level.
>> > > > > > >
>> > > > > > > Regards,
>> > > > > > >
>> > > > > > > Rajini
>> > > > > > >
>> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM David Jacot <
>> djacot@confluent.io>
>> > > wrote:
>> > > > > > >
>> > > > > > > > Hi Rajini,
>> > > > > > > >
>> > > > > > > > Thank you for the KIP. Overall, it looks good to me. I have
>> few
>> > > > > > > > questions/suggestions:
>> > > > > > > >
>> > > > > > > > 1. It is hard to grasp what `Action#count` is for. I guess I
>> > > understand
>> > > > > > > > where you want to go with it but it took me a while to
>> figure it
>> > > out.
>> > > > > > > > Perhaps, we could come up with a better name than `count`?
>> > > > > > > >
>> > > > > > > > 2. I had a hard time trying to understand the
>> `AuthorizationMode`
>> > > > > > > concept,
>> > > > > > > > especially wrt. the OPTIONAL one. My understanding is that
>> an
>> > > ACL is
>> > > > > > > either
>> > > > > > > > defined or not. Could you elaborate a bit more on that?
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > David
>> > > > > > > >
>> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai <
>> > > bosco@apache.org>
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Rajini
>> > > > > > > > >
>> > > > > > > > > 3.2 - This makes sense. Thanks for clarifying.
>> > > > > > > > >
>> > > > > > > > > Rest looks fine. Once the implementations are done, it
>> will be
>> > > more
>> > > > > > > clear
>> > > > > > > > > on the different values RequestType and Mode.
>> > > > > > > > >
>> > > > > > > > > Thanks
>> > > > > > > > >
>> > > > > > > > > Bosco
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <
>> rajinisivaram@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > Hi Don,
>> > > > > > > > >
>> > > > > > > > > Thanks for the suggestions. A few responses below:
>> > > > > > > > >
>> > > > > > > > > 3.1 Can rename and improve docs if we keep this. Let's
>> > > finish the
>> > > > > > > > > discussion on Colin's suggestions regarding this
>> first.
>> > > > > > > > > 3.2 No, I was thinking of some requests that have an
>> old
>> > > way of
>> > > > > > > > > authorizing
>> > > > > > > > > and a new way where we have retained the old way for
>> > > backward
>> > > > > > > > > compatibility. One example is Cluster:Create
>> permission to
>> > > create
>> > > > > > > > > topics.
>> > > > > > > > > We have replaced this with fine-grained topic create
>> > > access using
>> > > > > > > > > Topic:Create
>> > > > > > > > > for topic patterns. But we still check if user has
>> > > Cluster:Create
>> > > > > > > > > first. If
>> > > > > > > > > Denied, the deny is ignored and we check
>> Topic:Create. We
>> > > dont
>> > > > > > want
>> > > > > > > > to
>> > > > > > > > > log
>> > > > > > > > > DENY for Cluster:Create at INFO level for this, since
>> this
>> > > is
>> > > > > > not a
>> > > > > > > > > mandatory ACL for creating topics. We will get
>> appropriate
>> > > logs
>> > > > > > > from
>> > > > > > > > > the
>> > > > > > > > > subsequent Topic:Create in this case.
>> > > > > > > > > 3.3 They are not quite the same. FILTER implies that
>> user
>> > > > > > actually
>> > > > > > > > > requested access to and performed some operation on
>> the
>> > > filtered
>> > > > > > > > > resources.
>> > > > > > > > > LIST_AUTHORZED did not result in any actual access.
>> User
>> > > simply
>> > > > > > > > wanted
>> > > > > > > > > to
>> > > > > > > > > know what they are allowed to access.
>> > > > > > > > > 3.4 Request types are Produce, JoinGroup, OffsetCommit
>> > > etc. So
>> > > > > > that
>> > > > > > > > is
>> > > > > > > > > different from authorization mode, operation etc.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <
>> > > > > > bosco@apache.org>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi Rajini
>> > > > > > > > > >
>> > > > > > > > > > Thanks for clarifying. This is very helpful...
>> > > > > > > > > >
>> > > > > > > > > > #1 - On the Ranger side, we should be able to handle
>> > > multiple
>> > > > > > > > > requests at
>> > > > > > > > > > the same time. I was just not sure how much
>> processing
>> > > overhead
>> > > > > > > > will
>> > > > > > > > > be
>> > > > > > > > > > there on the Broker side to split and then
>> consolidate
>> > > the
>> > > > > > > results.
>> > > > > > > > > If it
>> > > > > > > > > > is negligible, then this is the better way. It will
>> > > make it
>> > > > > > > future
>> > > > > > > > > proof.
>> > > > > > > > > > #2 - I agree, having a single "start" call makes it
>> > > cleaner.
>> > > > > > The
>> > > > > > > > > > Authorization implementation will only have to do
>> the
>> > > > > > > > initialization
>> > > > > > > > > only
>> > > > > > > > > > once.
>> > > > > > > > > > #3.1 - Thanks for the clarification. I think it
>> makes
>> > > sense to
>> > > > > > > have
>> > > > > > > > > this.
>> > > > > > > > > > The term "Mode" might not be explicit enough.
>> > > Essentially it
>> > > > > > > seems
>> > > > > > > > > you want
>> > > > > > > > > > the Authorizer to know the Intent/Purpose of the
>> > > authorize call
>> > > > > > > and
>> > > > > > > > > let the
>> > > > > > > > > > Authorizer decide what to log as Audit event.
>> Changing
>> > > the
>> > > > > > > > > class/field name
>> > > > > > > > > > or giving more documentation will do.
>> > > > > > > > > > #3.2 - Regarding the option "OPTIONAL". Are you
>> thinking
>> > > from
>> > > > > > > > > chaining
>> > > > > > > > > > multiple Authorizer? If so, I am not sure whether
>> the
>> > > Broker
>> > > > > > > would
>> > > > > > > > > have
>> > > > > > > > > > enough information to make that decision. I feel the
>> > > Authorizer
>> > > > > > > > will
>> > > > > > > > > be the
>> > > > > > > > > > one who would have that knowledge. E.g. in Ranger
>> we have
>> > > > > > > explicit
>> > > > > > > > > deny,
>> > > > > > > > > > which means no matter what, the request should be
>> denied
>> > > for
>> > > > > > the
>> > > > > > > > > user/group
>> > > > > > > > > > or condition. So if you are thinking of chaining
>> > > Authorizers,
>> > > > > > > then
>> > > > > > > > > > responses should have the third state, e.g.
>> > > "DENIED_FINAL", in
>> > > > > > > > which
>> > > > > > > > > case
>> > > > > > > > > > if there is an Authorization chain, it will be stop
>> and
>> > > the
>> > > > > > > request
>> > > > > > > > > will be
>> > > > > > > > > > denied and if it is just denied, then you might fall
>> > > back to
>> > > > > > next
>> > > > > > > > > > authorizer. If we don't have chaining of
>> Authorizing in
>> > > mind,
>> > > > > > > then
>> > > > > > > > we
>> > > > > > > > > > should reconsider OPTIONAL for now. Or clarify under
>> > > which
>> > > > > > > scenario
>> > > > > > > > > > OPTIONAL will be called.
>> > > > > > > > > > #3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't
>> > > > > > > LIST_AUTHORIZED a
>> > > > > > > > > > special case for "FILTER"?
>> > > > > > > > > > #3.4 KafkaRequestContext. requestType() v/s Action.
>> > > > > > > > > authorizationMode. I
>> > > > > > > > > > am not sure about the overlap or ambiguity.
>> > > > > > > > > > #4 - Cool. This is good, it will be less stress on
>> the
>> > > > > > > Authorizer.
>> > > > > > > > > Ranger
>> > > > > > > > > > already supports the "count" concept and also has
>> > > batching
>> > > > > > > > > capability to
>> > > > > > > > > > aggregate similar requests to reduce the number of
>> audit
>> > > logs
>> > > > > > to
>> > > > > > > > > write. We
>> > > > > > > > > > should be able to pass this through.
>> > > > > > > > > > #5 - Assuming if the object instance is going out of
>> > > scope, we
>> > > > > > > > > should be
>> > > > > > > > > > fine. Not a super important ask. Ranger is already
>> > > catching
>> > > > > > KILL
>> > > > > > > > > signal for
>> > > > > > > > > > clean up.
>> > > > > > > > > >
>> > > > > > > > > > Thanks again. These are good enhancements.
>> Appreciate
>> > > your
>> > > > > > > efforts
>> > > > > > > > > here.
>> > > > > > > > > >
>> > > > > > > > > > Bosco
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On 8/8/19, 2:03 AM, "Rajini Sivaram" <
>> > > rajinisivaram@gmail.com
>> > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > Hi Don,
>> > > > > > > > > >
>> > > > > > > > > > Thanks for reviewing the KIP.
>> > > > > > > > > >
>> > > > > > > > > > 1. I had this originally as a single Action, but
>> > > thought it
>> > > > > > > may
>> > > > > > > > > be
>> > > > > > > > > > useful
>> > > > > > > > > > to support batched authorize calls as well and
>> keep
>> > > it
>> > > > > > > > > consistent with
>> > > > > > > > > > other methods. Single requests can contain
>> multiple
>> > > topics.
>> > > > > > > For
>> > > > > > > > > > example a
>> > > > > > > > > > produce request can contain records for several
>> > > partitions
>> > > > > > of
>> > > > > > > > > different
>> > > > > > > > > > topics. Broker could potentially authorize these
>> > > together.
>> > > > > > > For
>> > > > > > > > > > SimpleAclAuthorizer, batched authorize methods
>> don't
>> > > > > > provide
>> > > > > > > > any
>> > > > > > > > > > optimisation since lookup is based on resources
>> > > followed by
>> > > > > > > the
>> > > > > > > > > > matching
>> > > > > > > > > > logic. But some authorizers may manage ACLs by
>> user
>> > > > > > principal
>> > > > > > > > > rather
>> > > > > > > > > > than
>> > > > > > > > > > resource and may be able to optimize batched
>> > > requests. I am
>> > > > > > > ok
>> > > > > > > > > with
>> > > > > > > > > > using
>> > > > > > > > > > single Action if this is likely to cause issues.
>> > > > > > > > > > 2. If you have two listeners, one for
>> inter-broker
>> > > traffic
>> > > > > > > and
>> > > > > > > > > another
>> > > > > > > > > > for
>> > > > > > > > > > external clients, start method is invoked twice,
>> > > once for
>> > > > > > > each
>> > > > > > > > > > listener. On
>> > > > > > > > > > second thought, that may be confusing and a
>> single
>> > > start()
>> > > > > > > > > invocation
>> > > > > > > > > > that
>> > > > > > > > > > provides all listener information and returns
>> > > multiple
>> > > > > > > futures
>> > > > > > > > > would be
>> > > > > > > > > > better. Will update the KIP.
>> > > > > > > > > > 3. A typical example is a consumer subscribing
>> to a
>> > > regex
>> > > > > > > > > pattern. We
>> > > > > > > > > > request all topic metadata from the broker in
>> order
>> > > to
>> > > > > > decide
>> > > > > > > > > whether
>> > > > > > > > > > the
>> > > > > > > > > > pattern matches, expecting to receive a list of
>> > > authorised
>> > > > > > > > > topics. The
>> > > > > > > > > > user
>> > > > > > > > > > is not asking to subscribe to an unauthorized
>> topic.
>> > > If
>> > > > > > there
>> > > > > > > > > are 10000
>> > > > > > > > > > topics in the cluster and the user has access
>> to 100
>> > > of
>> > > > > > them,
>> > > > > > > > at
>> > > > > > > > > the
>> > > > > > > > > > moment
>> > > > > > > > > > we log 9900 DENIED log entries at INFO level in
>> > > > > > > > > SimpleAclAuthorizer.
>> > > > > > > > > > The
>> > > > > > > > > > proposal is to authorize this request with
>> > > > > > > > > AuthorizationMode.FILTER, so
>> > > > > > > > > > that authorizers can log resources that are
>> filtered
>> > > out at
>> > > > > > > > > lower level
>> > > > > > > > > > like DEBUG since this is not an attempt to
>> access
>> > > > > > > unauthorized
>> > > > > > > > > > resources.
>> > > > > > > > > > Brokers already handle these differently since
>> no
>> > > > > > > authorization
>> > > > > > > > > error
>> > > > > > > > > > is
>> > > > > > > > > > returned to the client in these cases. Providing
>> > > > > > > authorization
>> > > > > > > > > mode to
>> > > > > > > > > > authorizers enables authorizer implementations
>> to
>> > > generate
>> > > > > > > > > better audit
>> > > > > > > > > > logs.
>> > > > > > > > > > 4. Each request may contain multiple instances
>> of
>> > > the same
>> > > > > > > > > authorizable
>> > > > > > > > > > resource. For example a produce request may
>> contain
>> > > records
>> > > > > > > for
>> > > > > > > > > 10
>> > > > > > > > > > partitions of the same topic. At the moment, we
>> > > invoke
>> > > > > > > > authorize
>> > > > > > > > > > method 10
>> > > > > > > > > > times. The proposal is to invoke it once with
>> > > count=10. The
>> > > > > > > > > count is
>> > > > > > > > > > provided to authorizer just for audit logging
>> > > purposes.
>> > > > > > > > > > 5. Authorizer implements Closeable, so you
>> could use
>> > > > > > close()
>> > > > > > > to
>> > > > > > > > > flush
>> > > > > > > > > > audits?
>> > > > > > > > > >
>> > > > > > > > > > On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <
>> > > > > > > > bosco@apache.org
>> > > > > > > > > >
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Rajini
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks for putting this together. It is
>> looking
>> > > good. I
>> > > > > > > have
>> > > > > > > > > few
>> > > > > > > > > > > questions...
>> > > > > > > > > > >
>> > > > > > > > > > > 1. List<AuthorizationResult> authorize(...,
>> > > List<Action>
>> > > > > > > > > actions).
>> > > > > > > > > > Do you
>> > > > > > > > > > > see a scenario where the broker will call
>> > > authorize for
>> > > > > > > > > multiple
>> > > > > > > > > > topics at
>> > > > > > > > > > > the same time? I can understand that during
>> > > > > > > creating/deleting
>> > > > > > > > > ACLS,
>> > > > > > > > > > > multiple permissions for multiple resources
>> might
>> > > be
>> > > > > > done.
>> > > > > > > > For
>> > > > > > > > > > authorize
>> > > > > > > > > > > call, would this be a case? And does the
>> Authorize
>> > > > > > > > > implementation
>> > > > > > > > > > will be
>> > > > > > > > > > > able to do performance optimization because of
>> > > this? Or
>> > > > > > > > should
>> > > > > > > > > we
>> > > > > > > > > > just keep
>> > > > > > > > > > > it simple? I don't see it as an issue from
>> Apache
>> > > Ranger
>> > > > > > > > side,
>> > > > > > > > > but
>> > > > > > > > > > just
>> > > > > > > > > > > checking to see whether we need to be aware of
>> > > something.
>> > > > > > > > > > > 2. Should I assume that the SecurityProtocol
>> passed
>> > > > > > during
>> > > > > > > > > start and
>> > > > > > > > > > the
>> > > > > > > > > > > one return by
>> > > KafkaRequestContext.securityProtocol() will
>> > > > > > > be
>> > > > > > > > > the
>> > > > > > > > > > same?
>> > > > > > > > > > > CompletableFuture<Void> start(String
>> listenerName,
>> > > > > > > > > SecurityProtocol
>> > > > > > > > > > > securityProtocol);
>> > > > > > > > > > > KafkaRequestContext.securityProtocol()
>> > > > > > > > > > > 3. What is the purpose of AuthorizationMode?
>> How
>> > > does the
>> > > > > > > > > broker
>> > > > > > > > > > decide
>> > > > > > > > > > > what mode to use when the authorize() method
>> is
>> > > called?
>> > > > > > > > > > > 4. Can we clarify "count" in Action a bit
>> more?
>> > > How is it
>> > > > > > > > used?
>> > > > > > > > > > > 5. Do you feel having "stop" along with
>> "start" be
>> > > > > > helpful?
>> > > > > > > > > E.g. In
>> > > > > > > > > > Ranger
>> > > > > > > > > > > we try to optimize the Audit writing by
>> caching
>> > > the logs
>> > > > > > > for
>> > > > > > > > a
>> > > > > > > > > fixed
>> > > > > > > > > > > interval. But when the Broker terminates, we
>> do a
>> > > forced
>> > > > > > > > flush.
>> > > > > > > > > > Having an
>> > > > > > > > > > > explicit "stop" might give us a formal way to
>> > > flush our
>> > > > > > > > audits.
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks
>> > > > > > > > > > >
>> > > > > > > > > > > Bosco
>> > > > > > > > > > >
>> > > > > > > > > > > On 8/7/19, 3:59 PM, "Rajini Sivaram" <
>> > > > > > > > rajinisivaram@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > Hi Ron/Harsha/Satish,
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks for reviewing the KIP!
>> > > > > > > > > > >
>> > > > > > > > > > > We should perhaps have a wider discussion
>> > > outside
>> > > > > > this
>> > > > > > > > KIP
>> > > > > > > > > for
>> > > > > > > > > > > refactoring
>> > > > > > > > > > > clients so that others who are not
>> following
>> > > this KIP
>> > > > > > > > also
>> > > > > > > > > > notice the
>> > > > > > > > > > > discussion. Satish, would you like to
>> start a
>> > > > > > > discussion
>> > > > > > > > > thread
>> > > > > > > > > > on dev?
>> > > > > > > > > > >
>> > > > > > > > > > > Regards,
>> > > > > > > > > > >
>> > > > > > > > > > > Rajini
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > On Wed, Aug 7, 2019 at 6:21 PM Satish
>> Duggana <
>> > > > > > > > > > > satish.duggana@gmail.com>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > I felt the same need when we want to
>> add a
>> > > > > > pluggable
>> > > > > > > > API
>> > > > > > > > > for
>> > > > > > > > > > core
>> > > > > > > > > > > > server functionality. This does not
>> need to
>> > > be part
>> > > > > > > of
>> > > > > > > > > this
>> > > > > > > > > > KIP, it
>> > > > > > > > > > > > can be a separate KIP. I can contribute
>> those
>> > > > > > > > refactoring
>> > > > > > > > > > changes if
>> > > > > > > > > > > > others are OK with that.
>> > > > > > > > > > > >
>> > > > > > > > > > > > It is better to have a structure like
>> below.
>> > > > > > > > > > > >
>> > > > > > > > > > > > kafka-common:
>> > > > > > > > > > > > common classes which can be used in any
>> of
>> > > the
>> > > > > > other
>> > > > > > > > > modules
>> > > > > > > > > > in Kafka
>> > > > > > > > > > > > like client, Kafka-server-common and
>> server
>> > > etc.
>> > > > > > > > > > > >
>> > > > > > > > > > > > kafka-client-common:
>> > > > > > > > > > > > common classes which can be used in the
>> > > client
>> > > > > > > module.
>> > > > > > > > > This
>> > > > > > > > > > can be
>> > > > > > > > > > > > part of client module itself.
>> > > > > > > > > > > >
>> > > > > > > > > > > > kafka-server-common:
>> > > > > > > > > > > > classes required only for kafka-server.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks.
>> > > > > > > > > > > > Satish.
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Wed, Aug 7, 2019 at 9:28 PM Harsha
>> > > Chintalapani
>> > > > > > <
>> > > > > > > > > > kafka@harsha.io>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks for the KIP Rajini.
>> > > > > > > > > > > > > Quick thought, it would be good to
>> have a
>> > > common
>> > > > > > > > module
>> > > > > > > > > > outside of
>> > > > > > > > > > > > clients
>> > > > > > > > > > > > > that only applies to server side
>> > > interfaces &
>> > > > > > > > changes.
>> > > > > > > > > It
>> > > > > > > > > > looks
>> > > > > > > > > > > like we
>> > > > > > > > > > > > are
>> > > > > > > > > > > > > increasingly in favor of using Java
>> > > interface for
>> > > > > > > > > pluggable
>> > > > > > > > > > > modules on
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > broker side.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > Harsha
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Tue, Aug 06, 2019 at 2:31 PM,
>> Rajini
>> > > Sivaram <
>> > > > > > > > > > > rajinisivaram@gmail.com
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > Hi all,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > I have created a KIP to replace the
>> Scala
>> > > > > > > > Authorizer
>> > > > > > > > > API
>> > > > > > > > > > with a
>> > > > > > > > > > > new
>> > > > > > > > > > > > Java
>> > > > > > > > > > > > > > API:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > -
>> > > > > > > > > > > > > >
>> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
>> > > > > > > > > > > > > >
>> > > KIP-504+-+Add+new+Java+Authorizer+Interface
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > This is replacement for KIP-50
>> which was
>> > > > > > accepted
>> > > > > > > > > but never
>> > > > > > > > > > > merged.
>> > > > > > > > > > > > Apart
>> > > > > > > > > > > > > > from moving to a Java API consistent
>> > > with other
>> > > > > > > > > pluggable
>> > > > > > > > > > > interfaces
>> > > > > > > > > > > > in the
>> > > > > > > > > > > > > > broker, KIP-504 also attempts to
>> address
>> > > known
>> > > > > > > > > limitations
>> > > > > > > > > > in the
>> > > > > > > > > > > > > > authorizer. If you have come across
>> other
>> > > > > > > > > limitations that
>> > > > > > > > > > you
>> > > > > > > > > > > would
>> > > > > > > > > > > > like
>> > > > > > > > > > > > > > to see addressed in the new API,
>> please
>> > > raise
>> > > > > > > these
>> > > > > > > > > on the
>> > > > > > > > > > > discussion
>> > > > > > > > > > > > > > thread so that we can consider those
>> > > too. All
>> > > > > > > > > suggestions
>> > > > > > > > > > and
>> > > > > > > > > > > feedback
>> > > > > > > > > > > > are
>> > > > > > > > > > > > > > welcome.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Thank you,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Rajini
>> > > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi all,
I made another change to the KIP. The KIP was originally proposing to
extend SimpleAclAuthorizer to also implement the new API (in addition to
the existing API). But since we use the new API when available, this can
break custom authorizers that extend this class and override specific
methods of the old API. To avoid breaking any existing custom
implementations that extend this class, particularly because it is in the
public package kafka.security.auth, the KIP now proposes to retain the old
authorizer as-is. SimpleAclAuthorizer will continue to implement the old
API, but will be deprecated. A new authorizer implementation
kafka.security.authorizer.AclAuthorizer will be added for the new API (this
will not be in the public package).
Please let me know if you have any concerns.
Regards,
Rajini
On Fri, Aug 16, 2019 at 8:48 AM Rajini Sivaram <ra...@gmail.com>
wrote:
> Thanks Colin.
>
> If there are no other concerns, I will start vote later today. Many thanks
> to every one for the feedback.
>
> Regards,
>
> Rajini
>
>
> On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <cm...@apache.org> wrote:
>
>> Thanks, Rajini. It looks good to me.
>>
>> best,
>> Colin
>>
>>
>> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram wrote:
>> > Hi Colin,
>> >
>> > Thanks for the review. I have updated the KIP to move the interfaces for
>> > request context and server info to the authorizer package. These are now
>> > called AuthorizableRequestContext and AuthorizerServerInfo. Endpoint is
>> now
>> > a class in org.apache.kafka.common to make it reusable since we already
>> > have multiple implementations of it. I have removed requestName from the
>> > request context interface since authorizers can distinguish follower
>> fetch
>> > and consumer fetch from the operation being authorized. So 16-bit
>> request
>> > type should be sufficient for audit logging. Also replaced AuditFlag
>> with
>> > two booleans as you suggested.
>> >
>> > Can you take another look and see if the KIP is ready for voting?
>> >
>> > Thanks for all your help!
>> >
>> > Regards,
>> >
>> > Rajini
>> >
>> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <cm...@apache.org>
>> wrote:
>> >
>> > > Hi Rajini,
>> > >
>> > > I think it would be good to rename KafkaRequestContext to something
>> like
>> > > AuthorizableRequestContext, and put it in the
>> > > org.apache.kafka.server.authorizer namespace. If we put it in the
>> > > org.apache.kafka.common namespace, then it's not really clear that
>> it's
>> > > part of the Authorizer API. Since this class is purely an interface,
>> with
>> > > no concrete implementation of anything, there's nothing common to
>> really
>> > > reuse in any case. We definitely don't want someone to accidentally
>> add or
>> > > remove methods because they think this is just another internal class
>> used
>> > > for requests.
>> > >
>> > > The BrokerInfo class is a nice improvement. It looks like it will be
>> > > useful for passing in information about the context we're running
>> in. It
>> > > would be nice to call this class ServerInfo rather than BrokerInfo,
>> since
>> > > we will want to run the authorizer on controllers as well as on
>> brokers,
>> > > and the controller may run as a separate process post KIP-500. I also
>> > > think that this class should be in the
>> org.apache.kafka.server.authorizer
>> > > namespace. Again, it is an interface, not a concrete implementation,
>> and
>> > > it's an interface that is very specifically for the authorizer.
>> > >
>> > > I agree that we probably don't have enough information preserved for
>> > > requests currently to always know what entity made them. So we can
>> leave
>> > > that out for now (except in the special case of Fetch). Perhaps we
>> can add
>> > > this later if it's needed.
>> > >
>> > > I understand the intention behind AuthorizationMode (which is now
>> called
>> > > AuditFlag in the latest revision). But it still feels complex. What
>> if we
>> > > just had two booleans in Action: logSuccesses and logFailures? That
>> seems
>> > > to cover all the cases here. MANDATORY_AUTHORIZE = true, true.
>> > > OPTIONAL_AUTHORIZE = true, false. FILTER = true, false.
>> LIST_AUTHORIZED =
>> > > false, false. Would there be anything lost versus having the enum?
>> > >
>> > > best,
>> > > Colin
>> > >
>> > >
>> > > On Wed, Aug 14, 2019, at 06:29, Mickael Maison wrote:
>> > > > Hi Rajini,
>> > > >
>> > > > Thanks for the KIP!
>> > > > I really like that authorize() will be able to take a batch of
>> > > > requests, this will speed up many implementations!
>> > > >
>> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <
>> rajinisivaram@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > Thanks David! I have fixed the typo.
>> > > > >
>> > > > > Also made a couple of changes to make the context interfaces more
>> > > generic.
>> > > > > KafkaRequestContext now returns the 16-bit API key as Colin
>> suggested
>> > > as
>> > > > > well as the friendly name used in metrics which are useful in
>> audit
>> > > logs.
>> > > > > `Authorizer#start` is now provided a generic `BrokerInfo`
>> interface
>> > > that
>> > > > > gives cluster id, broker id and endpoint information. The generic
>> > > interface
>> > > > > can potentially be used in other broker plugins in future and
>> provides
>> > > > > dynamically generated configs like broker id and ports which are
>> > > currently
>> > > > > not available to plugins unless these configs are statically
>> > > configured.
>> > > > > Please let me know if there are any concerns.
>> > > > >
>> > > > > Regards,
>> > > > >
>> > > > > Rajini
>> > > > >
>> > > > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot <dj...@confluent.io>
>> > > wrote:
>> > > > >
>> > > > > > Hi Rajini,
>> > > > > >
>> > > > > > Thank you for the update! It looks good to me. There is a typo
>> in the
>> > > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` ->
>> `MANDATORY_AUTHORIZE`.
>> > > > > >
>> > > > > > Regards,
>> > > > > > David
>> > > > > >
>> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram <
>> > > rajinisivaram@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi David,
>> > > > > > >
>> > > > > > > Thanks for reviewing the KIP! Since questions about
>> `authorization
>> > > mode`
>> > > > > > > and `count` have come up multiple times, I have renamed both.
>> > > > > > >
>> > > > > > > 1) Renamed `count` to `resourceReferenceCount`. It is the
>> number
>> > > of times
>> > > > > > > the resource being authorized is referenced within the
>> request.
>> > > > > > >
>> > > > > > > 2) Renamed `AuthorizationMode` to `AuditFlag`. It is provided
>> to
>> > > improve
>> > > > > > > audit logging in the authorizer. The enum values have javadoc
>> which
>> > > > > > > indicate how the authorization result is used in each of the
>> modes
>> > > to
>> > > > > > > enable authorizers to log audit messages at the right severity
>> > > level.
>> > > > > > >
>> > > > > > > Regards,
>> > > > > > >
>> > > > > > > Rajini
>> > > > > > >
>> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM David Jacot <
>> djacot@confluent.io>
>> > > wrote:
>> > > > > > >
>> > > > > > > > Hi Rajini,
>> > > > > > > >
>> > > > > > > > Thank you for the KIP. Overall, it looks good to me. I have
>> few
>> > > > > > > > questions/suggestions:
>> > > > > > > >
>> > > > > > > > 1. It is hard to grasp what `Action#count` is for. I guess I
>> > > understand
>> > > > > > > > where you want to go with it but it took me a while to
>> figure it
>> > > out.
>> > > > > > > > Perhaps, we could come up with a better name than `count`?
>> > > > > > > >
>> > > > > > > > 2. I had a hard time trying to understand the
>> `AuthorizationMode`
>> > > > > > > concept,
>> > > > > > > > especially wrt. the OPTIONAL one. My understanding is that
>> an
>> > > ACL is
>> > > > > > > either
>> > > > > > > > defined or not. Could you elaborate a bit more on that?
>> > > > > > > >
>> > > > > > > > Thanks,
>> > > > > > > > David
>> > > > > > > >
>> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai <
>> > > bosco@apache.org>
>> > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Rajini
>> > > > > > > > >
>> > > > > > > > > 3.2 - This makes sense. Thanks for clarifying.
>> > > > > > > > >
>> > > > > > > > > Rest looks fine. Once the implementations are done, it
>> will be
>> > > more
>> > > > > > > clear
>> > > > > > > > > on the different values RequestType and Mode.
>> > > > > > > > >
>> > > > > > > > > Thanks
>> > > > > > > > >
>> > > > > > > > > Bosco
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <
>> rajinisivaram@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > Hi Don,
>> > > > > > > > >
>> > > > > > > > > Thanks for the suggestions. A few responses below:
>> > > > > > > > >
>> > > > > > > > > 3.1 Can rename and improve docs if we keep this. Let's
>> > > finish the
>> > > > > > > > > discussion on Colin's suggestions regarding this
>> first.
>> > > > > > > > > 3.2 No, I was thinking of some requests that have an
>> old
>> > > way of
>> > > > > > > > > authorizing
>> > > > > > > > > and a new way where we have retained the old way for
>> > > backward
>> > > > > > > > > compatibility. One example is Cluster:Create
>> permission to
>> > > create
>> > > > > > > > > topics.
>> > > > > > > > > We have replaced this with fine-grained topic create
>> > > access using
>> > > > > > > > > Topic:Create
>> > > > > > > > > for topic patterns. But we still check if user has
>> > > Cluster:Create
>> > > > > > > > > first. If
>> > > > > > > > > Denied, the deny is ignored and we check
>> Topic:Create. We
>> > > dont
>> > > > > > want
>> > > > > > > > to
>> > > > > > > > > log
>> > > > > > > > > DENY for Cluster:Create at INFO level for this, since
>> this
>> > > is
>> > > > > > not a
>> > > > > > > > > mandatory ACL for creating topics. We will get
>> appropriate
>> > > logs
>> > > > > > > from
>> > > > > > > > > the
>> > > > > > > > > subsequent Topic:Create in this case.
>> > > > > > > > > 3.3 They are not quite the same. FILTER implies that
>> user
>> > > > > > actually
>> > > > > > > > > requested access to and performed some operation on
>> the
>> > > filtered
>> > > > > > > > > resources.
>> > > > > > > > > LIST_AUTHORZED did not result in any actual access.
>> User
>> > > simply
>> > > > > > > > wanted
>> > > > > > > > > to
>> > > > > > > > > know what they are allowed to access.
>> > > > > > > > > 3.4 Request types are Produce, JoinGroup, OffsetCommit
>> > > etc. So
>> > > > > > that
>> > > > > > > > is
>> > > > > > > > > different from authorization mode, operation etc.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <
>> > > > > > bosco@apache.org>
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Hi Rajini
>> > > > > > > > > >
>> > > > > > > > > > Thanks for clarifying. This is very helpful...
>> > > > > > > > > >
>> > > > > > > > > > #1 - On the Ranger side, we should be able to handle
>> > > multiple
>> > > > > > > > > requests at
>> > > > > > > > > > the same time. I was just not sure how much
>> processing
>> > > overhead
>> > > > > > > > will
>> > > > > > > > > be
>> > > > > > > > > > there on the Broker side to split and then
>> consolidate
>> > > the
>> > > > > > > results.
>> > > > > > > > > If it
>> > > > > > > > > > is negligible, then this is the better way. It will
>> > > make it
>> > > > > > > future
>> > > > > > > > > proof.
>> > > > > > > > > > #2 - I agree, having a single "start" call makes it
>> > > cleaner.
>> > > > > > The
>> > > > > > > > > > Authorization implementation will only have to do
>> the
>> > > > > > > > initialization
>> > > > > > > > > only
>> > > > > > > > > > once.
>> > > > > > > > > > #3.1 - Thanks for the clarification. I think it
>> makes
>> > > sense to
>> > > > > > > have
>> > > > > > > > > this.
>> > > > > > > > > > The term "Mode" might not be explicit enough.
>> > > Essentially it
>> > > > > > > seems
>> > > > > > > > > you want
>> > > > > > > > > > the Authorizer to know the Intent/Purpose of the
>> > > authorize call
>> > > > > > > and
>> > > > > > > > > let the
>> > > > > > > > > > Authorizer decide what to log as Audit event.
>> Changing
>> > > the
>> > > > > > > > > class/field name
>> > > > > > > > > > or giving more documentation will do.
>> > > > > > > > > > #3.2 - Regarding the option "OPTIONAL". Are you
>> thinking
>> > > from
>> > > > > > > > > chaining
>> > > > > > > > > > multiple Authorizer? If so, I am not sure whether
>> the
>> > > Broker
>> > > > > > > would
>> > > > > > > > > have
>> > > > > > > > > > enough information to make that decision. I feel the
>> > > Authorizer
>> > > > > > > > will
>> > > > > > > > > be the
>> > > > > > > > > > one who would have that knowledge. E.g. in Ranger
>> we have
>> > > > > > > explicit
>> > > > > > > > > deny,
>> > > > > > > > > > which means no matter what, the request should be
>> denied
>> > > for
>> > > > > > the
>> > > > > > > > > user/group
>> > > > > > > > > > or condition. So if you are thinking of chaining
>> > > Authorizers,
>> > > > > > > then
>> > > > > > > > > > responses should have the third state, e.g.
>> > > "DENIED_FINAL", in
>> > > > > > > > which
>> > > > > > > > > case
>> > > > > > > > > > if there is an Authorization chain, it will be stop
>> and
>> > > the
>> > > > > > > request
>> > > > > > > > > will be
>> > > > > > > > > > denied and if it is just denied, then you might fall
>> > > back to
>> > > > > > next
>> > > > > > > > > > authorizer. If we don't have chaining of
>> Authorizing in
>> > > mind,
>> > > > > > > then
>> > > > > > > > we
>> > > > > > > > > > should reconsider OPTIONAL for now. Or clarify under
>> > > which
>> > > > > > > scenario
>> > > > > > > > > > OPTIONAL will be called.
>> > > > > > > > > > #3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't
>> > > > > > > LIST_AUTHORIZED a
>> > > > > > > > > > special case for "FILTER"?
>> > > > > > > > > > #3.4 KafkaRequestContext. requestType() v/s Action.
>> > > > > > > > > authorizationMode. I
>> > > > > > > > > > am not sure about the overlap or ambiguity.
>> > > > > > > > > > #4 - Cool. This is good, it will be less stress on
>> the
>> > > > > > > Authorizer.
>> > > > > > > > > Ranger
>> > > > > > > > > > already supports the "count" concept and also has
>> > > batching
>> > > > > > > > > capability to
>> > > > > > > > > > aggregate similar requests to reduce the number of
>> audit
>> > > logs
>> > > > > > to
>> > > > > > > > > write. We
>> > > > > > > > > > should be able to pass this through.
>> > > > > > > > > > #5 - Assuming if the object instance is going out of
>> > > scope, we
>> > > > > > > > > should be
>> > > > > > > > > > fine. Not a super important ask. Ranger is already
>> > > catching
>> > > > > > KILL
>> > > > > > > > > signal for
>> > > > > > > > > > clean up.
>> > > > > > > > > >
>> > > > > > > > > > Thanks again. These are good enhancements.
>> Appreciate
>> > > your
>> > > > > > > efforts
>> > > > > > > > > here.
>> > > > > > > > > >
>> > > > > > > > > > Bosco
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > > On 8/8/19, 2:03 AM, "Rajini Sivaram" <
>> > > rajinisivaram@gmail.com
>> > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > Hi Don,
>> > > > > > > > > >
>> > > > > > > > > > Thanks for reviewing the KIP.
>> > > > > > > > > >
>> > > > > > > > > > 1. I had this originally as a single Action, but
>> > > thought it
>> > > > > > > may
>> > > > > > > > > be
>> > > > > > > > > > useful
>> > > > > > > > > > to support batched authorize calls as well and
>> keep
>> > > it
>> > > > > > > > > consistent with
>> > > > > > > > > > other methods. Single requests can contain
>> multiple
>> > > topics.
>> > > > > > > For
>> > > > > > > > > > example a
>> > > > > > > > > > produce request can contain records for several
>> > > partitions
>> > > > > > of
>> > > > > > > > > different
>> > > > > > > > > > topics. Broker could potentially authorize these
>> > > together.
>> > > > > > > For
>> > > > > > > > > > SimpleAclAuthorizer, batched authorize methods
>> don't
>> > > > > > provide
>> > > > > > > > any
>> > > > > > > > > > optimisation since lookup is based on resources
>> > > followed by
>> > > > > > > the
>> > > > > > > > > > matching
>> > > > > > > > > > logic. But some authorizers may manage ACLs by
>> user
>> > > > > > principal
>> > > > > > > > > rather
>> > > > > > > > > > than
>> > > > > > > > > > resource and may be able to optimize batched
>> > > requests. I am
>> > > > > > > ok
>> > > > > > > > > with
>> > > > > > > > > > using
>> > > > > > > > > > single Action if this is likely to cause issues.
>> > > > > > > > > > 2. If you have two listeners, one for
>> inter-broker
>> > > traffic
>> > > > > > > and
>> > > > > > > > > another
>> > > > > > > > > > for
>> > > > > > > > > > external clients, start method is invoked twice,
>> > > once for
>> > > > > > > each
>> > > > > > > > > > listener. On
>> > > > > > > > > > second thought, that may be confusing and a
>> single
>> > > start()
>> > > > > > > > > invocation
>> > > > > > > > > > that
>> > > > > > > > > > provides all listener information and returns
>> > > multiple
>> > > > > > > futures
>> > > > > > > > > would be
>> > > > > > > > > > better. Will update the KIP.
>> > > > > > > > > > 3. A typical example is a consumer subscribing
>> to a
>> > > regex
>> > > > > > > > > pattern. We
>> > > > > > > > > > request all topic metadata from the broker in
>> order
>> > > to
>> > > > > > decide
>> > > > > > > > > whether
>> > > > > > > > > > the
>> > > > > > > > > > pattern matches, expecting to receive a list of
>> > > authorised
>> > > > > > > > > topics. The
>> > > > > > > > > > user
>> > > > > > > > > > is not asking to subscribe to an unauthorized
>> topic.
>> > > If
>> > > > > > there
>> > > > > > > > > are 10000
>> > > > > > > > > > topics in the cluster and the user has access
>> to 100
>> > > of
>> > > > > > them,
>> > > > > > > > at
>> > > > > > > > > the
>> > > > > > > > > > moment
>> > > > > > > > > > we log 9900 DENIED log entries at INFO level in
>> > > > > > > > > SimpleAclAuthorizer.
>> > > > > > > > > > The
>> > > > > > > > > > proposal is to authorize this request with
>> > > > > > > > > AuthorizationMode.FILTER, so
>> > > > > > > > > > that authorizers can log resources that are
>> filtered
>> > > out at
>> > > > > > > > > lower level
>> > > > > > > > > > like DEBUG since this is not an attempt to
>> access
>> > > > > > > unauthorized
>> > > > > > > > > > resources.
>> > > > > > > > > > Brokers already handle these differently since
>> no
>> > > > > > > authorization
>> > > > > > > > > error
>> > > > > > > > > > is
>> > > > > > > > > > returned to the client in these cases. Providing
>> > > > > > > authorization
>> > > > > > > > > mode to
>> > > > > > > > > > authorizers enables authorizer implementations
>> to
>> > > generate
>> > > > > > > > > better audit
>> > > > > > > > > > logs.
>> > > > > > > > > > 4. Each request may contain multiple instances
>> of
>> > > the same
>> > > > > > > > > authorizable
>> > > > > > > > > > resource. For example a produce request may
>> contain
>> > > records
>> > > > > > > for
>> > > > > > > > > 10
>> > > > > > > > > > partitions of the same topic. At the moment, we
>> > > invoke
>> > > > > > > > authorize
>> > > > > > > > > > method 10
>> > > > > > > > > > times. The proposal is to invoke it once with
>> > > count=10. The
>> > > > > > > > > count is
>> > > > > > > > > > provided to authorizer just for audit logging
>> > > purposes.
>> > > > > > > > > > 5. Authorizer implements Closeable, so you
>> could use
>> > > > > > close()
>> > > > > > > to
>> > > > > > > > > flush
>> > > > > > > > > > audits?
>> > > > > > > > > >
>> > > > > > > > > > On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <
>> > > > > > > > bosco@apache.org
>> > > > > > > > > >
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Rajini
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks for putting this together. It is
>> looking
>> > > good. I
>> > > > > > > have
>> > > > > > > > > few
>> > > > > > > > > > > questions...
>> > > > > > > > > > >
>> > > > > > > > > > > 1. List<AuthorizationResult> authorize(...,
>> > > List<Action>
>> > > > > > > > > actions).
>> > > > > > > > > > Do you
>> > > > > > > > > > > see a scenario where the broker will call
>> > > authorize for
>> > > > > > > > > multiple
>> > > > > > > > > > topics at
>> > > > > > > > > > > the same time? I can understand that during
>> > > > > > > creating/deleting
>> > > > > > > > > ACLS,
>> > > > > > > > > > > multiple permissions for multiple resources
>> might
>> > > be
>> > > > > > done.
>> > > > > > > > For
>> > > > > > > > > > authorize
>> > > > > > > > > > > call, would this be a case? And does the
>> Authorize
>> > > > > > > > > implementation
>> > > > > > > > > > will be
>> > > > > > > > > > > able to do performance optimization because of
>> > > this? Or
>> > > > > > > > should
>> > > > > > > > > we
>> > > > > > > > > > just keep
>> > > > > > > > > > > it simple? I don't see it as an issue from
>> Apache
>> > > Ranger
>> > > > > > > > side,
>> > > > > > > > > but
>> > > > > > > > > > just
>> > > > > > > > > > > checking to see whether we need to be aware of
>> > > something.
>> > > > > > > > > > > 2. Should I assume that the SecurityProtocol
>> passed
>> > > > > > during
>> > > > > > > > > start and
>> > > > > > > > > > the
>> > > > > > > > > > > one return by
>> > > KafkaRequestContext.securityProtocol() will
>> > > > > > > be
>> > > > > > > > > the
>> > > > > > > > > > same?
>> > > > > > > > > > > CompletableFuture<Void> start(String
>> listenerName,
>> > > > > > > > > SecurityProtocol
>> > > > > > > > > > > securityProtocol);
>> > > > > > > > > > > KafkaRequestContext.securityProtocol()
>> > > > > > > > > > > 3. What is the purpose of AuthorizationMode?
>> How
>> > > does the
>> > > > > > > > > broker
>> > > > > > > > > > decide
>> > > > > > > > > > > what mode to use when the authorize() method
>> is
>> > > called?
>> > > > > > > > > > > 4. Can we clarify "count" in Action a bit
>> more?
>> > > How is it
>> > > > > > > > used?
>> > > > > > > > > > > 5. Do you feel having "stop" along with
>> "start" be
>> > > > > > helpful?
>> > > > > > > > > E.g. In
>> > > > > > > > > > Ranger
>> > > > > > > > > > > we try to optimize the Audit writing by
>> caching
>> > > the logs
>> > > > > > > for
>> > > > > > > > a
>> > > > > > > > > fixed
>> > > > > > > > > > > interval. But when the Broker terminates, we
>> do a
>> > > forced
>> > > > > > > > flush.
>> > > > > > > > > > Having an
>> > > > > > > > > > > explicit "stop" might give us a formal way to
>> > > flush our
>> > > > > > > > audits.
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks
>> > > > > > > > > > >
>> > > > > > > > > > > Bosco
>> > > > > > > > > > >
>> > > > > > > > > > > On 8/7/19, 3:59 PM, "Rajini Sivaram" <
>> > > > > > > > rajinisivaram@gmail.com
>> > > > > > > > > >
>> > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > Hi Ron/Harsha/Satish,
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks for reviewing the KIP!
>> > > > > > > > > > >
>> > > > > > > > > > > We should perhaps have a wider discussion
>> > > outside
>> > > > > > this
>> > > > > > > > KIP
>> > > > > > > > > for
>> > > > > > > > > > > refactoring
>> > > > > > > > > > > clients so that others who are not
>> following
>> > > this KIP
>> > > > > > > > also
>> > > > > > > > > > notice the
>> > > > > > > > > > > discussion. Satish, would you like to
>> start a
>> > > > > > > discussion
>> > > > > > > > > thread
>> > > > > > > > > > on dev?
>> > > > > > > > > > >
>> > > > > > > > > > > Regards,
>> > > > > > > > > > >
>> > > > > > > > > > > Rajini
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > On Wed, Aug 7, 2019 at 6:21 PM Satish
>> Duggana <
>> > > > > > > > > > > satish.duggana@gmail.com>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > I felt the same need when we want to
>> add a
>> > > > > > pluggable
>> > > > > > > > API
>> > > > > > > > > for
>> > > > > > > > > > core
>> > > > > > > > > > > > server functionality. This does not
>> need to
>> > > be part
>> > > > > > > of
>> > > > > > > > > this
>> > > > > > > > > > KIP, it
>> > > > > > > > > > > > can be a separate KIP. I can contribute
>> those
>> > > > > > > > refactoring
>> > > > > > > > > > changes if
>> > > > > > > > > > > > others are OK with that.
>> > > > > > > > > > > >
>> > > > > > > > > > > > It is better to have a structure like
>> below.
>> > > > > > > > > > > >
>> > > > > > > > > > > > kafka-common:
>> > > > > > > > > > > > common classes which can be used in any
>> of
>> > > the
>> > > > > > other
>> > > > > > > > > modules
>> > > > > > > > > > in Kafka
>> > > > > > > > > > > > like client, Kafka-server-common and
>> server
>> > > etc.
>> > > > > > > > > > > >
>> > > > > > > > > > > > kafka-client-common:
>> > > > > > > > > > > > common classes which can be used in the
>> > > client
>> > > > > > > module.
>> > > > > > > > > This
>> > > > > > > > > > can be
>> > > > > > > > > > > > part of client module itself.
>> > > > > > > > > > > >
>> > > > > > > > > > > > kafka-server-common:
>> > > > > > > > > > > > classes required only for kafka-server.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks.
>> > > > > > > > > > > > Satish.
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Wed, Aug 7, 2019 at 9:28 PM Harsha
>> > > Chintalapani
>> > > > > > <
>> > > > > > > > > > kafka@harsha.io>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks for the KIP Rajini.
>> > > > > > > > > > > > > Quick thought, it would be good to
>> have a
>> > > common
>> > > > > > > > module
>> > > > > > > > > > outside of
>> > > > > > > > > > > > clients
>> > > > > > > > > > > > > that only applies to server side
>> > > interfaces &
>> > > > > > > > changes.
>> > > > > > > > > It
>> > > > > > > > > > looks
>> > > > > > > > > > > like we
>> > > > > > > > > > > > are
>> > > > > > > > > > > > > increasingly in favor of using Java
>> > > interface for
>> > > > > > > > > pluggable
>> > > > > > > > > > > modules on
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > broker side.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > Harsha
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Tue, Aug 06, 2019 at 2:31 PM,
>> Rajini
>> > > Sivaram <
>> > > > > > > > > > > rajinisivaram@gmail.com
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > Hi all,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > I have created a KIP to replace the
>> Scala
>> > > > > > > > Authorizer
>> > > > > > > > > API
>> > > > > > > > > > with a
>> > > > > > > > > > > new
>> > > > > > > > > > > > Java
>> > > > > > > > > > > > > > API:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > -
>> > > > > > > > > > > > > >
>> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
>> > > > > > > > > > > > > >
>> > > KIP-504+-+Add+new+Java+Authorizer+Interface
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > This is replacement for KIP-50
>> which was
>> > > > > > accepted
>> > > > > > > > > but never
>> > > > > > > > > > > merged.
>> > > > > > > > > > > > Apart
>> > > > > > > > > > > > > > from moving to a Java API consistent
>> > > with other
>> > > > > > > > > pluggable
>> > > > > > > > > > > interfaces
>> > > > > > > > > > > > in the
>> > > > > > > > > > > > > > broker, KIP-504 also attempts to
>> address
>> > > known
>> > > > > > > > > limitations
>> > > > > > > > > > in the
>> > > > > > > > > > > > > > authorizer. If you have come across
>> other
>> > > > > > > > > limitations that
>> > > > > > > > > > you
>> > > > > > > > > > > would
>> > > > > > > > > > > > like
>> > > > > > > > > > > > > > to see addressed in the new API,
>> please
>> > > raise
>> > > > > > > these
>> > > > > > > > > on the
>> > > > > > > > > > > discussion
>> > > > > > > > > > > > > > thread so that we can consider those
>> > > too. All
>> > > > > > > > > suggestions
>> > > > > > > > > > and
>> > > > > > > > > > > feedback
>> > > > > > > > > > > > are
>> > > > > > > > > > > > > > welcome.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Thank you,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Rajini
>> > > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > >
>> > >
>> >
>>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Thanks Colin.
If there are no other concerns, I will start vote later today. Many thanks
to every one for the feedback.
Regards,
Rajini
On Thu, Aug 15, 2019 at 11:57 PM Colin McCabe <cm...@apache.org> wrote:
> Thanks, Rajini. It looks good to me.
>
> best,
> Colin
>
>
> On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram wrote:
> > Hi Colin,
> >
> > Thanks for the review. I have updated the KIP to move the interfaces for
> > request context and server info to the authorizer package. These are now
> > called AuthorizableRequestContext and AuthorizerServerInfo. Endpoint is
> now
> > a class in org.apache.kafka.common to make it reusable since we already
> > have multiple implementations of it. I have removed requestName from the
> > request context interface since authorizers can distinguish follower
> fetch
> > and consumer fetch from the operation being authorized. So 16-bit request
> > type should be sufficient for audit logging. Also replaced AuditFlag
> with
> > two booleans as you suggested.
> >
> > Can you take another look and see if the KIP is ready for voting?
> >
> > Thanks for all your help!
> >
> > Regards,
> >
> > Rajini
> >
> > On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <cm...@apache.org> wrote:
> >
> > > Hi Rajini,
> > >
> > > I think it would be good to rename KafkaRequestContext to something
> like
> > > AuthorizableRequestContext, and put it in the
> > > org.apache.kafka.server.authorizer namespace. If we put it in the
> > > org.apache.kafka.common namespace, then it's not really clear that it's
> > > part of the Authorizer API. Since this class is purely an interface,
> with
> > > no concrete implementation of anything, there's nothing common to
> really
> > > reuse in any case. We definitely don't want someone to accidentally
> add or
> > > remove methods because they think this is just another internal class
> used
> > > for requests.
> > >
> > > The BrokerInfo class is a nice improvement. It looks like it will be
> > > useful for passing in information about the context we're running in.
> It
> > > would be nice to call this class ServerInfo rather than BrokerInfo,
> since
> > > we will want to run the authorizer on controllers as well as on
> brokers,
> > > and the controller may run as a separate process post KIP-500. I also
> > > think that this class should be in the
> org.apache.kafka.server.authorizer
> > > namespace. Again, it is an interface, not a concrete implementation,
> and
> > > it's an interface that is very specifically for the authorizer.
> > >
> > > I agree that we probably don't have enough information preserved for
> > > requests currently to always know what entity made them. So we can
> leave
> > > that out for now (except in the special case of Fetch). Perhaps we
> can add
> > > this later if it's needed.
> > >
> > > I understand the intention behind AuthorizationMode (which is now
> called
> > > AuditFlag in the latest revision). But it still feels complex. What
> if we
> > > just had two booleans in Action: logSuccesses and logFailures? That
> seems
> > > to cover all the cases here. MANDATORY_AUTHORIZE = true, true.
> > > OPTIONAL_AUTHORIZE = true, false. FILTER = true, false.
> LIST_AUTHORIZED =
> > > false, false. Would there be anything lost versus having the enum?
> > >
> > > best,
> > > Colin
> > >
> > >
> > > On Wed, Aug 14, 2019, at 06:29, Mickael Maison wrote:
> > > > Hi Rajini,
> > > >
> > > > Thanks for the KIP!
> > > > I really like that authorize() will be able to take a batch of
> > > > requests, this will speed up many implementations!
> > > >
> > > > On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <
> rajinisivaram@gmail.com>
> > > wrote:
> > > > >
> > > > > Thanks David! I have fixed the typo.
> > > > >
> > > > > Also made a couple of changes to make the context interfaces more
> > > generic.
> > > > > KafkaRequestContext now returns the 16-bit API key as Colin
> suggested
> > > as
> > > > > well as the friendly name used in metrics which are useful in audit
> > > logs.
> > > > > `Authorizer#start` is now provided a generic `BrokerInfo` interface
> > > that
> > > > > gives cluster id, broker id and endpoint information. The generic
> > > interface
> > > > > can potentially be used in other broker plugins in future and
> provides
> > > > > dynamically generated configs like broker id and ports which are
> > > currently
> > > > > not available to plugins unless these configs are statically
> > > configured.
> > > > > Please let me know if there are any concerns.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot <dj...@confluent.io>
> > > wrote:
> > > > >
> > > > > > Hi Rajini,
> > > > > >
> > > > > > Thank you for the update! It looks good to me. There is a typo
> in the
> > > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` -> `MANDATORY_AUTHORIZE`.
> > > > > >
> > > > > > Regards,
> > > > > > David
> > > > > >
> > > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram <
> > > rajinisivaram@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi David,
> > > > > > >
> > > > > > > Thanks for reviewing the KIP! Since questions about
> `authorization
> > > mode`
> > > > > > > and `count` have come up multiple times, I have renamed both.
> > > > > > >
> > > > > > > 1) Renamed `count` to `resourceReferenceCount`. It is the
> number
> > > of times
> > > > > > > the resource being authorized is referenced within the request.
> > > > > > >
> > > > > > > 2) Renamed `AuthorizationMode` to `AuditFlag`. It is provided
> to
> > > improve
> > > > > > > audit logging in the authorizer. The enum values have javadoc
> which
> > > > > > > indicate how the authorization result is used in each of the
> modes
> > > to
> > > > > > > enable authorizers to log audit messages at the right severity
> > > level.
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > > On Mon, Aug 12, 2019 at 5:54 PM David Jacot <
> djacot@confluent.io>
> > > wrote:
> > > > > > >
> > > > > > > > Hi Rajini,
> > > > > > > >
> > > > > > > > Thank you for the KIP. Overall, it looks good to me. I have
> few
> > > > > > > > questions/suggestions:
> > > > > > > >
> > > > > > > > 1. It is hard to grasp what `Action#count` is for. I guess I
> > > understand
> > > > > > > > where you want to go with it but it took me a while to
> figure it
> > > out.
> > > > > > > > Perhaps, we could come up with a better name than `count`?
> > > > > > > >
> > > > > > > > 2. I had a hard time trying to understand the
> `AuthorizationMode`
> > > > > > > concept,
> > > > > > > > especially wrt. the OPTIONAL one. My understanding is that an
> > > ACL is
> > > > > > > either
> > > > > > > > defined or not. Could you elaborate a bit more on that?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > David
> > > > > > > >
> > > > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai <
> > > bosco@apache.org>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Rajini
> > > > > > > > >
> > > > > > > > > 3.2 - This makes sense. Thanks for clarifying.
> > > > > > > > >
> > > > > > > > > Rest looks fine. Once the implementations are done, it
> will be
> > > more
> > > > > > > clear
> > > > > > > > > on the different values RequestType and Mode.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > Bosco
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <
> rajinisivaram@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Don,
> > > > > > > > >
> > > > > > > > > Thanks for the suggestions. A few responses below:
> > > > > > > > >
> > > > > > > > > 3.1 Can rename and improve docs if we keep this. Let's
> > > finish the
> > > > > > > > > discussion on Colin's suggestions regarding this first.
> > > > > > > > > 3.2 No, I was thinking of some requests that have an
> old
> > > way of
> > > > > > > > > authorizing
> > > > > > > > > and a new way where we have retained the old way for
> > > backward
> > > > > > > > > compatibility. One example is Cluster:Create
> permission to
> > > create
> > > > > > > > > topics.
> > > > > > > > > We have replaced this with fine-grained topic create
> > > access using
> > > > > > > > > Topic:Create
> > > > > > > > > for topic patterns. But we still check if user has
> > > Cluster:Create
> > > > > > > > > first. If
> > > > > > > > > Denied, the deny is ignored and we check Topic:Create.
> We
> > > dont
> > > > > > want
> > > > > > > > to
> > > > > > > > > log
> > > > > > > > > DENY for Cluster:Create at INFO level for this, since
> this
> > > is
> > > > > > not a
> > > > > > > > > mandatory ACL for creating topics. We will get
> appropriate
> > > logs
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > subsequent Topic:Create in this case.
> > > > > > > > > 3.3 They are not quite the same. FILTER implies that
> user
> > > > > > actually
> > > > > > > > > requested access to and performed some operation on the
> > > filtered
> > > > > > > > > resources.
> > > > > > > > > LIST_AUTHORZED did not result in any actual access.
> User
> > > simply
> > > > > > > > wanted
> > > > > > > > > to
> > > > > > > > > know what they are allowed to access.
> > > > > > > > > 3.4 Request types are Produce, JoinGroup, OffsetCommit
> > > etc. So
> > > > > > that
> > > > > > > > is
> > > > > > > > > different from authorization mode, operation etc.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <
> > > > > > bosco@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Rajini
> > > > > > > > > >
> > > > > > > > > > Thanks for clarifying. This is very helpful...
> > > > > > > > > >
> > > > > > > > > > #1 - On the Ranger side, we should be able to handle
> > > multiple
> > > > > > > > > requests at
> > > > > > > > > > the same time. I was just not sure how much
> processing
> > > overhead
> > > > > > > > will
> > > > > > > > > be
> > > > > > > > > > there on the Broker side to split and then
> consolidate
> > > the
> > > > > > > results.
> > > > > > > > > If it
> > > > > > > > > > is negligible, then this is the better way. It will
> > > make it
> > > > > > > future
> > > > > > > > > proof.
> > > > > > > > > > #2 - I agree, having a single "start" call makes it
> > > cleaner.
> > > > > > The
> > > > > > > > > > Authorization implementation will only have to do the
> > > > > > > > initialization
> > > > > > > > > only
> > > > > > > > > > once.
> > > > > > > > > > #3.1 - Thanks for the clarification. I think it makes
> > > sense to
> > > > > > > have
> > > > > > > > > this.
> > > > > > > > > > The term "Mode" might not be explicit enough.
> > > Essentially it
> > > > > > > seems
> > > > > > > > > you want
> > > > > > > > > > the Authorizer to know the Intent/Purpose of the
> > > authorize call
> > > > > > > and
> > > > > > > > > let the
> > > > > > > > > > Authorizer decide what to log as Audit event.
> Changing
> > > the
> > > > > > > > > class/field name
> > > > > > > > > > or giving more documentation will do.
> > > > > > > > > > #3.2 - Regarding the option "OPTIONAL". Are you
> thinking
> > > from
> > > > > > > > > chaining
> > > > > > > > > > multiple Authorizer? If so, I am not sure whether
> the
> > > Broker
> > > > > > > would
> > > > > > > > > have
> > > > > > > > > > enough information to make that decision. I feel the
> > > Authorizer
> > > > > > > > will
> > > > > > > > > be the
> > > > > > > > > > one who would have that knowledge. E.g. in Ranger we
> have
> > > > > > > explicit
> > > > > > > > > deny,
> > > > > > > > > > which means no matter what, the request should be
> denied
> > > for
> > > > > > the
> > > > > > > > > user/group
> > > > > > > > > > or condition. So if you are thinking of chaining
> > > Authorizers,
> > > > > > > then
> > > > > > > > > > responses should have the third state, e.g.
> > > "DENIED_FINAL", in
> > > > > > > > which
> > > > > > > > > case
> > > > > > > > > > if there is an Authorization chain, it will be stop
> and
> > > the
> > > > > > > request
> > > > > > > > > will be
> > > > > > > > > > denied and if it is just denied, then you might fall
> > > back to
> > > > > > next
> > > > > > > > > > authorizer. If we don't have chaining of Authorizing
> in
> > > mind,
> > > > > > > then
> > > > > > > > we
> > > > > > > > > > should reconsider OPTIONAL for now. Or clarify under
> > > which
> > > > > > > scenario
> > > > > > > > > > OPTIONAL will be called.
> > > > > > > > > > #3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't
> > > > > > > LIST_AUTHORIZED a
> > > > > > > > > > special case for "FILTER"?
> > > > > > > > > > #3.4 KafkaRequestContext. requestType() v/s Action.
> > > > > > > > > authorizationMode. I
> > > > > > > > > > am not sure about the overlap or ambiguity.
> > > > > > > > > > #4 - Cool. This is good, it will be less stress on
> the
> > > > > > > Authorizer.
> > > > > > > > > Ranger
> > > > > > > > > > already supports the "count" concept and also has
> > > batching
> > > > > > > > > capability to
> > > > > > > > > > aggregate similar requests to reduce the number of
> audit
> > > logs
> > > > > > to
> > > > > > > > > write. We
> > > > > > > > > > should be able to pass this through.
> > > > > > > > > > #5 - Assuming if the object instance is going out of
> > > scope, we
> > > > > > > > > should be
> > > > > > > > > > fine. Not a super important ask. Ranger is already
> > > catching
> > > > > > KILL
> > > > > > > > > signal for
> > > > > > > > > > clean up.
> > > > > > > > > >
> > > > > > > > > > Thanks again. These are good enhancements. Appreciate
> > > your
> > > > > > > efforts
> > > > > > > > > here.
> > > > > > > > > >
> > > > > > > > > > Bosco
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 8/8/19, 2:03 AM, "Rajini Sivaram" <
> > > rajinisivaram@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Don,
> > > > > > > > > >
> > > > > > > > > > Thanks for reviewing the KIP.
> > > > > > > > > >
> > > > > > > > > > 1. I had this originally as a single Action, but
> > > thought it
> > > > > > > may
> > > > > > > > > be
> > > > > > > > > > useful
> > > > > > > > > > to support batched authorize calls as well and
> keep
> > > it
> > > > > > > > > consistent with
> > > > > > > > > > other methods. Single requests can contain
> multiple
> > > topics.
> > > > > > > For
> > > > > > > > > > example a
> > > > > > > > > > produce request can contain records for several
> > > partitions
> > > > > > of
> > > > > > > > > different
> > > > > > > > > > topics. Broker could potentially authorize these
> > > together.
> > > > > > > For
> > > > > > > > > > SimpleAclAuthorizer, batched authorize methods
> don't
> > > > > > provide
> > > > > > > > any
> > > > > > > > > > optimisation since lookup is based on resources
> > > followed by
> > > > > > > the
> > > > > > > > > > matching
> > > > > > > > > > logic. But some authorizers may manage ACLs by
> user
> > > > > > principal
> > > > > > > > > rather
> > > > > > > > > > than
> > > > > > > > > > resource and may be able to optimize batched
> > > requests. I am
> > > > > > > ok
> > > > > > > > > with
> > > > > > > > > > using
> > > > > > > > > > single Action if this is likely to cause issues.
> > > > > > > > > > 2. If you have two listeners, one for
> inter-broker
> > > traffic
> > > > > > > and
> > > > > > > > > another
> > > > > > > > > > for
> > > > > > > > > > external clients, start method is invoked twice,
> > > once for
> > > > > > > each
> > > > > > > > > > listener. On
> > > > > > > > > > second thought, that may be confusing and a
> single
> > > start()
> > > > > > > > > invocation
> > > > > > > > > > that
> > > > > > > > > > provides all listener information and returns
> > > multiple
> > > > > > > futures
> > > > > > > > > would be
> > > > > > > > > > better. Will update the KIP.
> > > > > > > > > > 3. A typical example is a consumer subscribing
> to a
> > > regex
> > > > > > > > > pattern. We
> > > > > > > > > > request all topic metadata from the broker in
> order
> > > to
> > > > > > decide
> > > > > > > > > whether
> > > > > > > > > > the
> > > > > > > > > > pattern matches, expecting to receive a list of
> > > authorised
> > > > > > > > > topics. The
> > > > > > > > > > user
> > > > > > > > > > is not asking to subscribe to an unauthorized
> topic.
> > > If
> > > > > > there
> > > > > > > > > are 10000
> > > > > > > > > > topics in the cluster and the user has access to
> 100
> > > of
> > > > > > them,
> > > > > > > > at
> > > > > > > > > the
> > > > > > > > > > moment
> > > > > > > > > > we log 9900 DENIED log entries at INFO level in
> > > > > > > > > SimpleAclAuthorizer.
> > > > > > > > > > The
> > > > > > > > > > proposal is to authorize this request with
> > > > > > > > > AuthorizationMode.FILTER, so
> > > > > > > > > > that authorizers can log resources that are
> filtered
> > > out at
> > > > > > > > > lower level
> > > > > > > > > > like DEBUG since this is not an attempt to access
> > > > > > > unauthorized
> > > > > > > > > > resources.
> > > > > > > > > > Brokers already handle these differently since no
> > > > > > > authorization
> > > > > > > > > error
> > > > > > > > > > is
> > > > > > > > > > returned to the client in these cases. Providing
> > > > > > > authorization
> > > > > > > > > mode to
> > > > > > > > > > authorizers enables authorizer implementations to
> > > generate
> > > > > > > > > better audit
> > > > > > > > > > logs.
> > > > > > > > > > 4. Each request may contain multiple instances of
> > > the same
> > > > > > > > > authorizable
> > > > > > > > > > resource. For example a produce request may
> contain
> > > records
> > > > > > > for
> > > > > > > > > 10
> > > > > > > > > > partitions of the same topic. At the moment, we
> > > invoke
> > > > > > > > authorize
> > > > > > > > > > method 10
> > > > > > > > > > times. The proposal is to invoke it once with
> > > count=10. The
> > > > > > > > > count is
> > > > > > > > > > provided to authorizer just for audit logging
> > > purposes.
> > > > > > > > > > 5. Authorizer implements Closeable, so you could
> use
> > > > > > close()
> > > > > > > to
> > > > > > > > > flush
> > > > > > > > > > audits?
> > > > > > > > > >
> > > > > > > > > > On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <
> > > > > > > > bosco@apache.org
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Rajini
> > > > > > > > > > >
> > > > > > > > > > > Thanks for putting this together. It is looking
> > > good. I
> > > > > > > have
> > > > > > > > > few
> > > > > > > > > > > questions...
> > > > > > > > > > >
> > > > > > > > > > > 1. List<AuthorizationResult> authorize(...,
> > > List<Action>
> > > > > > > > > actions).
> > > > > > > > > > Do you
> > > > > > > > > > > see a scenario where the broker will call
> > > authorize for
> > > > > > > > > multiple
> > > > > > > > > > topics at
> > > > > > > > > > > the same time? I can understand that during
> > > > > > > creating/deleting
> > > > > > > > > ACLS,
> > > > > > > > > > > multiple permissions for multiple resources
> might
> > > be
> > > > > > done.
> > > > > > > > For
> > > > > > > > > > authorize
> > > > > > > > > > > call, would this be a case? And does the
> Authorize
> > > > > > > > > implementation
> > > > > > > > > > will be
> > > > > > > > > > > able to do performance optimization because of
> > > this? Or
> > > > > > > > should
> > > > > > > > > we
> > > > > > > > > > just keep
> > > > > > > > > > > it simple? I don't see it as an issue from
> Apache
> > > Ranger
> > > > > > > > side,
> > > > > > > > > but
> > > > > > > > > > just
> > > > > > > > > > > checking to see whether we need to be aware of
> > > something.
> > > > > > > > > > > 2. Should I assume that the SecurityProtocol
> passed
> > > > > > during
> > > > > > > > > start and
> > > > > > > > > > the
> > > > > > > > > > > one return by
> > > KafkaRequestContext.securityProtocol() will
> > > > > > > be
> > > > > > > > > the
> > > > > > > > > > same?
> > > > > > > > > > > CompletableFuture<Void> start(String
> listenerName,
> > > > > > > > > SecurityProtocol
> > > > > > > > > > > securityProtocol);
> > > > > > > > > > > KafkaRequestContext.securityProtocol()
> > > > > > > > > > > 3. What is the purpose of AuthorizationMode?
> How
> > > does the
> > > > > > > > > broker
> > > > > > > > > > decide
> > > > > > > > > > > what mode to use when the authorize() method is
> > > called?
> > > > > > > > > > > 4. Can we clarify "count" in Action a bit more?
> > > How is it
> > > > > > > > used?
> > > > > > > > > > > 5. Do you feel having "stop" along with
> "start" be
> > > > > > helpful?
> > > > > > > > > E.g. In
> > > > > > > > > > Ranger
> > > > > > > > > > > we try to optimize the Audit writing by caching
> > > the logs
> > > > > > > for
> > > > > > > > a
> > > > > > > > > fixed
> > > > > > > > > > > interval. But when the Broker terminates, we
> do a
> > > forced
> > > > > > > > flush.
> > > > > > > > > > Having an
> > > > > > > > > > > explicit "stop" might give us a formal way to
> > > flush our
> > > > > > > > audits.
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > >
> > > > > > > > > > > Bosco
> > > > > > > > > > >
> > > > > > > > > > > On 8/7/19, 3:59 PM, "Rajini Sivaram" <
> > > > > > > > rajinisivaram@gmail.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hi Ron/Harsha/Satish,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for reviewing the KIP!
> > > > > > > > > > >
> > > > > > > > > > > We should perhaps have a wider discussion
> > > outside
> > > > > > this
> > > > > > > > KIP
> > > > > > > > > for
> > > > > > > > > > > refactoring
> > > > > > > > > > > clients so that others who are not
> following
> > > this KIP
> > > > > > > > also
> > > > > > > > > > notice the
> > > > > > > > > > > discussion. Satish, would you like to
> start a
> > > > > > > discussion
> > > > > > > > > thread
> > > > > > > > > > on dev?
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > >
> > > > > > > > > > > Rajini
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Aug 7, 2019 at 6:21 PM Satish
> Duggana <
> > > > > > > > > > > satish.duggana@gmail.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I felt the same need when we want to add
> a
> > > > > > pluggable
> > > > > > > > API
> > > > > > > > > for
> > > > > > > > > > core
> > > > > > > > > > > > server functionality. This does not need
> to
> > > be part
> > > > > > > of
> > > > > > > > > this
> > > > > > > > > > KIP, it
> > > > > > > > > > > > can be a separate KIP. I can contribute
> those
> > > > > > > > refactoring
> > > > > > > > > > changes if
> > > > > > > > > > > > others are OK with that.
> > > > > > > > > > > >
> > > > > > > > > > > > It is better to have a structure like
> below.
> > > > > > > > > > > >
> > > > > > > > > > > > kafka-common:
> > > > > > > > > > > > common classes which can be used in any
> of
> > > the
> > > > > > other
> > > > > > > > > modules
> > > > > > > > > > in Kafka
> > > > > > > > > > > > like client, Kafka-server-common and
> server
> > > etc.
> > > > > > > > > > > >
> > > > > > > > > > > > kafka-client-common:
> > > > > > > > > > > > common classes which can be used in the
> > > client
> > > > > > > module.
> > > > > > > > > This
> > > > > > > > > > can be
> > > > > > > > > > > > part of client module itself.
> > > > > > > > > > > >
> > > > > > > > > > > > kafka-server-common:
> > > > > > > > > > > > classes required only for kafka-server.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks.
> > > > > > > > > > > > Satish.
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Aug 7, 2019 at 9:28 PM Harsha
> > > Chintalapani
> > > > > > <
> > > > > > > > > > kafka@harsha.io>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for the KIP Rajini.
> > > > > > > > > > > > > Quick thought, it would be good to
> have a
> > > common
> > > > > > > > module
> > > > > > > > > > outside of
> > > > > > > > > > > > clients
> > > > > > > > > > > > > that only applies to server side
> > > interfaces &
> > > > > > > > changes.
> > > > > > > > > It
> > > > > > > > > > looks
> > > > > > > > > > > like we
> > > > > > > > > > > > are
> > > > > > > > > > > > > increasingly in favor of using Java
> > > interface for
> > > > > > > > > pluggable
> > > > > > > > > > > modules on
> > > > > > > > > > > > the
> > > > > > > > > > > > > broker side.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Harsha
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini
> > > Sivaram <
> > > > > > > > > > > rajinisivaram@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have created a KIP to replace the
> Scala
> > > > > > > > Authorizer
> > > > > > > > > API
> > > > > > > > > > with a
> > > > > > > > > > > new
> > > > > > > > > > > > Java
> > > > > > > > > > > > > > API:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > -
> > > > > > > > > > > > > >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > > > > > > > > >
> > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This is replacement for KIP-50 which
> was
> > > > > > accepted
> > > > > > > > > but never
> > > > > > > > > > > merged.
> > > > > > > > > > > > Apart
> > > > > > > > > > > > > > from moving to a Java API consistent
> > > with other
> > > > > > > > > pluggable
> > > > > > > > > > > interfaces
> > > > > > > > > > > > in the
> > > > > > > > > > > > > > broker, KIP-504 also attempts to
> address
> > > known
> > > > > > > > > limitations
> > > > > > > > > > in the
> > > > > > > > > > > > > > authorizer. If you have come across
> other
> > > > > > > > > limitations that
> > > > > > > > > > you
> > > > > > > > > > > would
> > > > > > > > > > > > like
> > > > > > > > > > > > > > to see addressed in the new API,
> please
> > > raise
> > > > > > > these
> > > > > > > > > on the
> > > > > > > > > > > discussion
> > > > > > > > > > > > > > thread so that we can consider those
> > > too. All
> > > > > > > > > suggestions
> > > > > > > > > > and
> > > > > > > > > > > feedback
> > > > > > > > > > > > are
> > > > > > > > > > > > > > welcome.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thank you,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Rajini
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Colin McCabe <cm...@apache.org>.
Thanks, Rajini. It looks good to me.
best,
Colin
On Thu, Aug 15, 2019, at 11:37, Rajini Sivaram wrote:
> Hi Colin,
>
> Thanks for the review. I have updated the KIP to move the interfaces for
> request context and server info to the authorizer package. These are now
> called AuthorizableRequestContext and AuthorizerServerInfo. Endpoint is now
> a class in org.apache.kafka.common to make it reusable since we already
> have multiple implementations of it. I have removed requestName from the
> request context interface since authorizers can distinguish follower fetch
> and consumer fetch from the operation being authorized. So 16-bit request
> type should be sufficient for audit logging. Also replaced AuditFlag with
> two booleans as you suggested.
>
> Can you take another look and see if the KIP is ready for voting?
>
> Thanks for all your help!
>
> Regards,
>
> Rajini
>
> On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <cm...@apache.org> wrote:
>
> > Hi Rajini,
> >
> > I think it would be good to rename KafkaRequestContext to something like
> > AuthorizableRequestContext, and put it in the
> > org.apache.kafka.server.authorizer namespace. If we put it in the
> > org.apache.kafka.common namespace, then it's not really clear that it's
> > part of the Authorizer API. Since this class is purely an interface, with
> > no concrete implementation of anything, there's nothing common to really
> > reuse in any case. We definitely don't want someone to accidentally add or
> > remove methods because they think this is just another internal class used
> > for requests.
> >
> > The BrokerInfo class is a nice improvement. It looks like it will be
> > useful for passing in information about the context we're running in. It
> > would be nice to call this class ServerInfo rather than BrokerInfo, since
> > we will want to run the authorizer on controllers as well as on brokers,
> > and the controller may run as a separate process post KIP-500. I also
> > think that this class should be in the org.apache.kafka.server.authorizer
> > namespace. Again, it is an interface, not a concrete implementation, and
> > it's an interface that is very specifically for the authorizer.
> >
> > I agree that we probably don't have enough information preserved for
> > requests currently to always know what entity made them. So we can leave
> > that out for now (except in the special case of Fetch). Perhaps we can add
> > this later if it's needed.
> >
> > I understand the intention behind AuthorizationMode (which is now called
> > AuditFlag in the latest revision). But it still feels complex. What if we
> > just had two booleans in Action: logSuccesses and logFailures? That seems
> > to cover all the cases here. MANDATORY_AUTHORIZE = true, true.
> > OPTIONAL_AUTHORIZE = true, false. FILTER = true, false. LIST_AUTHORIZED =
> > false, false. Would there be anything lost versus having the enum?
> >
> > best,
> > Colin
> >
> >
> > On Wed, Aug 14, 2019, at 06:29, Mickael Maison wrote:
> > > Hi Rajini,
> > >
> > > Thanks for the KIP!
> > > I really like that authorize() will be able to take a batch of
> > > requests, this will speed up many implementations!
> > >
> > > On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <ra...@gmail.com>
> > wrote:
> > > >
> > > > Thanks David! I have fixed the typo.
> > > >
> > > > Also made a couple of changes to make the context interfaces more
> > generic.
> > > > KafkaRequestContext now returns the 16-bit API key as Colin suggested
> > as
> > > > well as the friendly name used in metrics which are useful in audit
> > logs.
> > > > `Authorizer#start` is now provided a generic `BrokerInfo` interface
> > that
> > > > gives cluster id, broker id and endpoint information. The generic
> > interface
> > > > can potentially be used in other broker plugins in future and provides
> > > > dynamically generated configs like broker id and ports which are
> > currently
> > > > not available to plugins unless these configs are statically
> > configured.
> > > > Please let me know if there are any concerns.
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot <dj...@confluent.io>
> > wrote:
> > > >
> > > > > Hi Rajini,
> > > > >
> > > > > Thank you for the update! It looks good to me. There is a typo in the
> > > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` -> `MANDATORY_AUTHORIZE`.
> > > > >
> > > > > Regards,
> > > > > David
> > > > >
> > > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram <
> > rajinisivaram@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi David,
> > > > > >
> > > > > > Thanks for reviewing the KIP! Since questions about `authorization
> > mode`
> > > > > > and `count` have come up multiple times, I have renamed both.
> > > > > >
> > > > > > 1) Renamed `count` to `resourceReferenceCount`. It is the number
> > of times
> > > > > > the resource being authorized is referenced within the request.
> > > > > >
> > > > > > 2) Renamed `AuthorizationMode` to `AuditFlag`. It is provided to
> > improve
> > > > > > audit logging in the authorizer. The enum values have javadoc which
> > > > > > indicate how the authorization result is used in each of the modes
> > to
> > > > > > enable authorizers to log audit messages at the right severity
> > level.
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > > On Mon, Aug 12, 2019 at 5:54 PM David Jacot <dj...@confluent.io>
> > wrote:
> > > > > >
> > > > > > > Hi Rajini,
> > > > > > >
> > > > > > > Thank you for the KIP. Overall, it looks good to me. I have few
> > > > > > > questions/suggestions:
> > > > > > >
> > > > > > > 1. It is hard to grasp what `Action#count` is for. I guess I
> > understand
> > > > > > > where you want to go with it but it took me a while to figure it
> > out.
> > > > > > > Perhaps, we could come up with a better name than `count`?
> > > > > > >
> > > > > > > 2. I had a hard time trying to understand the `AuthorizationMode`
> > > > > > concept,
> > > > > > > especially wrt. the OPTIONAL one. My understanding is that an
> > ACL is
> > > > > > either
> > > > > > > defined or not. Could you elaborate a bit more on that?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > David
> > > > > > >
> > > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai <
> > bosco@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Rajini
> > > > > > > >
> > > > > > > > 3.2 - This makes sense. Thanks for clarifying.
> > > > > > > >
> > > > > > > > Rest looks fine. Once the implementations are done, it will be
> > more
> > > > > > clear
> > > > > > > > on the different values RequestType and Mode.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > Bosco
> > > > > > > >
> > > > > > > >
> > > > > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <rajinisivaram@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Don,
> > > > > > > >
> > > > > > > > Thanks for the suggestions. A few responses below:
> > > > > > > >
> > > > > > > > 3.1 Can rename and improve docs if we keep this. Let's
> > finish the
> > > > > > > > discussion on Colin's suggestions regarding this first.
> > > > > > > > 3.2 No, I was thinking of some requests that have an old
> > way of
> > > > > > > > authorizing
> > > > > > > > and a new way where we have retained the old way for
> > backward
> > > > > > > > compatibility. One example is Cluster:Create permission to
> > create
> > > > > > > > topics.
> > > > > > > > We have replaced this with fine-grained topic create
> > access using
> > > > > > > > Topic:Create
> > > > > > > > for topic patterns. But we still check if user has
> > Cluster:Create
> > > > > > > > first. If
> > > > > > > > Denied, the deny is ignored and we check Topic:Create. We
> > dont
> > > > > want
> > > > > > > to
> > > > > > > > log
> > > > > > > > DENY for Cluster:Create at INFO level for this, since this
> > is
> > > > > not a
> > > > > > > > mandatory ACL for creating topics. We will get appropriate
> > logs
> > > > > > from
> > > > > > > > the
> > > > > > > > subsequent Topic:Create in this case.
> > > > > > > > 3.3 They are not quite the same. FILTER implies that user
> > > > > actually
> > > > > > > > requested access to and performed some operation on the
> > filtered
> > > > > > > > resources.
> > > > > > > > LIST_AUTHORZED did not result in any actual access. User
> > simply
> > > > > > > wanted
> > > > > > > > to
> > > > > > > > know what they are allowed to access.
> > > > > > > > 3.4 Request types are Produce, JoinGroup, OffsetCommit
> > etc. So
> > > > > that
> > > > > > > is
> > > > > > > > different from authorization mode, operation etc.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <
> > > > > bosco@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Rajini
> > > > > > > > >
> > > > > > > > > Thanks for clarifying. This is very helpful...
> > > > > > > > >
> > > > > > > > > #1 - On the Ranger side, we should be able to handle
> > multiple
> > > > > > > > requests at
> > > > > > > > > the same time. I was just not sure how much processing
> > overhead
> > > > > > > will
> > > > > > > > be
> > > > > > > > > there on the Broker side to split and then consolidate
> > the
> > > > > > results.
> > > > > > > > If it
> > > > > > > > > is negligible, then this is the better way. It will
> > make it
> > > > > > future
> > > > > > > > proof.
> > > > > > > > > #2 - I agree, having a single "start" call makes it
> > cleaner.
> > > > > The
> > > > > > > > > Authorization implementation will only have to do the
> > > > > > > initialization
> > > > > > > > only
> > > > > > > > > once.
> > > > > > > > > #3.1 - Thanks for the clarification. I think it makes
> > sense to
> > > > > > have
> > > > > > > > this.
> > > > > > > > > The term "Mode" might not be explicit enough.
> > Essentially it
> > > > > > seems
> > > > > > > > you want
> > > > > > > > > the Authorizer to know the Intent/Purpose of the
> > authorize call
> > > > > > and
> > > > > > > > let the
> > > > > > > > > Authorizer decide what to log as Audit event. Changing
> > the
> > > > > > > > class/field name
> > > > > > > > > or giving more documentation will do.
> > > > > > > > > #3.2 - Regarding the option "OPTIONAL". Are you thinking
> > from
> > > > > > > > chaining
> > > > > > > > > multiple Authorizer? If so, I am not sure whether the
> > Broker
> > > > > > would
> > > > > > > > have
> > > > > > > > > enough information to make that decision. I feel the
> > Authorizer
> > > > > > > will
> > > > > > > > be the
> > > > > > > > > one who would have that knowledge. E.g. in Ranger we have
> > > > > > explicit
> > > > > > > > deny,
> > > > > > > > > which means no matter what, the request should be denied
> > for
> > > > > the
> > > > > > > > user/group
> > > > > > > > > or condition. So if you are thinking of chaining
> > Authorizers,
> > > > > > then
> > > > > > > > > responses should have the third state, e.g.
> > "DENIED_FINAL", in
> > > > > > > which
> > > > > > > > case
> > > > > > > > > if there is an Authorization chain, it will be stop and
> > the
> > > > > > request
> > > > > > > > will be
> > > > > > > > > denied and if it is just denied, then you might fall
> > back to
> > > > > next
> > > > > > > > > authorizer. If we don't have chaining of Authorizing in
> > mind,
> > > > > > then
> > > > > > > we
> > > > > > > > > should reconsider OPTIONAL for now. Or clarify under
> > which
> > > > > > scenario
> > > > > > > > > OPTIONAL will be called.
> > > > > > > > > #3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't
> > > > > > LIST_AUTHORIZED a
> > > > > > > > > special case for "FILTER"?
> > > > > > > > > #3.4 KafkaRequestContext. requestType() v/s Action.
> > > > > > > > authorizationMode. I
> > > > > > > > > am not sure about the overlap or ambiguity.
> > > > > > > > > #4 - Cool. This is good, it will be less stress on the
> > > > > > Authorizer.
> > > > > > > > Ranger
> > > > > > > > > already supports the "count" concept and also has
> > batching
> > > > > > > > capability to
> > > > > > > > > aggregate similar requests to reduce the number of audit
> > logs
> > > > > to
> > > > > > > > write. We
> > > > > > > > > should be able to pass this through.
> > > > > > > > > #5 - Assuming if the object instance is going out of
> > scope, we
> > > > > > > > should be
> > > > > > > > > fine. Not a super important ask. Ranger is already
> > catching
> > > > > KILL
> > > > > > > > signal for
> > > > > > > > > clean up.
> > > > > > > > >
> > > > > > > > > Thanks again. These are good enhancements. Appreciate
> > your
> > > > > > efforts
> > > > > > > > here.
> > > > > > > > >
> > > > > > > > > Bosco
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On 8/8/19, 2:03 AM, "Rajini Sivaram" <
> > rajinisivaram@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Don,
> > > > > > > > >
> > > > > > > > > Thanks for reviewing the KIP.
> > > > > > > > >
> > > > > > > > > 1. I had this originally as a single Action, but
> > thought it
> > > > > > may
> > > > > > > > be
> > > > > > > > > useful
> > > > > > > > > to support batched authorize calls as well and keep
> > it
> > > > > > > > consistent with
> > > > > > > > > other methods. Single requests can contain multiple
> > topics.
> > > > > > For
> > > > > > > > > example a
> > > > > > > > > produce request can contain records for several
> > partitions
> > > > > of
> > > > > > > > different
> > > > > > > > > topics. Broker could potentially authorize these
> > together.
> > > > > > For
> > > > > > > > > SimpleAclAuthorizer, batched authorize methods don't
> > > > > provide
> > > > > > > any
> > > > > > > > > optimisation since lookup is based on resources
> > followed by
> > > > > > the
> > > > > > > > > matching
> > > > > > > > > logic. But some authorizers may manage ACLs by user
> > > > > principal
> > > > > > > > rather
> > > > > > > > > than
> > > > > > > > > resource and may be able to optimize batched
> > requests. I am
> > > > > > ok
> > > > > > > > with
> > > > > > > > > using
> > > > > > > > > single Action if this is likely to cause issues.
> > > > > > > > > 2. If you have two listeners, one for inter-broker
> > traffic
> > > > > > and
> > > > > > > > another
> > > > > > > > > for
> > > > > > > > > external clients, start method is invoked twice,
> > once for
> > > > > > each
> > > > > > > > > listener. On
> > > > > > > > > second thought, that may be confusing and a single
> > start()
> > > > > > > > invocation
> > > > > > > > > that
> > > > > > > > > provides all listener information and returns
> > multiple
> > > > > > futures
> > > > > > > > would be
> > > > > > > > > better. Will update the KIP.
> > > > > > > > > 3. A typical example is a consumer subscribing to a
> > regex
> > > > > > > > pattern. We
> > > > > > > > > request all topic metadata from the broker in order
> > to
> > > > > decide
> > > > > > > > whether
> > > > > > > > > the
> > > > > > > > > pattern matches, expecting to receive a list of
> > authorised
> > > > > > > > topics. The
> > > > > > > > > user
> > > > > > > > > is not asking to subscribe to an unauthorized topic.
> > If
> > > > > there
> > > > > > > > are 10000
> > > > > > > > > topics in the cluster and the user has access to 100
> > of
> > > > > them,
> > > > > > > at
> > > > > > > > the
> > > > > > > > > moment
> > > > > > > > > we log 9900 DENIED log entries at INFO level in
> > > > > > > > SimpleAclAuthorizer.
> > > > > > > > > The
> > > > > > > > > proposal is to authorize this request with
> > > > > > > > AuthorizationMode.FILTER, so
> > > > > > > > > that authorizers can log resources that are filtered
> > out at
> > > > > > > > lower level
> > > > > > > > > like DEBUG since this is not an attempt to access
> > > > > > unauthorized
> > > > > > > > > resources.
> > > > > > > > > Brokers already handle these differently since no
> > > > > > authorization
> > > > > > > > error
> > > > > > > > > is
> > > > > > > > > returned to the client in these cases. Providing
> > > > > > authorization
> > > > > > > > mode to
> > > > > > > > > authorizers enables authorizer implementations to
> > generate
> > > > > > > > better audit
> > > > > > > > > logs.
> > > > > > > > > 4. Each request may contain multiple instances of
> > the same
> > > > > > > > authorizable
> > > > > > > > > resource. For example a produce request may contain
> > records
> > > > > > for
> > > > > > > > 10
> > > > > > > > > partitions of the same topic. At the moment, we
> > invoke
> > > > > > > authorize
> > > > > > > > > method 10
> > > > > > > > > times. The proposal is to invoke it once with
> > count=10. The
> > > > > > > > count is
> > > > > > > > > provided to authorizer just for audit logging
> > purposes.
> > > > > > > > > 5. Authorizer implements Closeable, so you could use
> > > > > close()
> > > > > > to
> > > > > > > > flush
> > > > > > > > > audits?
> > > > > > > > >
> > > > > > > > > On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <
> > > > > > > bosco@apache.org
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Rajini
> > > > > > > > > >
> > > > > > > > > > Thanks for putting this together. It is looking
> > good. I
> > > > > > have
> > > > > > > > few
> > > > > > > > > > questions...
> > > > > > > > > >
> > > > > > > > > > 1. List<AuthorizationResult> authorize(...,
> > List<Action>
> > > > > > > > actions).
> > > > > > > > > Do you
> > > > > > > > > > see a scenario where the broker will call
> > authorize for
> > > > > > > > multiple
> > > > > > > > > topics at
> > > > > > > > > > the same time? I can understand that during
> > > > > > creating/deleting
> > > > > > > > ACLS,
> > > > > > > > > > multiple permissions for multiple resources might
> > be
> > > > > done.
> > > > > > > For
> > > > > > > > > authorize
> > > > > > > > > > call, would this be a case? And does the Authorize
> > > > > > > > implementation
> > > > > > > > > will be
> > > > > > > > > > able to do performance optimization because of
> > this? Or
> > > > > > > should
> > > > > > > > we
> > > > > > > > > just keep
> > > > > > > > > > it simple? I don't see it as an issue from Apache
> > Ranger
> > > > > > > side,
> > > > > > > > but
> > > > > > > > > just
> > > > > > > > > > checking to see whether we need to be aware of
> > something.
> > > > > > > > > > 2. Should I assume that the SecurityProtocol passed
> > > > > during
> > > > > > > > start and
> > > > > > > > > the
> > > > > > > > > > one return by
> > KafkaRequestContext.securityProtocol() will
> > > > > > be
> > > > > > > > the
> > > > > > > > > same?
> > > > > > > > > > CompletableFuture<Void> start(String listenerName,
> > > > > > > > SecurityProtocol
> > > > > > > > > > securityProtocol);
> > > > > > > > > > KafkaRequestContext.securityProtocol()
> > > > > > > > > > 3. What is the purpose of AuthorizationMode? How
> > does the
> > > > > > > > broker
> > > > > > > > > decide
> > > > > > > > > > what mode to use when the authorize() method is
> > called?
> > > > > > > > > > 4. Can we clarify "count" in Action a bit more?
> > How is it
> > > > > > > used?
> > > > > > > > > > 5. Do you feel having "stop" along with "start" be
> > > > > helpful?
> > > > > > > > E.g. In
> > > > > > > > > Ranger
> > > > > > > > > > we try to optimize the Audit writing by caching
> > the logs
> > > > > > for
> > > > > > > a
> > > > > > > > fixed
> > > > > > > > > > interval. But when the Broker terminates, we do a
> > forced
> > > > > > > flush.
> > > > > > > > > Having an
> > > > > > > > > > explicit "stop" might give us a formal way to
> > flush our
> > > > > > > audits.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > >
> > > > > > > > > > Bosco
> > > > > > > > > >
> > > > > > > > > > On 8/7/19, 3:59 PM, "Rajini Sivaram" <
> > > > > > > rajinisivaram@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hi Ron/Harsha/Satish,
> > > > > > > > > >
> > > > > > > > > > Thanks for reviewing the KIP!
> > > > > > > > > >
> > > > > > > > > > We should perhaps have a wider discussion
> > outside
> > > > > this
> > > > > > > KIP
> > > > > > > > for
> > > > > > > > > > refactoring
> > > > > > > > > > clients so that others who are not following
> > this KIP
> > > > > > > also
> > > > > > > > > notice the
> > > > > > > > > > discussion. Satish, would you like to start a
> > > > > > discussion
> > > > > > > > thread
> > > > > > > > > on dev?
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > >
> > > > > > > > > > Rajini
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <
> > > > > > > > > > satish.duggana@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I felt the same need when we want to add a
> > > > > pluggable
> > > > > > > API
> > > > > > > > for
> > > > > > > > > core
> > > > > > > > > > > server functionality. This does not need to
> > be part
> > > > > > of
> > > > > > > > this
> > > > > > > > > KIP, it
> > > > > > > > > > > can be a separate KIP. I can contribute those
> > > > > > > refactoring
> > > > > > > > > changes if
> > > > > > > > > > > others are OK with that.
> > > > > > > > > > >
> > > > > > > > > > > It is better to have a structure like below.
> > > > > > > > > > >
> > > > > > > > > > > kafka-common:
> > > > > > > > > > > common classes which can be used in any of
> > the
> > > > > other
> > > > > > > > modules
> > > > > > > > > in Kafka
> > > > > > > > > > > like client, Kafka-server-common and server
> > etc.
> > > > > > > > > > >
> > > > > > > > > > > kafka-client-common:
> > > > > > > > > > > common classes which can be used in the
> > client
> > > > > > module.
> > > > > > > > This
> > > > > > > > > can be
> > > > > > > > > > > part of client module itself.
> > > > > > > > > > >
> > > > > > > > > > > kafka-server-common:
> > > > > > > > > > > classes required only for kafka-server.
> > > > > > > > > > >
> > > > > > > > > > > Thanks.
> > > > > > > > > > > Satish.
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Aug 7, 2019 at 9:28 PM Harsha
> > Chintalapani
> > > > > <
> > > > > > > > > kafka@harsha.io>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for the KIP Rajini.
> > > > > > > > > > > > Quick thought, it would be good to have a
> > common
> > > > > > > module
> > > > > > > > > outside of
> > > > > > > > > > > clients
> > > > > > > > > > > > that only applies to server side
> > interfaces &
> > > > > > > changes.
> > > > > > > > It
> > > > > > > > > looks
> > > > > > > > > > like we
> > > > > > > > > > > are
> > > > > > > > > > > > increasingly in favor of using Java
> > interface for
> > > > > > > > pluggable
> > > > > > > > > > modules on
> > > > > > > > > > > the
> > > > > > > > > > > > broker side.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Harsha
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini
> > Sivaram <
> > > > > > > > > > rajinisivaram@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have created a KIP to replace the Scala
> > > > > > > Authorizer
> > > > > > > > API
> > > > > > > > > with a
> > > > > > > > > > new
> > > > > > > > > > > Java
> > > > > > > > > > > > > API:
> > > > > > > > > > > > >
> > > > > > > > > > > > > -
> > > > > > > > > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > > > > > > > >
> > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > > > > > > > > > >
> > > > > > > > > > > > > This is replacement for KIP-50 which was
> > > > > accepted
> > > > > > > > but never
> > > > > > > > > > merged.
> > > > > > > > > > > Apart
> > > > > > > > > > > > > from moving to a Java API consistent
> > with other
> > > > > > > > pluggable
> > > > > > > > > > interfaces
> > > > > > > > > > > in the
> > > > > > > > > > > > > broker, KIP-504 also attempts to address
> > known
> > > > > > > > limitations
> > > > > > > > > in the
> > > > > > > > > > > > > authorizer. If you have come across other
> > > > > > > > limitations that
> > > > > > > > > you
> > > > > > > > > > would
> > > > > > > > > > > like
> > > > > > > > > > > > > to see addressed in the new API, please
> > raise
> > > > > > these
> > > > > > > > on the
> > > > > > > > > > discussion
> > > > > > > > > > > > > thread so that we can consider those
> > too. All
> > > > > > > > suggestions
> > > > > > > > > and
> > > > > > > > > > feedback
> > > > > > > > > > > are
> > > > > > > > > > > > > welcome.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thank you,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Rajini
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Colin,
Thanks for the review. I have updated the KIP to move the interfaces for
request context and server info to the authorizer package. These are now
called AuthorizableRequestContext and AuthorizerServerInfo. Endpoint is now
a class in org.apache.kafka.common to make it reusable since we already
have multiple implementations of it. I have removed requestName from the
request context interface since authorizers can distinguish follower fetch
and consumer fetch from the operation being authorized. So 16-bit request
type should be sufficient for audit logging. Also replaced AuditFlag with
two booleans as you suggested.
Can you take another look and see if the KIP is ready for voting?
Thanks for all your help!
Regards,
Rajini
On Wed, Aug 14, 2019 at 8:59 PM Colin McCabe <cm...@apache.org> wrote:
> Hi Rajini,
>
> I think it would be good to rename KafkaRequestContext to something like
> AuthorizableRequestContext, and put it in the
> org.apache.kafka.server.authorizer namespace. If we put it in the
> org.apache.kafka.common namespace, then it's not really clear that it's
> part of the Authorizer API. Since this class is purely an interface, with
> no concrete implementation of anything, there's nothing common to really
> reuse in any case. We definitely don't want someone to accidentally add or
> remove methods because they think this is just another internal class used
> for requests.
>
> The BrokerInfo class is a nice improvement. It looks like it will be
> useful for passing in information about the context we're running in. It
> would be nice to call this class ServerInfo rather than BrokerInfo, since
> we will want to run the authorizer on controllers as well as on brokers,
> and the controller may run as a separate process post KIP-500. I also
> think that this class should be in the org.apache.kafka.server.authorizer
> namespace. Again, it is an interface, not a concrete implementation, and
> it's an interface that is very specifically for the authorizer.
>
> I agree that we probably don't have enough information preserved for
> requests currently to always know what entity made them. So we can leave
> that out for now (except in the special case of Fetch). Perhaps we can add
> this later if it's needed.
>
> I understand the intention behind AuthorizationMode (which is now called
> AuditFlag in the latest revision). But it still feels complex. What if we
> just had two booleans in Action: logSuccesses and logFailures? That seems
> to cover all the cases here. MANDATORY_AUTHORIZE = true, true.
> OPTIONAL_AUTHORIZE = true, false. FILTER = true, false. LIST_AUTHORIZED =
> false, false. Would there be anything lost versus having the enum?
>
> best,
> Colin
>
>
> On Wed, Aug 14, 2019, at 06:29, Mickael Maison wrote:
> > Hi Rajini,
> >
> > Thanks for the KIP!
> > I really like that authorize() will be able to take a batch of
> > requests, this will speed up many implementations!
> >
> > On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <ra...@gmail.com>
> wrote:
> > >
> > > Thanks David! I have fixed the typo.
> > >
> > > Also made a couple of changes to make the context interfaces more
> generic.
> > > KafkaRequestContext now returns the 16-bit API key as Colin suggested
> as
> > > well as the friendly name used in metrics which are useful in audit
> logs.
> > > `Authorizer#start` is now provided a generic `BrokerInfo` interface
> that
> > > gives cluster id, broker id and endpoint information. The generic
> interface
> > > can potentially be used in other broker plugins in future and provides
> > > dynamically generated configs like broker id and ports which are
> currently
> > > not available to plugins unless these configs are statically
> configured.
> > > Please let me know if there are any concerns.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Tue, Aug 13, 2019 at 4:30 PM David Jacot <dj...@confluent.io>
> wrote:
> > >
> > > > Hi Rajini,
> > > >
> > > > Thank you for the update! It looks good to me. There is a typo in the
> > > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` -> `MANDATORY_AUTHORIZE`.
> > > >
> > > > Regards,
> > > > David
> > > >
> > > > On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram <
> rajinisivaram@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi David,
> > > > >
> > > > > Thanks for reviewing the KIP! Since questions about `authorization
> mode`
> > > > > and `count` have come up multiple times, I have renamed both.
> > > > >
> > > > > 1) Renamed `count` to `resourceReferenceCount`. It is the number
> of times
> > > > > the resource being authorized is referenced within the request.
> > > > >
> > > > > 2) Renamed `AuthorizationMode` to `AuditFlag`. It is provided to
> improve
> > > > > audit logging in the authorizer. The enum values have javadoc which
> > > > > indicate how the authorization result is used in each of the modes
> to
> > > > > enable authorizers to log audit messages at the right severity
> level.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > > > On Mon, Aug 12, 2019 at 5:54 PM David Jacot <dj...@confluent.io>
> wrote:
> > > > >
> > > > > > Hi Rajini,
> > > > > >
> > > > > > Thank you for the KIP. Overall, it looks good to me. I have few
> > > > > > questions/suggestions:
> > > > > >
> > > > > > 1. It is hard to grasp what `Action#count` is for. I guess I
> understand
> > > > > > where you want to go with it but it took me a while to figure it
> out.
> > > > > > Perhaps, we could come up with a better name than `count`?
> > > > > >
> > > > > > 2. I had a hard time trying to understand the `AuthorizationMode`
> > > > > concept,
> > > > > > especially wrt. the OPTIONAL one. My understanding is that an
> ACL is
> > > > > either
> > > > > > defined or not. Could you elaborate a bit more on that?
> > > > > >
> > > > > > Thanks,
> > > > > > David
> > > > > >
> > > > > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai <
> bosco@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Rajini
> > > > > > >
> > > > > > > 3.2 - This makes sense. Thanks for clarifying.
> > > > > > >
> > > > > > > Rest looks fine. Once the implementations are done, it will be
> more
> > > > > clear
> > > > > > > on the different values RequestType and Mode.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > Bosco
> > > > > > >
> > > > > > >
> > > > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <rajinisivaram@gmail.com
> >
> > > > wrote:
> > > > > > >
> > > > > > > Hi Don,
> > > > > > >
> > > > > > > Thanks for the suggestions. A few responses below:
> > > > > > >
> > > > > > > 3.1 Can rename and improve docs if we keep this. Let's
> finish the
> > > > > > > discussion on Colin's suggestions regarding this first.
> > > > > > > 3.2 No, I was thinking of some requests that have an old
> way of
> > > > > > > authorizing
> > > > > > > and a new way where we have retained the old way for
> backward
> > > > > > > compatibility. One example is Cluster:Create permission to
> create
> > > > > > > topics.
> > > > > > > We have replaced this with fine-grained topic create
> access using
> > > > > > > Topic:Create
> > > > > > > for topic patterns. But we still check if user has
> Cluster:Create
> > > > > > > first. If
> > > > > > > Denied, the deny is ignored and we check Topic:Create. We
> dont
> > > > want
> > > > > > to
> > > > > > > log
> > > > > > > DENY for Cluster:Create at INFO level for this, since this
> is
> > > > not a
> > > > > > > mandatory ACL for creating topics. We will get appropriate
> logs
> > > > > from
> > > > > > > the
> > > > > > > subsequent Topic:Create in this case.
> > > > > > > 3.3 They are not quite the same. FILTER implies that user
> > > > actually
> > > > > > > requested access to and performed some operation on the
> filtered
> > > > > > > resources.
> > > > > > > LIST_AUTHORZED did not result in any actual access. User
> simply
> > > > > > wanted
> > > > > > > to
> > > > > > > know what they are allowed to access.
> > > > > > > 3.4 Request types are Produce, JoinGroup, OffsetCommit
> etc. So
> > > > that
> > > > > > is
> > > > > > > different from authorization mode, operation etc.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <
> > > > bosco@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Rajini
> > > > > > > >
> > > > > > > > Thanks for clarifying. This is very helpful...
> > > > > > > >
> > > > > > > > #1 - On the Ranger side, we should be able to handle
> multiple
> > > > > > > requests at
> > > > > > > > the same time. I was just not sure how much processing
> overhead
> > > > > > will
> > > > > > > be
> > > > > > > > there on the Broker side to split and then consolidate
> the
> > > > > results.
> > > > > > > If it
> > > > > > > > is negligible, then this is the better way. It will
> make it
> > > > > future
> > > > > > > proof.
> > > > > > > > #2 - I agree, having a single "start" call makes it
> cleaner.
> > > > The
> > > > > > > > Authorization implementation will only have to do the
> > > > > > initialization
> > > > > > > only
> > > > > > > > once.
> > > > > > > > #3.1 - Thanks for the clarification. I think it makes
> sense to
> > > > > have
> > > > > > > this.
> > > > > > > > The term "Mode" might not be explicit enough.
> Essentially it
> > > > > seems
> > > > > > > you want
> > > > > > > > the Authorizer to know the Intent/Purpose of the
> authorize call
> > > > > and
> > > > > > > let the
> > > > > > > > Authorizer decide what to log as Audit event. Changing
> the
> > > > > > > class/field name
> > > > > > > > or giving more documentation will do.
> > > > > > > > #3.2 - Regarding the option "OPTIONAL". Are you thinking
> from
> > > > > > > chaining
> > > > > > > > multiple Authorizer? If so, I am not sure whether the
> Broker
> > > > > would
> > > > > > > have
> > > > > > > > enough information to make that decision. I feel the
> Authorizer
> > > > > > will
> > > > > > > be the
> > > > > > > > one who would have that knowledge. E.g. in Ranger we have
> > > > > explicit
> > > > > > > deny,
> > > > > > > > which means no matter what, the request should be denied
> for
> > > > the
> > > > > > > user/group
> > > > > > > > or condition. So if you are thinking of chaining
> Authorizers,
> > > > > then
> > > > > > > > responses should have the third state, e.g.
> "DENIED_FINAL", in
> > > > > > which
> > > > > > > case
> > > > > > > > if there is an Authorization chain, it will be stop and
> the
> > > > > request
> > > > > > > will be
> > > > > > > > denied and if it is just denied, then you might fall
> back to
> > > > next
> > > > > > > > authorizer. If we don't have chaining of Authorizing in
> mind,
> > > > > then
> > > > > > we
> > > > > > > > should reconsider OPTIONAL for now. Or clarify under
> which
> > > > > scenario
> > > > > > > > OPTIONAL will be called.
> > > > > > > > #3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't
> > > > > LIST_AUTHORIZED a
> > > > > > > > special case for "FILTER"?
> > > > > > > > #3.4 KafkaRequestContext. requestType() v/s Action.
> > > > > > > authorizationMode. I
> > > > > > > > am not sure about the overlap or ambiguity.
> > > > > > > > #4 - Cool. This is good, it will be less stress on the
> > > > > Authorizer.
> > > > > > > Ranger
> > > > > > > > already supports the "count" concept and also has
> batching
> > > > > > > capability to
> > > > > > > > aggregate similar requests to reduce the number of audit
> logs
> > > > to
> > > > > > > write. We
> > > > > > > > should be able to pass this through.
> > > > > > > > #5 - Assuming if the object instance is going out of
> scope, we
> > > > > > > should be
> > > > > > > > fine. Not a super important ask. Ranger is already
> catching
> > > > KILL
> > > > > > > signal for
> > > > > > > > clean up.
> > > > > > > >
> > > > > > > > Thanks again. These are good enhancements. Appreciate
> your
> > > > > efforts
> > > > > > > here.
> > > > > > > >
> > > > > > > > Bosco
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On 8/8/19, 2:03 AM, "Rajini Sivaram" <
> rajinisivaram@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Don,
> > > > > > > >
> > > > > > > > Thanks for reviewing the KIP.
> > > > > > > >
> > > > > > > > 1. I had this originally as a single Action, but
> thought it
> > > > > may
> > > > > > > be
> > > > > > > > useful
> > > > > > > > to support batched authorize calls as well and keep
> it
> > > > > > > consistent with
> > > > > > > > other methods. Single requests can contain multiple
> topics.
> > > > > For
> > > > > > > > example a
> > > > > > > > produce request can contain records for several
> partitions
> > > > of
> > > > > > > different
> > > > > > > > topics. Broker could potentially authorize these
> together.
> > > > > For
> > > > > > > > SimpleAclAuthorizer, batched authorize methods don't
> > > > provide
> > > > > > any
> > > > > > > > optimisation since lookup is based on resources
> followed by
> > > > > the
> > > > > > > > matching
> > > > > > > > logic. But some authorizers may manage ACLs by user
> > > > principal
> > > > > > > rather
> > > > > > > > than
> > > > > > > > resource and may be able to optimize batched
> requests. I am
> > > > > ok
> > > > > > > with
> > > > > > > > using
> > > > > > > > single Action if this is likely to cause issues.
> > > > > > > > 2. If you have two listeners, one for inter-broker
> traffic
> > > > > and
> > > > > > > another
> > > > > > > > for
> > > > > > > > external clients, start method is invoked twice,
> once for
> > > > > each
> > > > > > > > listener. On
> > > > > > > > second thought, that may be confusing and a single
> start()
> > > > > > > invocation
> > > > > > > > that
> > > > > > > > provides all listener information and returns
> multiple
> > > > > futures
> > > > > > > would be
> > > > > > > > better. Will update the KIP.
> > > > > > > > 3. A typical example is a consumer subscribing to a
> regex
> > > > > > > pattern. We
> > > > > > > > request all topic metadata from the broker in order
> to
> > > > decide
> > > > > > > whether
> > > > > > > > the
> > > > > > > > pattern matches, expecting to receive a list of
> authorised
> > > > > > > topics. The
> > > > > > > > user
> > > > > > > > is not asking to subscribe to an unauthorized topic.
> If
> > > > there
> > > > > > > are 10000
> > > > > > > > topics in the cluster and the user has access to 100
> of
> > > > them,
> > > > > > at
> > > > > > > the
> > > > > > > > moment
> > > > > > > > we log 9900 DENIED log entries at INFO level in
> > > > > > > SimpleAclAuthorizer.
> > > > > > > > The
> > > > > > > > proposal is to authorize this request with
> > > > > > > AuthorizationMode.FILTER, so
> > > > > > > > that authorizers can log resources that are filtered
> out at
> > > > > > > lower level
> > > > > > > > like DEBUG since this is not an attempt to access
> > > > > unauthorized
> > > > > > > > resources.
> > > > > > > > Brokers already handle these differently since no
> > > > > authorization
> > > > > > > error
> > > > > > > > is
> > > > > > > > returned to the client in these cases. Providing
> > > > > authorization
> > > > > > > mode to
> > > > > > > > authorizers enables authorizer implementations to
> generate
> > > > > > > better audit
> > > > > > > > logs.
> > > > > > > > 4. Each request may contain multiple instances of
> the same
> > > > > > > authorizable
> > > > > > > > resource. For example a produce request may contain
> records
> > > > > for
> > > > > > > 10
> > > > > > > > partitions of the same topic. At the moment, we
> invoke
> > > > > > authorize
> > > > > > > > method 10
> > > > > > > > times. The proposal is to invoke it once with
> count=10. The
> > > > > > > count is
> > > > > > > > provided to authorizer just for audit logging
> purposes.
> > > > > > > > 5. Authorizer implements Closeable, so you could use
> > > > close()
> > > > > to
> > > > > > > flush
> > > > > > > > audits?
> > > > > > > >
> > > > > > > > On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <
> > > > > > bosco@apache.org
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Rajini
> > > > > > > > >
> > > > > > > > > Thanks for putting this together. It is looking
> good. I
> > > > > have
> > > > > > > few
> > > > > > > > > questions...
> > > > > > > > >
> > > > > > > > > 1. List<AuthorizationResult> authorize(...,
> List<Action>
> > > > > > > actions).
> > > > > > > > Do you
> > > > > > > > > see a scenario where the broker will call
> authorize for
> > > > > > > multiple
> > > > > > > > topics at
> > > > > > > > > the same time? I can understand that during
> > > > > creating/deleting
> > > > > > > ACLS,
> > > > > > > > > multiple permissions for multiple resources might
> be
> > > > done.
> > > > > > For
> > > > > > > > authorize
> > > > > > > > > call, would this be a case? And does the Authorize
> > > > > > > implementation
> > > > > > > > will be
> > > > > > > > > able to do performance optimization because of
> this? Or
> > > > > > should
> > > > > > > we
> > > > > > > > just keep
> > > > > > > > > it simple? I don't see it as an issue from Apache
> Ranger
> > > > > > side,
> > > > > > > but
> > > > > > > > just
> > > > > > > > > checking to see whether we need to be aware of
> something.
> > > > > > > > > 2. Should I assume that the SecurityProtocol passed
> > > > during
> > > > > > > start and
> > > > > > > > the
> > > > > > > > > one return by
> KafkaRequestContext.securityProtocol() will
> > > > > be
> > > > > > > the
> > > > > > > > same?
> > > > > > > > > CompletableFuture<Void> start(String listenerName,
> > > > > > > SecurityProtocol
> > > > > > > > > securityProtocol);
> > > > > > > > > KafkaRequestContext.securityProtocol()
> > > > > > > > > 3. What is the purpose of AuthorizationMode? How
> does the
> > > > > > > broker
> > > > > > > > decide
> > > > > > > > > what mode to use when the authorize() method is
> called?
> > > > > > > > > 4. Can we clarify "count" in Action a bit more?
> How is it
> > > > > > used?
> > > > > > > > > 5. Do you feel having "stop" along with "start" be
> > > > helpful?
> > > > > > > E.g. In
> > > > > > > > Ranger
> > > > > > > > > we try to optimize the Audit writing by caching
> the logs
> > > > > for
> > > > > > a
> > > > > > > fixed
> > > > > > > > > interval. But when the Broker terminates, we do a
> forced
> > > > > > flush.
> > > > > > > > Having an
> > > > > > > > > explicit "stop" might give us a formal way to
> flush our
> > > > > > audits.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > >
> > > > > > > > > Bosco
> > > > > > > > >
> > > > > > > > > On 8/7/19, 3:59 PM, "Rajini Sivaram" <
> > > > > > rajinisivaram@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi Ron/Harsha/Satish,
> > > > > > > > >
> > > > > > > > > Thanks for reviewing the KIP!
> > > > > > > > >
> > > > > > > > > We should perhaps have a wider discussion
> outside
> > > > this
> > > > > > KIP
> > > > > > > for
> > > > > > > > > refactoring
> > > > > > > > > clients so that others who are not following
> this KIP
> > > > > > also
> > > > > > > > notice the
> > > > > > > > > discussion. Satish, would you like to start a
> > > > > discussion
> > > > > > > thread
> > > > > > > > on dev?
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > >
> > > > > > > > > Rajini
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <
> > > > > > > > > satish.duggana@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I felt the same need when we want to add a
> > > > pluggable
> > > > > > API
> > > > > > > for
> > > > > > > > core
> > > > > > > > > > server functionality. This does not need to
> be part
> > > > > of
> > > > > > > this
> > > > > > > > KIP, it
> > > > > > > > > > can be a separate KIP. I can contribute those
> > > > > > refactoring
> > > > > > > > changes if
> > > > > > > > > > others are OK with that.
> > > > > > > > > >
> > > > > > > > > > It is better to have a structure like below.
> > > > > > > > > >
> > > > > > > > > > kafka-common:
> > > > > > > > > > common classes which can be used in any of
> the
> > > > other
> > > > > > > modules
> > > > > > > > in Kafka
> > > > > > > > > > like client, Kafka-server-common and server
> etc.
> > > > > > > > > >
> > > > > > > > > > kafka-client-common:
> > > > > > > > > > common classes which can be used in the
> client
> > > > > module.
> > > > > > > This
> > > > > > > > can be
> > > > > > > > > > part of client module itself.
> > > > > > > > > >
> > > > > > > > > > kafka-server-common:
> > > > > > > > > > classes required only for kafka-server.
> > > > > > > > > >
> > > > > > > > > > Thanks.
> > > > > > > > > > Satish.
> > > > > > > > > >
> > > > > > > > > > On Wed, Aug 7, 2019 at 9:28 PM Harsha
> Chintalapani
> > > > <
> > > > > > > > kafka@harsha.io>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Thanks for the KIP Rajini.
> > > > > > > > > > > Quick thought, it would be good to have a
> common
> > > > > > module
> > > > > > > > outside of
> > > > > > > > > > clients
> > > > > > > > > > > that only applies to server side
> interfaces &
> > > > > > changes.
> > > > > > > It
> > > > > > > > looks
> > > > > > > > > like we
> > > > > > > > > > are
> > > > > > > > > > > increasingly in favor of using Java
> interface for
> > > > > > > pluggable
> > > > > > > > > modules on
> > > > > > > > > > the
> > > > > > > > > > > broker side.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Harsha
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini
> Sivaram <
> > > > > > > > > rajinisivaram@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I have created a KIP to replace the Scala
> > > > > > Authorizer
> > > > > > > API
> > > > > > > > with a
> > > > > > > > > new
> > > > > > > > > > Java
> > > > > > > > > > > > API:
> > > > > > > > > > > >
> > > > > > > > > > > > -
> > > > > > > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > > > > > > >
> KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > > > > > > > > >
> > > > > > > > > > > > This is replacement for KIP-50 which was
> > > > accepted
> > > > > > > but never
> > > > > > > > > merged.
> > > > > > > > > > Apart
> > > > > > > > > > > > from moving to a Java API consistent
> with other
> > > > > > > pluggable
> > > > > > > > > interfaces
> > > > > > > > > > in the
> > > > > > > > > > > > broker, KIP-504 also attempts to address
> known
> > > > > > > limitations
> > > > > > > > in the
> > > > > > > > > > > > authorizer. If you have come across other
> > > > > > > limitations that
> > > > > > > > you
> > > > > > > > > would
> > > > > > > > > > like
> > > > > > > > > > > > to see addressed in the new API, please
> raise
> > > > > these
> > > > > > > on the
> > > > > > > > > discussion
> > > > > > > > > > > > thread so that we can consider those
> too. All
> > > > > > > suggestions
> > > > > > > > and
> > > > > > > > > feedback
> > > > > > > > > > are
> > > > > > > > > > > > welcome.
> > > > > > > > > > > >
> > > > > > > > > > > > Thank you,
> > > > > > > > > > > >
> > > > > > > > > > > > Rajini
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Colin McCabe <cm...@apache.org>.
Hi Rajini,
I think it would be good to rename KafkaRequestContext to something like AuthorizableRequestContext, and put it in the org.apache.kafka.server.authorizer namespace. If we put it in the org.apache.kafka.common namespace, then it's not really clear that it's part of the Authorizer API. Since this class is purely an interface, with no concrete implementation of anything, there's nothing common to really reuse in any case. We definitely don't want someone to accidentally add or remove methods because they think this is just another internal class used for requests.
The BrokerInfo class is a nice improvement. It looks like it will be useful for passing in information about the context we're running in. It would be nice to call this class ServerInfo rather than BrokerInfo, since we will want to run the authorizer on controllers as well as on brokers, and the controller may run as a separate process post KIP-500. I also think that this class should be in the org.apache.kafka.server.authorizer namespace. Again, it is an interface, not a concrete implementation, and it's an interface that is very specifically for the authorizer.
I agree that we probably don't have enough information preserved for requests currently to always know what entity made them. So we can leave that out for now (except in the special case of Fetch). Perhaps we can add this later if it's needed.
I understand the intention behind AuthorizationMode (which is now called AuditFlag in the latest revision). But it still feels complex. What if we just had two booleans in Action: logSuccesses and logFailures? That seems to cover all the cases here. MANDATORY_AUTHORIZE = true, true. OPTIONAL_AUTHORIZE = true, false. FILTER = true, false. LIST_AUTHORIZED = false, false. Would there be anything lost versus having the enum?
best,
Colin
On Wed, Aug 14, 2019, at 06:29, Mickael Maison wrote:
> Hi Rajini,
>
> Thanks for the KIP!
> I really like that authorize() will be able to take a batch of
> requests, this will speed up many implementations!
>
> On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <ra...@gmail.com> wrote:
> >
> > Thanks David! I have fixed the typo.
> >
> > Also made a couple of changes to make the context interfaces more generic.
> > KafkaRequestContext now returns the 16-bit API key as Colin suggested as
> > well as the friendly name used in metrics which are useful in audit logs.
> > `Authorizer#start` is now provided a generic `BrokerInfo` interface that
> > gives cluster id, broker id and endpoint information. The generic interface
> > can potentially be used in other broker plugins in future and provides
> > dynamically generated configs like broker id and ports which are currently
> > not available to plugins unless these configs are statically configured.
> > Please let me know if there are any concerns.
> >
> > Regards,
> >
> > Rajini
> >
> > On Tue, Aug 13, 2019 at 4:30 PM David Jacot <dj...@confluent.io> wrote:
> >
> > > Hi Rajini,
> > >
> > > Thank you for the update! It looks good to me. There is a typo in the
> > > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` -> `MANDATORY_AUTHORIZE`.
> > >
> > > Regards,
> > > David
> > >
> > > On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram <ra...@gmail.com>
> > > wrote:
> > >
> > > > Hi David,
> > > >
> > > > Thanks for reviewing the KIP! Since questions about `authorization mode`
> > > > and `count` have come up multiple times, I have renamed both.
> > > >
> > > > 1) Renamed `count` to `resourceReferenceCount`. It is the number of times
> > > > the resource being authorized is referenced within the request.
> > > >
> > > > 2) Renamed `AuthorizationMode` to `AuditFlag`. It is provided to improve
> > > > audit logging in the authorizer. The enum values have javadoc which
> > > > indicate how the authorization result is used in each of the modes to
> > > > enable authorizers to log audit messages at the right severity level.
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > > On Mon, Aug 12, 2019 at 5:54 PM David Jacot <dj...@confluent.io> wrote:
> > > >
> > > > > Hi Rajini,
> > > > >
> > > > > Thank you for the KIP. Overall, it looks good to me. I have few
> > > > > questions/suggestions:
> > > > >
> > > > > 1. It is hard to grasp what `Action#count` is for. I guess I understand
> > > > > where you want to go with it but it took me a while to figure it out.
> > > > > Perhaps, we could come up with a better name than `count`?
> > > > >
> > > > > 2. I had a hard time trying to understand the `AuthorizationMode`
> > > > concept,
> > > > > especially wrt. the OPTIONAL one. My understanding is that an ACL is
> > > > either
> > > > > defined or not. Could you elaborate a bit more on that?
> > > > >
> > > > > Thanks,
> > > > > David
> > > > >
> > > > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai <bo...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Hi Rajini
> > > > > >
> > > > > > 3.2 - This makes sense. Thanks for clarifying.
> > > > > >
> > > > > > Rest looks fine. Once the implementations are done, it will be more
> > > > clear
> > > > > > on the different values RequestType and Mode.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Bosco
> > > > > >
> > > > > >
> > > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <ra...@gmail.com>
> > > wrote:
> > > > > >
> > > > > > Hi Don,
> > > > > >
> > > > > > Thanks for the suggestions. A few responses below:
> > > > > >
> > > > > > 3.1 Can rename and improve docs if we keep this. Let's finish the
> > > > > > discussion on Colin's suggestions regarding this first.
> > > > > > 3.2 No, I was thinking of some requests that have an old way of
> > > > > > authorizing
> > > > > > and a new way where we have retained the old way for backward
> > > > > > compatibility. One example is Cluster:Create permission to create
> > > > > > topics.
> > > > > > We have replaced this with fine-grained topic create access using
> > > > > > Topic:Create
> > > > > > for topic patterns. But we still check if user has Cluster:Create
> > > > > > first. If
> > > > > > Denied, the deny is ignored and we check Topic:Create. We dont
> > > want
> > > > > to
> > > > > > log
> > > > > > DENY for Cluster:Create at INFO level for this, since this is
> > > not a
> > > > > > mandatory ACL for creating topics. We will get appropriate logs
> > > > from
> > > > > > the
> > > > > > subsequent Topic:Create in this case.
> > > > > > 3.3 They are not quite the same. FILTER implies that user
> > > actually
> > > > > > requested access to and performed some operation on the filtered
> > > > > > resources.
> > > > > > LIST_AUTHORZED did not result in any actual access. User simply
> > > > > wanted
> > > > > > to
> > > > > > know what they are allowed to access.
> > > > > > 3.4 Request types are Produce, JoinGroup, OffsetCommit etc. So
> > > that
> > > > > is
> > > > > > different from authorization mode, operation etc.
> > > > > >
> > > > > >
> > > > > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <
> > > bosco@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Rajini
> > > > > > >
> > > > > > > Thanks for clarifying. This is very helpful...
> > > > > > >
> > > > > > > #1 - On the Ranger side, we should be able to handle multiple
> > > > > > requests at
> > > > > > > the same time. I was just not sure how much processing overhead
> > > > > will
> > > > > > be
> > > > > > > there on the Broker side to split and then consolidate the
> > > > results.
> > > > > > If it
> > > > > > > is negligible, then this is the better way. It will make it
> > > > future
> > > > > > proof.
> > > > > > > #2 - I agree, having a single "start" call makes it cleaner.
> > > The
> > > > > > > Authorization implementation will only have to do the
> > > > > initialization
> > > > > > only
> > > > > > > once.
> > > > > > > #3.1 - Thanks for the clarification. I think it makes sense to
> > > > have
> > > > > > this.
> > > > > > > The term "Mode" might not be explicit enough. Essentially it
> > > > seems
> > > > > > you want
> > > > > > > the Authorizer to know the Intent/Purpose of the authorize call
> > > > and
> > > > > > let the
> > > > > > > Authorizer decide what to log as Audit event. Changing the
> > > > > > class/field name
> > > > > > > or giving more documentation will do.
> > > > > > > #3.2 - Regarding the option "OPTIONAL". Are you thinking from
> > > > > > chaining
> > > > > > > multiple Authorizer? If so, I am not sure whether the Broker
> > > > would
> > > > > > have
> > > > > > > enough information to make that decision. I feel the Authorizer
> > > > > will
> > > > > > be the
> > > > > > > one who would have that knowledge. E.g. in Ranger we have
> > > > explicit
> > > > > > deny,
> > > > > > > which means no matter what, the request should be denied for
> > > the
> > > > > > user/group
> > > > > > > or condition. So if you are thinking of chaining Authorizers,
> > > > then
> > > > > > > responses should have the third state, e.g. "DENIED_FINAL", in
> > > > > which
> > > > > > case
> > > > > > > if there is an Authorization chain, it will be stop and the
> > > > request
> > > > > > will be
> > > > > > > denied and if it is just denied, then you might fall back to
> > > next
> > > > > > > authorizer. If we don't have chaining of Authorizing in mind,
> > > > then
> > > > > we
> > > > > > > should reconsider OPTIONAL for now. Or clarify under which
> > > > scenario
> > > > > > > OPTIONAL will be called.
> > > > > > > #3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't
> > > > LIST_AUTHORIZED a
> > > > > > > special case for "FILTER"?
> > > > > > > #3.4 KafkaRequestContext. requestType() v/s Action.
> > > > > > authorizationMode. I
> > > > > > > am not sure about the overlap or ambiguity.
> > > > > > > #4 - Cool. This is good, it will be less stress on the
> > > > Authorizer.
> > > > > > Ranger
> > > > > > > already supports the "count" concept and also has batching
> > > > > > capability to
> > > > > > > aggregate similar requests to reduce the number of audit logs
> > > to
> > > > > > write. We
> > > > > > > should be able to pass this through.
> > > > > > > #5 - Assuming if the object instance is going out of scope, we
> > > > > > should be
> > > > > > > fine. Not a super important ask. Ranger is already catching
> > > KILL
> > > > > > signal for
> > > > > > > clean up.
> > > > > > >
> > > > > > > Thanks again. These are good enhancements. Appreciate your
> > > > efforts
> > > > > > here.
> > > > > > >
> > > > > > > Bosco
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 8/8/19, 2:03 AM, "Rajini Sivaram" <rajinisivaram@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Don,
> > > > > > >
> > > > > > > Thanks for reviewing the KIP.
> > > > > > >
> > > > > > > 1. I had this originally as a single Action, but thought it
> > > > may
> > > > > > be
> > > > > > > useful
> > > > > > > to support batched authorize calls as well and keep it
> > > > > > consistent with
> > > > > > > other methods. Single requests can contain multiple topics.
> > > > For
> > > > > > > example a
> > > > > > > produce request can contain records for several partitions
> > > of
> > > > > > different
> > > > > > > topics. Broker could potentially authorize these together.
> > > > For
> > > > > > > SimpleAclAuthorizer, batched authorize methods don't
> > > provide
> > > > > any
> > > > > > > optimisation since lookup is based on resources followed by
> > > > the
> > > > > > > matching
> > > > > > > logic. But some authorizers may manage ACLs by user
> > > principal
> > > > > > rather
> > > > > > > than
> > > > > > > resource and may be able to optimize batched requests. I am
> > > > ok
> > > > > > with
> > > > > > > using
> > > > > > > single Action if this is likely to cause issues.
> > > > > > > 2. If you have two listeners, one for inter-broker traffic
> > > > and
> > > > > > another
> > > > > > > for
> > > > > > > external clients, start method is invoked twice, once for
> > > > each
> > > > > > > listener. On
> > > > > > > second thought, that may be confusing and a single start()
> > > > > > invocation
> > > > > > > that
> > > > > > > provides all listener information and returns multiple
> > > > futures
> > > > > > would be
> > > > > > > better. Will update the KIP.
> > > > > > > 3. A typical example is a consumer subscribing to a regex
> > > > > > pattern. We
> > > > > > > request all topic metadata from the broker in order to
> > > decide
> > > > > > whether
> > > > > > > the
> > > > > > > pattern matches, expecting to receive a list of authorised
> > > > > > topics. The
> > > > > > > user
> > > > > > > is not asking to subscribe to an unauthorized topic. If
> > > there
> > > > > > are 10000
> > > > > > > topics in the cluster and the user has access to 100 of
> > > them,
> > > > > at
> > > > > > the
> > > > > > > moment
> > > > > > > we log 9900 DENIED log entries at INFO level in
> > > > > > SimpleAclAuthorizer.
> > > > > > > The
> > > > > > > proposal is to authorize this request with
> > > > > > AuthorizationMode.FILTER, so
> > > > > > > that authorizers can log resources that are filtered out at
> > > > > > lower level
> > > > > > > like DEBUG since this is not an attempt to access
> > > > unauthorized
> > > > > > > resources.
> > > > > > > Brokers already handle these differently since no
> > > > authorization
> > > > > > error
> > > > > > > is
> > > > > > > returned to the client in these cases. Providing
> > > > authorization
> > > > > > mode to
> > > > > > > authorizers enables authorizer implementations to generate
> > > > > > better audit
> > > > > > > logs.
> > > > > > > 4. Each request may contain multiple instances of the same
> > > > > > authorizable
> > > > > > > resource. For example a produce request may contain records
> > > > for
> > > > > > 10
> > > > > > > partitions of the same topic. At the moment, we invoke
> > > > > authorize
> > > > > > > method 10
> > > > > > > times. The proposal is to invoke it once with count=10. The
> > > > > > count is
> > > > > > > provided to authorizer just for audit logging purposes.
> > > > > > > 5. Authorizer implements Closeable, so you could use
> > > close()
> > > > to
> > > > > > flush
> > > > > > > audits?
> > > > > > >
> > > > > > > On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <
> > > > > bosco@apache.org
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Rajini
> > > > > > > >
> > > > > > > > Thanks for putting this together. It is looking good. I
> > > > have
> > > > > > few
> > > > > > > > questions...
> > > > > > > >
> > > > > > > > 1. List<AuthorizationResult> authorize(..., List<Action>
> > > > > > actions).
> > > > > > > Do you
> > > > > > > > see a scenario where the broker will call authorize for
> > > > > > multiple
> > > > > > > topics at
> > > > > > > > the same time? I can understand that during
> > > > creating/deleting
> > > > > > ACLS,
> > > > > > > > multiple permissions for multiple resources might be
> > > done.
> > > > > For
> > > > > > > authorize
> > > > > > > > call, would this be a case? And does the Authorize
> > > > > > implementation
> > > > > > > will be
> > > > > > > > able to do performance optimization because of this? Or
> > > > > should
> > > > > > we
> > > > > > > just keep
> > > > > > > > it simple? I don't see it as an issue from Apache Ranger
> > > > > side,
> > > > > > but
> > > > > > > just
> > > > > > > > checking to see whether we need to be aware of something.
> > > > > > > > 2. Should I assume that the SecurityProtocol passed
> > > during
> > > > > > start and
> > > > > > > the
> > > > > > > > one return by KafkaRequestContext.securityProtocol() will
> > > > be
> > > > > > the
> > > > > > > same?
> > > > > > > > CompletableFuture<Void> start(String listenerName,
> > > > > > SecurityProtocol
> > > > > > > > securityProtocol);
> > > > > > > > KafkaRequestContext.securityProtocol()
> > > > > > > > 3. What is the purpose of AuthorizationMode? How does the
> > > > > > broker
> > > > > > > decide
> > > > > > > > what mode to use when the authorize() method is called?
> > > > > > > > 4. Can we clarify "count" in Action a bit more? How is it
> > > > > used?
> > > > > > > > 5. Do you feel having "stop" along with "start" be
> > > helpful?
> > > > > > E.g. In
> > > > > > > Ranger
> > > > > > > > we try to optimize the Audit writing by caching the logs
> > > > for
> > > > > a
> > > > > > fixed
> > > > > > > > interval. But when the Broker terminates, we do a forced
> > > > > flush.
> > > > > > > Having an
> > > > > > > > explicit "stop" might give us a formal way to flush our
> > > > > audits.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > > Bosco
> > > > > > > >
> > > > > > > > On 8/7/19, 3:59 PM, "Rajini Sivaram" <
> > > > > rajinisivaram@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Ron/Harsha/Satish,
> > > > > > > >
> > > > > > > > Thanks for reviewing the KIP!
> > > > > > > >
> > > > > > > > We should perhaps have a wider discussion outside
> > > this
> > > > > KIP
> > > > > > for
> > > > > > > > refactoring
> > > > > > > > clients so that others who are not following this KIP
> > > > > also
> > > > > > > notice the
> > > > > > > > discussion. Satish, would you like to start a
> > > > discussion
> > > > > > thread
> > > > > > > on dev?
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Rajini
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <
> > > > > > > > satish.duggana@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I felt the same need when we want to add a
> > > pluggable
> > > > > API
> > > > > > for
> > > > > > > core
> > > > > > > > > server functionality. This does not need to be part
> > > > of
> > > > > > this
> > > > > > > KIP, it
> > > > > > > > > can be a separate KIP. I can contribute those
> > > > > refactoring
> > > > > > > changes if
> > > > > > > > > others are OK with that.
> > > > > > > > >
> > > > > > > > > It is better to have a structure like below.
> > > > > > > > >
> > > > > > > > > kafka-common:
> > > > > > > > > common classes which can be used in any of the
> > > other
> > > > > > modules
> > > > > > > in Kafka
> > > > > > > > > like client, Kafka-server-common and server etc.
> > > > > > > > >
> > > > > > > > > kafka-client-common:
> > > > > > > > > common classes which can be used in the client
> > > > module.
> > > > > > This
> > > > > > > can be
> > > > > > > > > part of client module itself.
> > > > > > > > >
> > > > > > > > > kafka-server-common:
> > > > > > > > > classes required only for kafka-server.
> > > > > > > > >
> > > > > > > > > Thanks.
> > > > > > > > > Satish.
> > > > > > > > >
> > > > > > > > > On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani
> > > <
> > > > > > > kafka@harsha.io>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Thanks for the KIP Rajini.
> > > > > > > > > > Quick thought, it would be good to have a common
> > > > > module
> > > > > > > outside of
> > > > > > > > > clients
> > > > > > > > > > that only applies to server side interfaces &
> > > > > changes.
> > > > > > It
> > > > > > > looks
> > > > > > > > like we
> > > > > > > > > are
> > > > > > > > > > increasingly in favor of using Java interface for
> > > > > > pluggable
> > > > > > > > modules on
> > > > > > > > > the
> > > > > > > > > > broker side.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Harsha
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <
> > > > > > > > rajinisivaram@gmail.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I have created a KIP to replace the Scala
> > > > > Authorizer
> > > > > > API
> > > > > > > with a
> > > > > > > > new
> > > > > > > > > Java
> > > > > > > > > > > API:
> > > > > > > > > > >
> > > > > > > > > > > -
> > > > > > > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > > > > > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > > > > > > > >
> > > > > > > > > > > This is replacement for KIP-50 which was
> > > accepted
> > > > > > but never
> > > > > > > > merged.
> > > > > > > > > Apart
> > > > > > > > > > > from moving to a Java API consistent with other
> > > > > > pluggable
> > > > > > > > interfaces
> > > > > > > > > in the
> > > > > > > > > > > broker, KIP-504 also attempts to address known
> > > > > > limitations
> > > > > > > in the
> > > > > > > > > > > authorizer. If you have come across other
> > > > > > limitations that
> > > > > > > you
> > > > > > > > would
> > > > > > > > > like
> > > > > > > > > > > to see addressed in the new API, please raise
> > > > these
> > > > > > on the
> > > > > > > > discussion
> > > > > > > > > > > thread so that we can consider those too. All
> > > > > > suggestions
> > > > > > > and
> > > > > > > > feedback
> > > > > > > > > are
> > > > > > > > > > > welcome.
> > > > > > > > > > >
> > > > > > > > > > > Thank you,
> > > > > > > > > > >
> > > > > > > > > > > Rajini
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Mickael Maison <mi...@gmail.com>.
Hi Rajini,
Thanks for the KIP!
I really like that authorize() will be able to take a batch of
requests, this will speed up many implementations!
On Tue, Aug 13, 2019 at 5:57 PM Rajini Sivaram <ra...@gmail.com> wrote:
>
> Thanks David! I have fixed the typo.
>
> Also made a couple of changes to make the context interfaces more generic.
> KafkaRequestContext now returns the 16-bit API key as Colin suggested as
> well as the friendly name used in metrics which are useful in audit logs.
> `Authorizer#start` is now provided a generic `BrokerInfo` interface that
> gives cluster id, broker id and endpoint information. The generic interface
> can potentially be used in other broker plugins in future and provides
> dynamically generated configs like broker id and ports which are currently
> not available to plugins unless these configs are statically configured.
> Please let me know if there are any concerns.
>
> Regards,
>
> Rajini
>
> On Tue, Aug 13, 2019 at 4:30 PM David Jacot <dj...@confluent.io> wrote:
>
> > Hi Rajini,
> >
> > Thank you for the update! It looks good to me. There is a typo in the
> > `AuditFlag` enum: `MANDATORY_AUTHOEIZE` -> `MANDATORY_AUTHORIZE`.
> >
> > Regards,
> > David
> >
> > On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram <ra...@gmail.com>
> > wrote:
> >
> > > Hi David,
> > >
> > > Thanks for reviewing the KIP! Since questions about `authorization mode`
> > > and `count` have come up multiple times, I have renamed both.
> > >
> > > 1) Renamed `count` to `resourceReferenceCount`. It is the number of times
> > > the resource being authorized is referenced within the request.
> > >
> > > 2) Renamed `AuthorizationMode` to `AuditFlag`. It is provided to improve
> > > audit logging in the authorizer. The enum values have javadoc which
> > > indicate how the authorization result is used in each of the modes to
> > > enable authorizers to log audit messages at the right severity level.
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > > On Mon, Aug 12, 2019 at 5:54 PM David Jacot <dj...@confluent.io> wrote:
> > >
> > > > Hi Rajini,
> > > >
> > > > Thank you for the KIP. Overall, it looks good to me. I have few
> > > > questions/suggestions:
> > > >
> > > > 1. It is hard to grasp what `Action#count` is for. I guess I understand
> > > > where you want to go with it but it took me a while to figure it out.
> > > > Perhaps, we could come up with a better name than `count`?
> > > >
> > > > 2. I had a hard time trying to understand the `AuthorizationMode`
> > > concept,
> > > > especially wrt. the OPTIONAL one. My understanding is that an ACL is
> > > either
> > > > defined or not. Could you elaborate a bit more on that?
> > > >
> > > > Thanks,
> > > > David
> > > >
> > > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai <bo...@apache.org>
> > > wrote:
> > > >
> > > > > Hi Rajini
> > > > >
> > > > > 3.2 - This makes sense. Thanks for clarifying.
> > > > >
> > > > > Rest looks fine. Once the implementations are done, it will be more
> > > clear
> > > > > on the different values RequestType and Mode.
> > > > >
> > > > > Thanks
> > > > >
> > > > > Bosco
> > > > >
> > > > >
> > > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <ra...@gmail.com>
> > wrote:
> > > > >
> > > > > Hi Don,
> > > > >
> > > > > Thanks for the suggestions. A few responses below:
> > > > >
> > > > > 3.1 Can rename and improve docs if we keep this. Let's finish the
> > > > > discussion on Colin's suggestions regarding this first.
> > > > > 3.2 No, I was thinking of some requests that have an old way of
> > > > > authorizing
> > > > > and a new way where we have retained the old way for backward
> > > > > compatibility. One example is Cluster:Create permission to create
> > > > > topics.
> > > > > We have replaced this with fine-grained topic create access using
> > > > > Topic:Create
> > > > > for topic patterns. But we still check if user has Cluster:Create
> > > > > first. If
> > > > > Denied, the deny is ignored and we check Topic:Create. We dont
> > want
> > > > to
> > > > > log
> > > > > DENY for Cluster:Create at INFO level for this, since this is
> > not a
> > > > > mandatory ACL for creating topics. We will get appropriate logs
> > > from
> > > > > the
> > > > > subsequent Topic:Create in this case.
> > > > > 3.3 They are not quite the same. FILTER implies that user
> > actually
> > > > > requested access to and performed some operation on the filtered
> > > > > resources.
> > > > > LIST_AUTHORZED did not result in any actual access. User simply
> > > > wanted
> > > > > to
> > > > > know what they are allowed to access.
> > > > > 3.4 Request types are Produce, JoinGroup, OffsetCommit etc. So
> > that
> > > > is
> > > > > different from authorization mode, operation etc.
> > > > >
> > > > >
> > > > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <
> > bosco@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hi Rajini
> > > > > >
> > > > > > Thanks for clarifying. This is very helpful...
> > > > > >
> > > > > > #1 - On the Ranger side, we should be able to handle multiple
> > > > > requests at
> > > > > > the same time. I was just not sure how much processing overhead
> > > > will
> > > > > be
> > > > > > there on the Broker side to split and then consolidate the
> > > results.
> > > > > If it
> > > > > > is negligible, then this is the better way. It will make it
> > > future
> > > > > proof.
> > > > > > #2 - I agree, having a single "start" call makes it cleaner.
> > The
> > > > > > Authorization implementation will only have to do the
> > > > initialization
> > > > > only
> > > > > > once.
> > > > > > #3.1 - Thanks for the clarification. I think it makes sense to
> > > have
> > > > > this.
> > > > > > The term "Mode" might not be explicit enough. Essentially it
> > > seems
> > > > > you want
> > > > > > the Authorizer to know the Intent/Purpose of the authorize call
> > > and
> > > > > let the
> > > > > > Authorizer decide what to log as Audit event. Changing the
> > > > > class/field name
> > > > > > or giving more documentation will do.
> > > > > > #3.2 - Regarding the option "OPTIONAL". Are you thinking from
> > > > > chaining
> > > > > > multiple Authorizer? If so, I am not sure whether the Broker
> > > would
> > > > > have
> > > > > > enough information to make that decision. I feel the Authorizer
> > > > will
> > > > > be the
> > > > > > one who would have that knowledge. E.g. in Ranger we have
> > > explicit
> > > > > deny,
> > > > > > which means no matter what, the request should be denied for
> > the
> > > > > user/group
> > > > > > or condition. So if you are thinking of chaining Authorizers,
> > > then
> > > > > > responses should have the third state, e.g. "DENIED_FINAL", in
> > > > which
> > > > > case
> > > > > > if there is an Authorization chain, it will be stop and the
> > > request
> > > > > will be
> > > > > > denied and if it is just denied, then you might fall back to
> > next
> > > > > > authorizer. If we don't have chaining of Authorizing in mind,
> > > then
> > > > we
> > > > > > should reconsider OPTIONAL for now. Or clarify under which
> > > scenario
> > > > > > OPTIONAL will be called.
> > > > > > #3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't
> > > LIST_AUTHORIZED a
> > > > > > special case for "FILTER"?
> > > > > > #3.4 KafkaRequestContext. requestType() v/s Action.
> > > > > authorizationMode. I
> > > > > > am not sure about the overlap or ambiguity.
> > > > > > #4 - Cool. This is good, it will be less stress on the
> > > Authorizer.
> > > > > Ranger
> > > > > > already supports the "count" concept and also has batching
> > > > > capability to
> > > > > > aggregate similar requests to reduce the number of audit logs
> > to
> > > > > write. We
> > > > > > should be able to pass this through.
> > > > > > #5 - Assuming if the object instance is going out of scope, we
> > > > > should be
> > > > > > fine. Not a super important ask. Ranger is already catching
> > KILL
> > > > > signal for
> > > > > > clean up.
> > > > > >
> > > > > > Thanks again. These are good enhancements. Appreciate your
> > > efforts
> > > > > here.
> > > > > >
> > > > > > Bosco
> > > > > >
> > > > > >
> > > > > >
> > > > > > On 8/8/19, 2:03 AM, "Rajini Sivaram" <rajinisivaram@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Hi Don,
> > > > > >
> > > > > > Thanks for reviewing the KIP.
> > > > > >
> > > > > > 1. I had this originally as a single Action, but thought it
> > > may
> > > > > be
> > > > > > useful
> > > > > > to support batched authorize calls as well and keep it
> > > > > consistent with
> > > > > > other methods. Single requests can contain multiple topics.
> > > For
> > > > > > example a
> > > > > > produce request can contain records for several partitions
> > of
> > > > > different
> > > > > > topics. Broker could potentially authorize these together.
> > > For
> > > > > > SimpleAclAuthorizer, batched authorize methods don't
> > provide
> > > > any
> > > > > > optimisation since lookup is based on resources followed by
> > > the
> > > > > > matching
> > > > > > logic. But some authorizers may manage ACLs by user
> > principal
> > > > > rather
> > > > > > than
> > > > > > resource and may be able to optimize batched requests. I am
> > > ok
> > > > > with
> > > > > > using
> > > > > > single Action if this is likely to cause issues.
> > > > > > 2. If you have two listeners, one for inter-broker traffic
> > > and
> > > > > another
> > > > > > for
> > > > > > external clients, start method is invoked twice, once for
> > > each
> > > > > > listener. On
> > > > > > second thought, that may be confusing and a single start()
> > > > > invocation
> > > > > > that
> > > > > > provides all listener information and returns multiple
> > > futures
> > > > > would be
> > > > > > better. Will update the KIP.
> > > > > > 3. A typical example is a consumer subscribing to a regex
> > > > > pattern. We
> > > > > > request all topic metadata from the broker in order to
> > decide
> > > > > whether
> > > > > > the
> > > > > > pattern matches, expecting to receive a list of authorised
> > > > > topics. The
> > > > > > user
> > > > > > is not asking to subscribe to an unauthorized topic. If
> > there
> > > > > are 10000
> > > > > > topics in the cluster and the user has access to 100 of
> > them,
> > > > at
> > > > > the
> > > > > > moment
> > > > > > we log 9900 DENIED log entries at INFO level in
> > > > > SimpleAclAuthorizer.
> > > > > > The
> > > > > > proposal is to authorize this request with
> > > > > AuthorizationMode.FILTER, so
> > > > > > that authorizers can log resources that are filtered out at
> > > > > lower level
> > > > > > like DEBUG since this is not an attempt to access
> > > unauthorized
> > > > > > resources.
> > > > > > Brokers already handle these differently since no
> > > authorization
> > > > > error
> > > > > > is
> > > > > > returned to the client in these cases. Providing
> > > authorization
> > > > > mode to
> > > > > > authorizers enables authorizer implementations to generate
> > > > > better audit
> > > > > > logs.
> > > > > > 4. Each request may contain multiple instances of the same
> > > > > authorizable
> > > > > > resource. For example a produce request may contain records
> > > for
> > > > > 10
> > > > > > partitions of the same topic. At the moment, we invoke
> > > > authorize
> > > > > > method 10
> > > > > > times. The proposal is to invoke it once with count=10. The
> > > > > count is
> > > > > > provided to authorizer just for audit logging purposes.
> > > > > > 5. Authorizer implements Closeable, so you could use
> > close()
> > > to
> > > > > flush
> > > > > > audits?
> > > > > >
> > > > > > On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <
> > > > bosco@apache.org
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > > Thanks for putting this together. It is looking good. I
> > > have
> > > > > few
> > > > > > > questions...
> > > > > > >
> > > > > > > 1. List<AuthorizationResult> authorize(..., List<Action>
> > > > > actions).
> > > > > > Do you
> > > > > > > see a scenario where the broker will call authorize for
> > > > > multiple
> > > > > > topics at
> > > > > > > the same time? I can understand that during
> > > creating/deleting
> > > > > ACLS,
> > > > > > > multiple permissions for multiple resources might be
> > done.
> > > > For
> > > > > > authorize
> > > > > > > call, would this be a case? And does the Authorize
> > > > > implementation
> > > > > > will be
> > > > > > > able to do performance optimization because of this? Or
> > > > should
> > > > > we
> > > > > > just keep
> > > > > > > it simple? I don't see it as an issue from Apache Ranger
> > > > side,
> > > > > but
> > > > > > just
> > > > > > > checking to see whether we need to be aware of something.
> > > > > > > 2. Should I assume that the SecurityProtocol passed
> > during
> > > > > start and
> > > > > > the
> > > > > > > one return by KafkaRequestContext.securityProtocol() will
> > > be
> > > > > the
> > > > > > same?
> > > > > > > CompletableFuture<Void> start(String listenerName,
> > > > > SecurityProtocol
> > > > > > > securityProtocol);
> > > > > > > KafkaRequestContext.securityProtocol()
> > > > > > > 3. What is the purpose of AuthorizationMode? How does the
> > > > > broker
> > > > > > decide
> > > > > > > what mode to use when the authorize() method is called?
> > > > > > > 4. Can we clarify "count" in Action a bit more? How is it
> > > > used?
> > > > > > > 5. Do you feel having "stop" along with "start" be
> > helpful?
> > > > > E.g. In
> > > > > > Ranger
> > > > > > > we try to optimize the Audit writing by caching the logs
> > > for
> > > > a
> > > > > fixed
> > > > > > > interval. But when the Broker terminates, we do a forced
> > > > flush.
> > > > > > Having an
> > > > > > > explicit "stop" might give us a formal way to flush our
> > > > audits.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > Bosco
> > > > > > >
> > > > > > > On 8/7/19, 3:59 PM, "Rajini Sivaram" <
> > > > rajinisivaram@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Ron/Harsha/Satish,
> > > > > > >
> > > > > > > Thanks for reviewing the KIP!
> > > > > > >
> > > > > > > We should perhaps have a wider discussion outside
> > this
> > > > KIP
> > > > > for
> > > > > > > refactoring
> > > > > > > clients so that others who are not following this KIP
> > > > also
> > > > > > notice the
> > > > > > > discussion. Satish, would you like to start a
> > > discussion
> > > > > thread
> > > > > > on dev?
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <
> > > > > > > satish.duggana@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I felt the same need when we want to add a
> > pluggable
> > > > API
> > > > > for
> > > > > > core
> > > > > > > > server functionality. This does not need to be part
> > > of
> > > > > this
> > > > > > KIP, it
> > > > > > > > can be a separate KIP. I can contribute those
> > > > refactoring
> > > > > > changes if
> > > > > > > > others are OK with that.
> > > > > > > >
> > > > > > > > It is better to have a structure like below.
> > > > > > > >
> > > > > > > > kafka-common:
> > > > > > > > common classes which can be used in any of the
> > other
> > > > > modules
> > > > > > in Kafka
> > > > > > > > like client, Kafka-server-common and server etc.
> > > > > > > >
> > > > > > > > kafka-client-common:
> > > > > > > > common classes which can be used in the client
> > > module.
> > > > > This
> > > > > > can be
> > > > > > > > part of client module itself.
> > > > > > > >
> > > > > > > > kafka-server-common:
> > > > > > > > classes required only for kafka-server.
> > > > > > > >
> > > > > > > > Thanks.
> > > > > > > > Satish.
> > > > > > > >
> > > > > > > > On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani
> > <
> > > > > > kafka@harsha.io>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Thanks for the KIP Rajini.
> > > > > > > > > Quick thought, it would be good to have a common
> > > > module
> > > > > > outside of
> > > > > > > > clients
> > > > > > > > > that only applies to server side interfaces &
> > > > changes.
> > > > > It
> > > > > > looks
> > > > > > > like we
> > > > > > > > are
> > > > > > > > > increasingly in favor of using Java interface for
> > > > > pluggable
> > > > > > > modules on
> > > > > > > > the
> > > > > > > > > broker side.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Harsha
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <
> > > > > > > rajinisivaram@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I have created a KIP to replace the Scala
> > > > Authorizer
> > > > > API
> > > > > > with a
> > > > > > > new
> > > > > > > > Java
> > > > > > > > > > API:
> > > > > > > > > >
> > > > > > > > > > -
> > > > > > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > > > > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > > > > > > >
> > > > > > > > > > This is replacement for KIP-50 which was
> > accepted
> > > > > but never
> > > > > > > merged.
> > > > > > > > Apart
> > > > > > > > > > from moving to a Java API consistent with other
> > > > > pluggable
> > > > > > > interfaces
> > > > > > > > in the
> > > > > > > > > > broker, KIP-504 also attempts to address known
> > > > > limitations
> > > > > > in the
> > > > > > > > > > authorizer. If you have come across other
> > > > > limitations that
> > > > > > you
> > > > > > > would
> > > > > > > > like
> > > > > > > > > > to see addressed in the new API, please raise
> > > these
> > > > > on the
> > > > > > > discussion
> > > > > > > > > > thread so that we can consider those too. All
> > > > > suggestions
> > > > > > and
> > > > > > > feedback
> > > > > > > > are
> > > > > > > > > > welcome.
> > > > > > > > > >
> > > > > > > > > > Thank you,
> > > > > > > > > >
> > > > > > > > > > Rajini
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Thanks David! I have fixed the typo.
Also made a couple of changes to make the context interfaces more generic.
KafkaRequestContext now returns the 16-bit API key as Colin suggested as
well as the friendly name used in metrics which are useful in audit logs.
`Authorizer#start` is now provided a generic `BrokerInfo` interface that
gives cluster id, broker id and endpoint information. The generic interface
can potentially be used in other broker plugins in future and provides
dynamically generated configs like broker id and ports which are currently
not available to plugins unless these configs are statically configured.
Please let me know if there are any concerns.
Regards,
Rajini
On Tue, Aug 13, 2019 at 4:30 PM David Jacot <dj...@confluent.io> wrote:
> Hi Rajini,
>
> Thank you for the update! It looks good to me. There is a typo in the
> `AuditFlag` enum: `MANDATORY_AUTHOEIZE` -> `MANDATORY_AUTHORIZE`.
>
> Regards,
> David
>
> On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram <ra...@gmail.com>
> wrote:
>
> > Hi David,
> >
> > Thanks for reviewing the KIP! Since questions about `authorization mode`
> > and `count` have come up multiple times, I have renamed both.
> >
> > 1) Renamed `count` to `resourceReferenceCount`. It is the number of times
> > the resource being authorized is referenced within the request.
> >
> > 2) Renamed `AuthorizationMode` to `AuditFlag`. It is provided to improve
> > audit logging in the authorizer. The enum values have javadoc which
> > indicate how the authorization result is used in each of the modes to
> > enable authorizers to log audit messages at the right severity level.
> >
> > Regards,
> >
> > Rajini
> >
> > On Mon, Aug 12, 2019 at 5:54 PM David Jacot <dj...@confluent.io> wrote:
> >
> > > Hi Rajini,
> > >
> > > Thank you for the KIP. Overall, it looks good to me. I have few
> > > questions/suggestions:
> > >
> > > 1. It is hard to grasp what `Action#count` is for. I guess I understand
> > > where you want to go with it but it took me a while to figure it out.
> > > Perhaps, we could come up with a better name than `count`?
> > >
> > > 2. I had a hard time trying to understand the `AuthorizationMode`
> > concept,
> > > especially wrt. the OPTIONAL one. My understanding is that an ACL is
> > either
> > > defined or not. Could you elaborate a bit more on that?
> > >
> > > Thanks,
> > > David
> > >
> > > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai <bo...@apache.org>
> > wrote:
> > >
> > > > Hi Rajini
> > > >
> > > > 3.2 - This makes sense. Thanks for clarifying.
> > > >
> > > > Rest looks fine. Once the implementations are done, it will be more
> > clear
> > > > on the different values RequestType and Mode.
> > > >
> > > > Thanks
> > > >
> > > > Bosco
> > > >
> > > >
> > > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <ra...@gmail.com>
> wrote:
> > > >
> > > > Hi Don,
> > > >
> > > > Thanks for the suggestions. A few responses below:
> > > >
> > > > 3.1 Can rename and improve docs if we keep this. Let's finish the
> > > > discussion on Colin's suggestions regarding this first.
> > > > 3.2 No, I was thinking of some requests that have an old way of
> > > > authorizing
> > > > and a new way where we have retained the old way for backward
> > > > compatibility. One example is Cluster:Create permission to create
> > > > topics.
> > > > We have replaced this with fine-grained topic create access using
> > > > Topic:Create
> > > > for topic patterns. But we still check if user has Cluster:Create
> > > > first. If
> > > > Denied, the deny is ignored and we check Topic:Create. We dont
> want
> > > to
> > > > log
> > > > DENY for Cluster:Create at INFO level for this, since this is
> not a
> > > > mandatory ACL for creating topics. We will get appropriate logs
> > from
> > > > the
> > > > subsequent Topic:Create in this case.
> > > > 3.3 They are not quite the same. FILTER implies that user
> actually
> > > > requested access to and performed some operation on the filtered
> > > > resources.
> > > > LIST_AUTHORZED did not result in any actual access. User simply
> > > wanted
> > > > to
> > > > know what they are allowed to access.
> > > > 3.4 Request types are Produce, JoinGroup, OffsetCommit etc. So
> that
> > > is
> > > > different from authorization mode, operation etc.
> > > >
> > > >
> > > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <
> bosco@apache.org>
> > > > wrote:
> > > >
> > > > > Hi Rajini
> > > > >
> > > > > Thanks for clarifying. This is very helpful...
> > > > >
> > > > > #1 - On the Ranger side, we should be able to handle multiple
> > > > requests at
> > > > > the same time. I was just not sure how much processing overhead
> > > will
> > > > be
> > > > > there on the Broker side to split and then consolidate the
> > results.
> > > > If it
> > > > > is negligible, then this is the better way. It will make it
> > future
> > > > proof.
> > > > > #2 - I agree, having a single "start" call makes it cleaner.
> The
> > > > > Authorization implementation will only have to do the
> > > initialization
> > > > only
> > > > > once.
> > > > > #3.1 - Thanks for the clarification. I think it makes sense to
> > have
> > > > this.
> > > > > The term "Mode" might not be explicit enough. Essentially it
> > seems
> > > > you want
> > > > > the Authorizer to know the Intent/Purpose of the authorize call
> > and
> > > > let the
> > > > > Authorizer decide what to log as Audit event. Changing the
> > > > class/field name
> > > > > or giving more documentation will do.
> > > > > #3.2 - Regarding the option "OPTIONAL". Are you thinking from
> > > > chaining
> > > > > multiple Authorizer? If so, I am not sure whether the Broker
> > would
> > > > have
> > > > > enough information to make that decision. I feel the Authorizer
> > > will
> > > > be the
> > > > > one who would have that knowledge. E.g. in Ranger we have
> > explicit
> > > > deny,
> > > > > which means no matter what, the request should be denied for
> the
> > > > user/group
> > > > > or condition. So if you are thinking of chaining Authorizers,
> > then
> > > > > responses should have the third state, e.g. "DENIED_FINAL", in
> > > which
> > > > case
> > > > > if there is an Authorization chain, it will be stop and the
> > request
> > > > will be
> > > > > denied and if it is just denied, then you might fall back to
> next
> > > > > authorizer. If we don't have chaining of Authorizing in mind,
> > then
> > > we
> > > > > should reconsider OPTIONAL for now. Or clarify under which
> > scenario
> > > > > OPTIONAL will be called.
> > > > > #3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't
> > LIST_AUTHORIZED a
> > > > > special case for "FILTER"?
> > > > > #3.4 KafkaRequestContext. requestType() v/s Action.
> > > > authorizationMode. I
> > > > > am not sure about the overlap or ambiguity.
> > > > > #4 - Cool. This is good, it will be less stress on the
> > Authorizer.
> > > > Ranger
> > > > > already supports the "count" concept and also has batching
> > > > capability to
> > > > > aggregate similar requests to reduce the number of audit logs
> to
> > > > write. We
> > > > > should be able to pass this through.
> > > > > #5 - Assuming if the object instance is going out of scope, we
> > > > should be
> > > > > fine. Not a super important ask. Ranger is already catching
> KILL
> > > > signal for
> > > > > clean up.
> > > > >
> > > > > Thanks again. These are good enhancements. Appreciate your
> > efforts
> > > > here.
> > > > >
> > > > > Bosco
> > > > >
> > > > >
> > > > >
> > > > > On 8/8/19, 2:03 AM, "Rajini Sivaram" <rajinisivaram@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > Hi Don,
> > > > >
> > > > > Thanks for reviewing the KIP.
> > > > >
> > > > > 1. I had this originally as a single Action, but thought it
> > may
> > > > be
> > > > > useful
> > > > > to support batched authorize calls as well and keep it
> > > > consistent with
> > > > > other methods. Single requests can contain multiple topics.
> > For
> > > > > example a
> > > > > produce request can contain records for several partitions
> of
> > > > different
> > > > > topics. Broker could potentially authorize these together.
> > For
> > > > > SimpleAclAuthorizer, batched authorize methods don't
> provide
> > > any
> > > > > optimisation since lookup is based on resources followed by
> > the
> > > > > matching
> > > > > logic. But some authorizers may manage ACLs by user
> principal
> > > > rather
> > > > > than
> > > > > resource and may be able to optimize batched requests. I am
> > ok
> > > > with
> > > > > using
> > > > > single Action if this is likely to cause issues.
> > > > > 2. If you have two listeners, one for inter-broker traffic
> > and
> > > > another
> > > > > for
> > > > > external clients, start method is invoked twice, once for
> > each
> > > > > listener. On
> > > > > second thought, that may be confusing and a single start()
> > > > invocation
> > > > > that
> > > > > provides all listener information and returns multiple
> > futures
> > > > would be
> > > > > better. Will update the KIP.
> > > > > 3. A typical example is a consumer subscribing to a regex
> > > > pattern. We
> > > > > request all topic metadata from the broker in order to
> decide
> > > > whether
> > > > > the
> > > > > pattern matches, expecting to receive a list of authorised
> > > > topics. The
> > > > > user
> > > > > is not asking to subscribe to an unauthorized topic. If
> there
> > > > are 10000
> > > > > topics in the cluster and the user has access to 100 of
> them,
> > > at
> > > > the
> > > > > moment
> > > > > we log 9900 DENIED log entries at INFO level in
> > > > SimpleAclAuthorizer.
> > > > > The
> > > > > proposal is to authorize this request with
> > > > AuthorizationMode.FILTER, so
> > > > > that authorizers can log resources that are filtered out at
> > > > lower level
> > > > > like DEBUG since this is not an attempt to access
> > unauthorized
> > > > > resources.
> > > > > Brokers already handle these differently since no
> > authorization
> > > > error
> > > > > is
> > > > > returned to the client in these cases. Providing
> > authorization
> > > > mode to
> > > > > authorizers enables authorizer implementations to generate
> > > > better audit
> > > > > logs.
> > > > > 4. Each request may contain multiple instances of the same
> > > > authorizable
> > > > > resource. For example a produce request may contain records
> > for
> > > > 10
> > > > > partitions of the same topic. At the moment, we invoke
> > > authorize
> > > > > method 10
> > > > > times. The proposal is to invoke it once with count=10. The
> > > > count is
> > > > > provided to authorizer just for audit logging purposes.
> > > > > 5. Authorizer implements Closeable, so you could use
> close()
> > to
> > > > flush
> > > > > audits?
> > > > >
> > > > > On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <
> > > bosco@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Rajini
> > > > > >
> > > > > > Thanks for putting this together. It is looking good. I
> > have
> > > > few
> > > > > > questions...
> > > > > >
> > > > > > 1. List<AuthorizationResult> authorize(..., List<Action>
> > > > actions).
> > > > > Do you
> > > > > > see a scenario where the broker will call authorize for
> > > > multiple
> > > > > topics at
> > > > > > the same time? I can understand that during
> > creating/deleting
> > > > ACLS,
> > > > > > multiple permissions for multiple resources might be
> done.
> > > For
> > > > > authorize
> > > > > > call, would this be a case? And does the Authorize
> > > > implementation
> > > > > will be
> > > > > > able to do performance optimization because of this? Or
> > > should
> > > > we
> > > > > just keep
> > > > > > it simple? I don't see it as an issue from Apache Ranger
> > > side,
> > > > but
> > > > > just
> > > > > > checking to see whether we need to be aware of something.
> > > > > > 2. Should I assume that the SecurityProtocol passed
> during
> > > > start and
> > > > > the
> > > > > > one return by KafkaRequestContext.securityProtocol() will
> > be
> > > > the
> > > > > same?
> > > > > > CompletableFuture<Void> start(String listenerName,
> > > > SecurityProtocol
> > > > > > securityProtocol);
> > > > > > KafkaRequestContext.securityProtocol()
> > > > > > 3. What is the purpose of AuthorizationMode? How does the
> > > > broker
> > > > > decide
> > > > > > what mode to use when the authorize() method is called?
> > > > > > 4. Can we clarify "count" in Action a bit more? How is it
> > > used?
> > > > > > 5. Do you feel having "stop" along with "start" be
> helpful?
> > > > E.g. In
> > > > > Ranger
> > > > > > we try to optimize the Audit writing by caching the logs
> > for
> > > a
> > > > fixed
> > > > > > interval. But when the Broker terminates, we do a forced
> > > flush.
> > > > > Having an
> > > > > > explicit "stop" might give us a formal way to flush our
> > > audits.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > Bosco
> > > > > >
> > > > > > On 8/7/19, 3:59 PM, "Rajini Sivaram" <
> > > rajinisivaram@gmail.com
> > > > >
> > > > > wrote:
> > > > > >
> > > > > > Hi Ron/Harsha/Satish,
> > > > > >
> > > > > > Thanks for reviewing the KIP!
> > > > > >
> > > > > > We should perhaps have a wider discussion outside
> this
> > > KIP
> > > > for
> > > > > > refactoring
> > > > > > clients so that others who are not following this KIP
> > > also
> > > > > notice the
> > > > > > discussion. Satish, would you like to start a
> > discussion
> > > > thread
> > > > > on dev?
> > > > > >
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > >
> > > > > > On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <
> > > > > > satish.duggana@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I felt the same need when we want to add a
> pluggable
> > > API
> > > > for
> > > > > core
> > > > > > > server functionality. This does not need to be part
> > of
> > > > this
> > > > > KIP, it
> > > > > > > can be a separate KIP. I can contribute those
> > > refactoring
> > > > > changes if
> > > > > > > others are OK with that.
> > > > > > >
> > > > > > > It is better to have a structure like below.
> > > > > > >
> > > > > > > kafka-common:
> > > > > > > common classes which can be used in any of the
> other
> > > > modules
> > > > > in Kafka
> > > > > > > like client, Kafka-server-common and server etc.
> > > > > > >
> > > > > > > kafka-client-common:
> > > > > > > common classes which can be used in the client
> > module.
> > > > This
> > > > > can be
> > > > > > > part of client module itself.
> > > > > > >
> > > > > > > kafka-server-common:
> > > > > > > classes required only for kafka-server.
> > > > > > >
> > > > > > > Thanks.
> > > > > > > Satish.
> > > > > > >
> > > > > > > On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani
> <
> > > > > kafka@harsha.io>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Thanks for the KIP Rajini.
> > > > > > > > Quick thought, it would be good to have a common
> > > module
> > > > > outside of
> > > > > > > clients
> > > > > > > > that only applies to server side interfaces &
> > > changes.
> > > > It
> > > > > looks
> > > > > > like we
> > > > > > > are
> > > > > > > > increasingly in favor of using Java interface for
> > > > pluggable
> > > > > > modules on
> > > > > > > the
> > > > > > > > broker side.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Harsha
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <
> > > > > > rajinisivaram@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I have created a KIP to replace the Scala
> > > Authorizer
> > > > API
> > > > > with a
> > > > > > new
> > > > > > > Java
> > > > > > > > > API:
> > > > > > > > >
> > > > > > > > > -
> > > > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > > > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > > > > > >
> > > > > > > > > This is replacement for KIP-50 which was
> accepted
> > > > but never
> > > > > > merged.
> > > > > > > Apart
> > > > > > > > > from moving to a Java API consistent with other
> > > > pluggable
> > > > > > interfaces
> > > > > > > in the
> > > > > > > > > broker, KIP-504 also attempts to address known
> > > > limitations
> > > > > in the
> > > > > > > > > authorizer. If you have come across other
> > > > limitations that
> > > > > you
> > > > > > would
> > > > > > > like
> > > > > > > > > to see addressed in the new API, please raise
> > these
> > > > on the
> > > > > > discussion
> > > > > > > > > thread so that we can consider those too. All
> > > > suggestions
> > > > > and
> > > > > > feedback
> > > > > > > are
> > > > > > > > > welcome.
> > > > > > > > >
> > > > > > > > > Thank you,
> > > > > > > > >
> > > > > > > > > Rajini
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by David Jacot <dj...@confluent.io>.
Hi Rajini,
Thank you for the update! It looks good to me. There is a typo in the
`AuditFlag` enum: `MANDATORY_AUTHOEIZE` -> `MANDATORY_AUTHORIZE`.
Regards,
David
On Mon, Aug 12, 2019 at 2:54 PM Rajini Sivaram <ra...@gmail.com>
wrote:
> Hi David,
>
> Thanks for reviewing the KIP! Since questions about `authorization mode`
> and `count` have come up multiple times, I have renamed both.
>
> 1) Renamed `count` to `resourceReferenceCount`. It is the number of times
> the resource being authorized is referenced within the request.
>
> 2) Renamed `AuthorizationMode` to `AuditFlag`. It is provided to improve
> audit logging in the authorizer. The enum values have javadoc which
> indicate how the authorization result is used in each of the modes to
> enable authorizers to log audit messages at the right severity level.
>
> Regards,
>
> Rajini
>
> On Mon, Aug 12, 2019 at 5:54 PM David Jacot <dj...@confluent.io> wrote:
>
> > Hi Rajini,
> >
> > Thank you for the KIP. Overall, it looks good to me. I have few
> > questions/suggestions:
> >
> > 1. It is hard to grasp what `Action#count` is for. I guess I understand
> > where you want to go with it but it took me a while to figure it out.
> > Perhaps, we could come up with a better name than `count`?
> >
> > 2. I had a hard time trying to understand the `AuthorizationMode`
> concept,
> > especially wrt. the OPTIONAL one. My understanding is that an ACL is
> either
> > defined or not. Could you elaborate a bit more on that?
> >
> > Thanks,
> > David
> >
> > On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai <bo...@apache.org>
> wrote:
> >
> > > Hi Rajini
> > >
> > > 3.2 - This makes sense. Thanks for clarifying.
> > >
> > > Rest looks fine. Once the implementations are done, it will be more
> clear
> > > on the different values RequestType and Mode.
> > >
> > > Thanks
> > >
> > > Bosco
> > >
> > >
> > > On 8/9/19, 5:08 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
> > >
> > > Hi Don,
> > >
> > > Thanks for the suggestions. A few responses below:
> > >
> > > 3.1 Can rename and improve docs if we keep this. Let's finish the
> > > discussion on Colin's suggestions regarding this first.
> > > 3.2 No, I was thinking of some requests that have an old way of
> > > authorizing
> > > and a new way where we have retained the old way for backward
> > > compatibility. One example is Cluster:Create permission to create
> > > topics.
> > > We have replaced this with fine-grained topic create access using
> > > Topic:Create
> > > for topic patterns. But we still check if user has Cluster:Create
> > > first. If
> > > Denied, the deny is ignored and we check Topic:Create. We dont want
> > to
> > > log
> > > DENY for Cluster:Create at INFO level for this, since this is not a
> > > mandatory ACL for creating topics. We will get appropriate logs
> from
> > > the
> > > subsequent Topic:Create in this case.
> > > 3.3 They are not quite the same. FILTER implies that user actually
> > > requested access to and performed some operation on the filtered
> > > resources.
> > > LIST_AUTHORZED did not result in any actual access. User simply
> > wanted
> > > to
> > > know what they are allowed to access.
> > > 3.4 Request types are Produce, JoinGroup, OffsetCommit etc. So that
> > is
> > > different from authorization mode, operation etc.
> > >
> > >
> > > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <bo...@apache.org>
> > > wrote:
> > >
> > > > Hi Rajini
> > > >
> > > > Thanks for clarifying. This is very helpful...
> > > >
> > > > #1 - On the Ranger side, we should be able to handle multiple
> > > requests at
> > > > the same time. I was just not sure how much processing overhead
> > will
> > > be
> > > > there on the Broker side to split and then consolidate the
> results.
> > > If it
> > > > is negligible, then this is the better way. It will make it
> future
> > > proof.
> > > > #2 - I agree, having a single "start" call makes it cleaner. The
> > > > Authorization implementation will only have to do the
> > initialization
> > > only
> > > > once.
> > > > #3.1 - Thanks for the clarification. I think it makes sense to
> have
> > > this.
> > > > The term "Mode" might not be explicit enough. Essentially it
> seems
> > > you want
> > > > the Authorizer to know the Intent/Purpose of the authorize call
> and
> > > let the
> > > > Authorizer decide what to log as Audit event. Changing the
> > > class/field name
> > > > or giving more documentation will do.
> > > > #3.2 - Regarding the option "OPTIONAL". Are you thinking from
> > > chaining
> > > > multiple Authorizer? If so, I am not sure whether the Broker
> would
> > > have
> > > > enough information to make that decision. I feel the Authorizer
> > will
> > > be the
> > > > one who would have that knowledge. E.g. in Ranger we have
> explicit
> > > deny,
> > > > which means no matter what, the request should be denied for the
> > > user/group
> > > > or condition. So if you are thinking of chaining Authorizers,
> then
> > > > responses should have the third state, e.g. "DENIED_FINAL", in
> > which
> > > case
> > > > if there is an Authorization chain, it will be stop and the
> request
> > > will be
> > > > denied and if it is just denied, then you might fall back to next
> > > > authorizer. If we don't have chaining of Authorizing in mind,
> then
> > we
> > > > should reconsider OPTIONAL for now. Or clarify under which
> scenario
> > > > OPTIONAL will be called.
> > > > #3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't
> LIST_AUTHORIZED a
> > > > special case for "FILTER"?
> > > > #3.4 KafkaRequestContext. requestType() v/s Action.
> > > authorizationMode. I
> > > > am not sure about the overlap or ambiguity.
> > > > #4 - Cool. This is good, it will be less stress on the
> Authorizer.
> > > Ranger
> > > > already supports the "count" concept and also has batching
> > > capability to
> > > > aggregate similar requests to reduce the number of audit logs to
> > > write. We
> > > > should be able to pass this through.
> > > > #5 - Assuming if the object instance is going out of scope, we
> > > should be
> > > > fine. Not a super important ask. Ranger is already catching KILL
> > > signal for
> > > > clean up.
> > > >
> > > > Thanks again. These are good enhancements. Appreciate your
> efforts
> > > here.
> > > >
> > > > Bosco
> > > >
> > > >
> > > >
> > > > On 8/8/19, 2:03 AM, "Rajini Sivaram" <ra...@gmail.com>
> > > wrote:
> > > >
> > > > Hi Don,
> > > >
> > > > Thanks for reviewing the KIP.
> > > >
> > > > 1. I had this originally as a single Action, but thought it
> may
> > > be
> > > > useful
> > > > to support batched authorize calls as well and keep it
> > > consistent with
> > > > other methods. Single requests can contain multiple topics.
> For
> > > > example a
> > > > produce request can contain records for several partitions of
> > > different
> > > > topics. Broker could potentially authorize these together.
> For
> > > > SimpleAclAuthorizer, batched authorize methods don't provide
> > any
> > > > optimisation since lookup is based on resources followed by
> the
> > > > matching
> > > > logic. But some authorizers may manage ACLs by user principal
> > > rather
> > > > than
> > > > resource and may be able to optimize batched requests. I am
> ok
> > > with
> > > > using
> > > > single Action if this is likely to cause issues.
> > > > 2. If you have two listeners, one for inter-broker traffic
> and
> > > another
> > > > for
> > > > external clients, start method is invoked twice, once for
> each
> > > > listener. On
> > > > second thought, that may be confusing and a single start()
> > > invocation
> > > > that
> > > > provides all listener information and returns multiple
> futures
> > > would be
> > > > better. Will update the KIP.
> > > > 3. A typical example is a consumer subscribing to a regex
> > > pattern. We
> > > > request all topic metadata from the broker in order to decide
> > > whether
> > > > the
> > > > pattern matches, expecting to receive a list of authorised
> > > topics. The
> > > > user
> > > > is not asking to subscribe to an unauthorized topic. If there
> > > are 10000
> > > > topics in the cluster and the user has access to 100 of them,
> > at
> > > the
> > > > moment
> > > > we log 9900 DENIED log entries at INFO level in
> > > SimpleAclAuthorizer.
> > > > The
> > > > proposal is to authorize this request with
> > > AuthorizationMode.FILTER, so
> > > > that authorizers can log resources that are filtered out at
> > > lower level
> > > > like DEBUG since this is not an attempt to access
> unauthorized
> > > > resources.
> > > > Brokers already handle these differently since no
> authorization
> > > error
> > > > is
> > > > returned to the client in these cases. Providing
> authorization
> > > mode to
> > > > authorizers enables authorizer implementations to generate
> > > better audit
> > > > logs.
> > > > 4. Each request may contain multiple instances of the same
> > > authorizable
> > > > resource. For example a produce request may contain records
> for
> > > 10
> > > > partitions of the same topic. At the moment, we invoke
> > authorize
> > > > method 10
> > > > times. The proposal is to invoke it once with count=10. The
> > > count is
> > > > provided to authorizer just for audit logging purposes.
> > > > 5. Authorizer implements Closeable, so you could use close()
> to
> > > flush
> > > > audits?
> > > >
> > > > On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <
> > bosco@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Rajini
> > > > >
> > > > > Thanks for putting this together. It is looking good. I
> have
> > > few
> > > > > questions...
> > > > >
> > > > > 1. List<AuthorizationResult> authorize(..., List<Action>
> > > actions).
> > > > Do you
> > > > > see a scenario where the broker will call authorize for
> > > multiple
> > > > topics at
> > > > > the same time? I can understand that during
> creating/deleting
> > > ACLS,
> > > > > multiple permissions for multiple resources might be done.
> > For
> > > > authorize
> > > > > call, would this be a case? And does the Authorize
> > > implementation
> > > > will be
> > > > > able to do performance optimization because of this? Or
> > should
> > > we
> > > > just keep
> > > > > it simple? I don't see it as an issue from Apache Ranger
> > side,
> > > but
> > > > just
> > > > > checking to see whether we need to be aware of something.
> > > > > 2. Should I assume that the SecurityProtocol passed during
> > > start and
> > > > the
> > > > > one return by KafkaRequestContext.securityProtocol() will
> be
> > > the
> > > > same?
> > > > > CompletableFuture<Void> start(String listenerName,
> > > SecurityProtocol
> > > > > securityProtocol);
> > > > > KafkaRequestContext.securityProtocol()
> > > > > 3. What is the purpose of AuthorizationMode? How does the
> > > broker
> > > > decide
> > > > > what mode to use when the authorize() method is called?
> > > > > 4. Can we clarify "count" in Action a bit more? How is it
> > used?
> > > > > 5. Do you feel having "stop" along with "start" be helpful?
> > > E.g. In
> > > > Ranger
> > > > > we try to optimize the Audit writing by caching the logs
> for
> > a
> > > fixed
> > > > > interval. But when the Broker terminates, we do a forced
> > flush.
> > > > Having an
> > > > > explicit "stop" might give us a formal way to flush our
> > audits.
> > > > >
> > > > > Thanks
> > > > >
> > > > > Bosco
> > > > >
> > > > > On 8/7/19, 3:59 PM, "Rajini Sivaram" <
> > rajinisivaram@gmail.com
> > > >
> > > > wrote:
> > > > >
> > > > > Hi Ron/Harsha/Satish,
> > > > >
> > > > > Thanks for reviewing the KIP!
> > > > >
> > > > > We should perhaps have a wider discussion outside this
> > KIP
> > > for
> > > > > refactoring
> > > > > clients so that others who are not following this KIP
> > also
> > > > notice the
> > > > > discussion. Satish, would you like to start a
> discussion
> > > thread
> > > > on dev?
> > > > >
> > > > > Regards,
> > > > >
> > > > > Rajini
> > > > >
> > > > >
> > > > > On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <
> > > > > satish.duggana@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I felt the same need when we want to add a pluggable
> > API
> > > for
> > > > core
> > > > > > server functionality. This does not need to be part
> of
> > > this
> > > > KIP, it
> > > > > > can be a separate KIP. I can contribute those
> > refactoring
> > > > changes if
> > > > > > others are OK with that.
> > > > > >
> > > > > > It is better to have a structure like below.
> > > > > >
> > > > > > kafka-common:
> > > > > > common classes which can be used in any of the other
> > > modules
> > > > in Kafka
> > > > > > like client, Kafka-server-common and server etc.
> > > > > >
> > > > > > kafka-client-common:
> > > > > > common classes which can be used in the client
> module.
> > > This
> > > > can be
> > > > > > part of client module itself.
> > > > > >
> > > > > > kafka-server-common:
> > > > > > classes required only for kafka-server.
> > > > > >
> > > > > > Thanks.
> > > > > > Satish.
> > > > > >
> > > > > > On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani <
> > > > kafka@harsha.io>
> > > > > > wrote:
> > > > > > >
> > > > > > > Thanks for the KIP Rajini.
> > > > > > > Quick thought, it would be good to have a common
> > module
> > > > outside of
> > > > > > clients
> > > > > > > that only applies to server side interfaces &
> > changes.
> > > It
> > > > looks
> > > > > like we
> > > > > > are
> > > > > > > increasingly in favor of using Java interface for
> > > pluggable
> > > > > modules on
> > > > > > the
> > > > > > > broker side.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Harsha
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <
> > > > > rajinisivaram@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I have created a KIP to replace the Scala
> > Authorizer
> > > API
> > > > with a
> > > > > new
> > > > > > Java
> > > > > > > > API:
> > > > > > > >
> > > > > > > > -
> > > > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > > > > >
> > > > > > > > This is replacement for KIP-50 which was accepted
> > > but never
> > > > > merged.
> > > > > > Apart
> > > > > > > > from moving to a Java API consistent with other
> > > pluggable
> > > > > interfaces
> > > > > > in the
> > > > > > > > broker, KIP-504 also attempts to address known
> > > limitations
> > > > in the
> > > > > > > > authorizer. If you have come across other
> > > limitations that
> > > > you
> > > > > would
> > > > > > like
> > > > > > > > to see addressed in the new API, please raise
> these
> > > on the
> > > > > discussion
> > > > > > > > thread so that we can consider those too. All
> > > suggestions
> > > > and
> > > > > feedback
> > > > > > are
> > > > > > > > welcome.
> > > > > > > >
> > > > > > > > Thank you,
> > > > > > > >
> > > > > > > > Rajini
> > > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi David,
Thanks for reviewing the KIP! Since questions about `authorization mode`
and `count` have come up multiple times, I have renamed both.
1) Renamed `count` to `resourceReferenceCount`. It is the number of times
the resource being authorized is referenced within the request.
2) Renamed `AuthorizationMode` to `AuditFlag`. It is provided to improve
audit logging in the authorizer. The enum values have javadoc which
indicate how the authorization result is used in each of the modes to
enable authorizers to log audit messages at the right severity level.
Regards,
Rajini
On Mon, Aug 12, 2019 at 5:54 PM David Jacot <dj...@confluent.io> wrote:
> Hi Rajini,
>
> Thank you for the KIP. Overall, it looks good to me. I have few
> questions/suggestions:
>
> 1. It is hard to grasp what `Action#count` is for. I guess I understand
> where you want to go with it but it took me a while to figure it out.
> Perhaps, we could come up with a better name than `count`?
>
> 2. I had a hard time trying to understand the `AuthorizationMode` concept,
> especially wrt. the OPTIONAL one. My understanding is that an ACL is either
> defined or not. Could you elaborate a bit more on that?
>
> Thanks,
> David
>
> On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai <bo...@apache.org> wrote:
>
> > Hi Rajini
> >
> > 3.2 - This makes sense. Thanks for clarifying.
> >
> > Rest looks fine. Once the implementations are done, it will be more clear
> > on the different values RequestType and Mode.
> >
> > Thanks
> >
> > Bosco
> >
> >
> > On 8/9/19, 5:08 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
> >
> > Hi Don,
> >
> > Thanks for the suggestions. A few responses below:
> >
> > 3.1 Can rename and improve docs if we keep this. Let's finish the
> > discussion on Colin's suggestions regarding this first.
> > 3.2 No, I was thinking of some requests that have an old way of
> > authorizing
> > and a new way where we have retained the old way for backward
> > compatibility. One example is Cluster:Create permission to create
> > topics.
> > We have replaced this with fine-grained topic create access using
> > Topic:Create
> > for topic patterns. But we still check if user has Cluster:Create
> > first. If
> > Denied, the deny is ignored and we check Topic:Create. We dont want
> to
> > log
> > DENY for Cluster:Create at INFO level for this, since this is not a
> > mandatory ACL for creating topics. We will get appropriate logs from
> > the
> > subsequent Topic:Create in this case.
> > 3.3 They are not quite the same. FILTER implies that user actually
> > requested access to and performed some operation on the filtered
> > resources.
> > LIST_AUTHORZED did not result in any actual access. User simply
> wanted
> > to
> > know what they are allowed to access.
> > 3.4 Request types are Produce, JoinGroup, OffsetCommit etc. So that
> is
> > different from authorization mode, operation etc.
> >
> >
> > On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <bo...@apache.org>
> > wrote:
> >
> > > Hi Rajini
> > >
> > > Thanks for clarifying. This is very helpful...
> > >
> > > #1 - On the Ranger side, we should be able to handle multiple
> > requests at
> > > the same time. I was just not sure how much processing overhead
> will
> > be
> > > there on the Broker side to split and then consolidate the results.
> > If it
> > > is negligible, then this is the better way. It will make it future
> > proof.
> > > #2 - I agree, having a single "start" call makes it cleaner. The
> > > Authorization implementation will only have to do the
> initialization
> > only
> > > once.
> > > #3.1 - Thanks for the clarification. I think it makes sense to have
> > this.
> > > The term "Mode" might not be explicit enough. Essentially it seems
> > you want
> > > the Authorizer to know the Intent/Purpose of the authorize call and
> > let the
> > > Authorizer decide what to log as Audit event. Changing the
> > class/field name
> > > or giving more documentation will do.
> > > #3.2 - Regarding the option "OPTIONAL". Are you thinking from
> > chaining
> > > multiple Authorizer? If so, I am not sure whether the Broker would
> > have
> > > enough information to make that decision. I feel the Authorizer
> will
> > be the
> > > one who would have that knowledge. E.g. in Ranger we have explicit
> > deny,
> > > which means no matter what, the request should be denied for the
> > user/group
> > > or condition. So if you are thinking of chaining Authorizers, then
> > > responses should have the third state, e.g. "DENIED_FINAL", in
> which
> > case
> > > if there is an Authorization chain, it will be stop and the request
> > will be
> > > denied and if it is just denied, then you might fall back to next
> > > authorizer. If we don't have chaining of Authorizing in mind, then
> we
> > > should reconsider OPTIONAL for now. Or clarify under which scenario
> > > OPTIONAL will be called.
> > > #3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't LIST_AUTHORIZED a
> > > special case for "FILTER"?
> > > #3.4 KafkaRequestContext. requestType() v/s Action.
> > authorizationMode. I
> > > am not sure about the overlap or ambiguity.
> > > #4 - Cool. This is good, it will be less stress on the Authorizer.
> > Ranger
> > > already supports the "count" concept and also has batching
> > capability to
> > > aggregate similar requests to reduce the number of audit logs to
> > write. We
> > > should be able to pass this through.
> > > #5 - Assuming if the object instance is going out of scope, we
> > should be
> > > fine. Not a super important ask. Ranger is already catching KILL
> > signal for
> > > clean up.
> > >
> > > Thanks again. These are good enhancements. Appreciate your efforts
> > here.
> > >
> > > Bosco
> > >
> > >
> > >
> > > On 8/8/19, 2:03 AM, "Rajini Sivaram" <ra...@gmail.com>
> > wrote:
> > >
> > > Hi Don,
> > >
> > > Thanks for reviewing the KIP.
> > >
> > > 1. I had this originally as a single Action, but thought it may
> > be
> > > useful
> > > to support batched authorize calls as well and keep it
> > consistent with
> > > other methods. Single requests can contain multiple topics. For
> > > example a
> > > produce request can contain records for several partitions of
> > different
> > > topics. Broker could potentially authorize these together. For
> > > SimpleAclAuthorizer, batched authorize methods don't provide
> any
> > > optimisation since lookup is based on resources followed by the
> > > matching
> > > logic. But some authorizers may manage ACLs by user principal
> > rather
> > > than
> > > resource and may be able to optimize batched requests. I am ok
> > with
> > > using
> > > single Action if this is likely to cause issues.
> > > 2. If you have two listeners, one for inter-broker traffic and
> > another
> > > for
> > > external clients, start method is invoked twice, once for each
> > > listener. On
> > > second thought, that may be confusing and a single start()
> > invocation
> > > that
> > > provides all listener information and returns multiple futures
> > would be
> > > better. Will update the KIP.
> > > 3. A typical example is a consumer subscribing to a regex
> > pattern. We
> > > request all topic metadata from the broker in order to decide
> > whether
> > > the
> > > pattern matches, expecting to receive a list of authorised
> > topics. The
> > > user
> > > is not asking to subscribe to an unauthorized topic. If there
> > are 10000
> > > topics in the cluster and the user has access to 100 of them,
> at
> > the
> > > moment
> > > we log 9900 DENIED log entries at INFO level in
> > SimpleAclAuthorizer.
> > > The
> > > proposal is to authorize this request with
> > AuthorizationMode.FILTER, so
> > > that authorizers can log resources that are filtered out at
> > lower level
> > > like DEBUG since this is not an attempt to access unauthorized
> > > resources.
> > > Brokers already handle these differently since no authorization
> > error
> > > is
> > > returned to the client in these cases. Providing authorization
> > mode to
> > > authorizers enables authorizer implementations to generate
> > better audit
> > > logs.
> > > 4. Each request may contain multiple instances of the same
> > authorizable
> > > resource. For example a produce request may contain records for
> > 10
> > > partitions of the same topic. At the moment, we invoke
> authorize
> > > method 10
> > > times. The proposal is to invoke it once with count=10. The
> > count is
> > > provided to authorizer just for audit logging purposes.
> > > 5. Authorizer implements Closeable, so you could use close() to
> > flush
> > > audits?
> > >
> > > On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <
> bosco@apache.org
> > >
> > > wrote:
> > >
> > > > Rajini
> > > >
> > > > Thanks for putting this together. It is looking good. I have
> > few
> > > > questions...
> > > >
> > > > 1. List<AuthorizationResult> authorize(..., List<Action>
> > actions).
> > > Do you
> > > > see a scenario where the broker will call authorize for
> > multiple
> > > topics at
> > > > the same time? I can understand that during creating/deleting
> > ACLS,
> > > > multiple permissions for multiple resources might be done.
> For
> > > authorize
> > > > call, would this be a case? And does the Authorize
> > implementation
> > > will be
> > > > able to do performance optimization because of this? Or
> should
> > we
> > > just keep
> > > > it simple? I don't see it as an issue from Apache Ranger
> side,
> > but
> > > just
> > > > checking to see whether we need to be aware of something.
> > > > 2. Should I assume that the SecurityProtocol passed during
> > start and
> > > the
> > > > one return by KafkaRequestContext.securityProtocol() will be
> > the
> > > same?
> > > > CompletableFuture<Void> start(String listenerName,
> > SecurityProtocol
> > > > securityProtocol);
> > > > KafkaRequestContext.securityProtocol()
> > > > 3. What is the purpose of AuthorizationMode? How does the
> > broker
> > > decide
> > > > what mode to use when the authorize() method is called?
> > > > 4. Can we clarify "count" in Action a bit more? How is it
> used?
> > > > 5. Do you feel having "stop" along with "start" be helpful?
> > E.g. In
> > > Ranger
> > > > we try to optimize the Audit writing by caching the logs for
> a
> > fixed
> > > > interval. But when the Broker terminates, we do a forced
> flush.
> > > Having an
> > > > explicit "stop" might give us a formal way to flush our
> audits.
> > > >
> > > > Thanks
> > > >
> > > > Bosco
> > > >
> > > > On 8/7/19, 3:59 PM, "Rajini Sivaram" <
> rajinisivaram@gmail.com
> > >
> > > wrote:
> > > >
> > > > Hi Ron/Harsha/Satish,
> > > >
> > > > Thanks for reviewing the KIP!
> > > >
> > > > We should perhaps have a wider discussion outside this
> KIP
> > for
> > > > refactoring
> > > > clients so that others who are not following this KIP
> also
> > > notice the
> > > > discussion. Satish, would you like to start a discussion
> > thread
> > > on dev?
> > > >
> > > > Regards,
> > > >
> > > > Rajini
> > > >
> > > >
> > > > On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <
> > > > satish.duggana@gmail.com>
> > > > wrote:
> > > >
> > > > > I felt the same need when we want to add a pluggable
> API
> > for
> > > core
> > > > > server functionality. This does not need to be part of
> > this
> > > KIP, it
> > > > > can be a separate KIP. I can contribute those
> refactoring
> > > changes if
> > > > > others are OK with that.
> > > > >
> > > > > It is better to have a structure like below.
> > > > >
> > > > > kafka-common:
> > > > > common classes which can be used in any of the other
> > modules
> > > in Kafka
> > > > > like client, Kafka-server-common and server etc.
> > > > >
> > > > > kafka-client-common:
> > > > > common classes which can be used in the client module.
> > This
> > > can be
> > > > > part of client module itself.
> > > > >
> > > > > kafka-server-common:
> > > > > classes required only for kafka-server.
> > > > >
> > > > > Thanks.
> > > > > Satish.
> > > > >
> > > > > On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani <
> > > kafka@harsha.io>
> > > > > wrote:
> > > > > >
> > > > > > Thanks for the KIP Rajini.
> > > > > > Quick thought, it would be good to have a common
> module
> > > outside of
> > > > > clients
> > > > > > that only applies to server side interfaces &
> changes.
> > It
> > > looks
> > > > like we
> > > > > are
> > > > > > increasingly in favor of using Java interface for
> > pluggable
> > > > modules on
> > > > > the
> > > > > > broker side.
> > > > > >
> > > > > > Thanks,
> > > > > > Harsha
> > > > > >
> > > > > >
> > > > > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <
> > > > rajinisivaram@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I have created a KIP to replace the Scala
> Authorizer
> > API
> > > with a
> > > > new
> > > > > Java
> > > > > > > API:
> > > > > > >
> > > > > > > -
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > > > >
> > > > > > > This is replacement for KIP-50 which was accepted
> > but never
> > > > merged.
> > > > > Apart
> > > > > > > from moving to a Java API consistent with other
> > pluggable
> > > > interfaces
> > > > > in the
> > > > > > > broker, KIP-504 also attempts to address known
> > limitations
> > > in the
> > > > > > > authorizer. If you have come across other
> > limitations that
> > > you
> > > > would
> > > > > like
> > > > > > > to see addressed in the new API, please raise these
> > on the
> > > > discussion
> > > > > > > thread so that we can consider those too. All
> > suggestions
> > > and
> > > > feedback
> > > > > are
> > > > > > > welcome.
> > > > > > >
> > > > > > > Thank you,
> > > > > > >
> > > > > > > Rajini
> > > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by David Jacot <dj...@confluent.io>.
Hi Rajini,
Thank you for the KIP. Overall, it looks good to me. I have few
questions/suggestions:
1. It is hard to grasp what `Action#count` is for. I guess I understand
where you want to go with it but it took me a while to figure it out.
Perhaps, we could come up with a better name than `count`?
2. I had a hard time trying to understand the `AuthorizationMode` concept,
especially wrt. the OPTIONAL one. My understanding is that an ACL is either
defined or not. Could you elaborate a bit more on that?
Thanks,
David
On Fri, Aug 9, 2019 at 10:26 PM Don Bosco Durai <bo...@apache.org> wrote:
> Hi Rajini
>
> 3.2 - This makes sense. Thanks for clarifying.
>
> Rest looks fine. Once the implementations are done, it will be more clear
> on the different values RequestType and Mode.
>
> Thanks
>
> Bosco
>
>
> On 8/9/19, 5:08 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
>
> Hi Don,
>
> Thanks for the suggestions. A few responses below:
>
> 3.1 Can rename and improve docs if we keep this. Let's finish the
> discussion on Colin's suggestions regarding this first.
> 3.2 No, I was thinking of some requests that have an old way of
> authorizing
> and a new way where we have retained the old way for backward
> compatibility. One example is Cluster:Create permission to create
> topics.
> We have replaced this with fine-grained topic create access using
> Topic:Create
> for topic patterns. But we still check if user has Cluster:Create
> first. If
> Denied, the deny is ignored and we check Topic:Create. We dont want to
> log
> DENY for Cluster:Create at INFO level for this, since this is not a
> mandatory ACL for creating topics. We will get appropriate logs from
> the
> subsequent Topic:Create in this case.
> 3.3 They are not quite the same. FILTER implies that user actually
> requested access to and performed some operation on the filtered
> resources.
> LIST_AUTHORZED did not result in any actual access. User simply wanted
> to
> know what they are allowed to access.
> 3.4 Request types are Produce, JoinGroup, OffsetCommit etc. So that is
> different from authorization mode, operation etc.
>
>
> On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <bo...@apache.org>
> wrote:
>
> > Hi Rajini
> >
> > Thanks for clarifying. This is very helpful...
> >
> > #1 - On the Ranger side, we should be able to handle multiple
> requests at
> > the same time. I was just not sure how much processing overhead will
> be
> > there on the Broker side to split and then consolidate the results.
> If it
> > is negligible, then this is the better way. It will make it future
> proof.
> > #2 - I agree, having a single "start" call makes it cleaner. The
> > Authorization implementation will only have to do the initialization
> only
> > once.
> > #3.1 - Thanks for the clarification. I think it makes sense to have
> this.
> > The term "Mode" might not be explicit enough. Essentially it seems
> you want
> > the Authorizer to know the Intent/Purpose of the authorize call and
> let the
> > Authorizer decide what to log as Audit event. Changing the
> class/field name
> > or giving more documentation will do.
> > #3.2 - Regarding the option "OPTIONAL". Are you thinking from
> chaining
> > multiple Authorizer? If so, I am not sure whether the Broker would
> have
> > enough information to make that decision. I feel the Authorizer will
> be the
> > one who would have that knowledge. E.g. in Ranger we have explicit
> deny,
> > which means no matter what, the request should be denied for the
> user/group
> > or condition. So if you are thinking of chaining Authorizers, then
> > responses should have the third state, e.g. "DENIED_FINAL", in which
> case
> > if there is an Authorization chain, it will be stop and the request
> will be
> > denied and if it is just denied, then you might fall back to next
> > authorizer. If we don't have chaining of Authorizing in mind, then we
> > should reconsider OPTIONAL for now. Or clarify under which scenario
> > OPTIONAL will be called.
> > #3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't LIST_AUTHORIZED a
> > special case for "FILTER"?
> > #3.4 KafkaRequestContext. requestType() v/s Action.
> authorizationMode. I
> > am not sure about the overlap or ambiguity.
> > #4 - Cool. This is good, it will be less stress on the Authorizer.
> Ranger
> > already supports the "count" concept and also has batching
> capability to
> > aggregate similar requests to reduce the number of audit logs to
> write. We
> > should be able to pass this through.
> > #5 - Assuming if the object instance is going out of scope, we
> should be
> > fine. Not a super important ask. Ranger is already catching KILL
> signal for
> > clean up.
> >
> > Thanks again. These are good enhancements. Appreciate your efforts
> here.
> >
> > Bosco
> >
> >
> >
> > On 8/8/19, 2:03 AM, "Rajini Sivaram" <ra...@gmail.com>
> wrote:
> >
> > Hi Don,
> >
> > Thanks for reviewing the KIP.
> >
> > 1. I had this originally as a single Action, but thought it may
> be
> > useful
> > to support batched authorize calls as well and keep it
> consistent with
> > other methods. Single requests can contain multiple topics. For
> > example a
> > produce request can contain records for several partitions of
> different
> > topics. Broker could potentially authorize these together. For
> > SimpleAclAuthorizer, batched authorize methods don't provide any
> > optimisation since lookup is based on resources followed by the
> > matching
> > logic. But some authorizers may manage ACLs by user principal
> rather
> > than
> > resource and may be able to optimize batched requests. I am ok
> with
> > using
> > single Action if this is likely to cause issues.
> > 2. If you have two listeners, one for inter-broker traffic and
> another
> > for
> > external clients, start method is invoked twice, once for each
> > listener. On
> > second thought, that may be confusing and a single start()
> invocation
> > that
> > provides all listener information and returns multiple futures
> would be
> > better. Will update the KIP.
> > 3. A typical example is a consumer subscribing to a regex
> pattern. We
> > request all topic metadata from the broker in order to decide
> whether
> > the
> > pattern matches, expecting to receive a list of authorised
> topics. The
> > user
> > is not asking to subscribe to an unauthorized topic. If there
> are 10000
> > topics in the cluster and the user has access to 100 of them, at
> the
> > moment
> > we log 9900 DENIED log entries at INFO level in
> SimpleAclAuthorizer.
> > The
> > proposal is to authorize this request with
> AuthorizationMode.FILTER, so
> > that authorizers can log resources that are filtered out at
> lower level
> > like DEBUG since this is not an attempt to access unauthorized
> > resources.
> > Brokers already handle these differently since no authorization
> error
> > is
> > returned to the client in these cases. Providing authorization
> mode to
> > authorizers enables authorizer implementations to generate
> better audit
> > logs.
> > 4. Each request may contain multiple instances of the same
> authorizable
> > resource. For example a produce request may contain records for
> 10
> > partitions of the same topic. At the moment, we invoke authorize
> > method 10
> > times. The proposal is to invoke it once with count=10. The
> count is
> > provided to authorizer just for audit logging purposes.
> > 5. Authorizer implements Closeable, so you could use close() to
> flush
> > audits?
> >
> > On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <bosco@apache.org
> >
> > wrote:
> >
> > > Rajini
> > >
> > > Thanks for putting this together. It is looking good. I have
> few
> > > questions...
> > >
> > > 1. List<AuthorizationResult> authorize(..., List<Action>
> actions).
> > Do you
> > > see a scenario where the broker will call authorize for
> multiple
> > topics at
> > > the same time? I can understand that during creating/deleting
> ACLS,
> > > multiple permissions for multiple resources might be done. For
> > authorize
> > > call, would this be a case? And does the Authorize
> implementation
> > will be
> > > able to do performance optimization because of this? Or should
> we
> > just keep
> > > it simple? I don't see it as an issue from Apache Ranger side,
> but
> > just
> > > checking to see whether we need to be aware of something.
> > > 2. Should I assume that the SecurityProtocol passed during
> start and
> > the
> > > one return by KafkaRequestContext.securityProtocol() will be
> the
> > same?
> > > CompletableFuture<Void> start(String listenerName,
> SecurityProtocol
> > > securityProtocol);
> > > KafkaRequestContext.securityProtocol()
> > > 3. What is the purpose of AuthorizationMode? How does the
> broker
> > decide
> > > what mode to use when the authorize() method is called?
> > > 4. Can we clarify "count" in Action a bit more? How is it used?
> > > 5. Do you feel having "stop" along with "start" be helpful?
> E.g. In
> > Ranger
> > > we try to optimize the Audit writing by caching the logs for a
> fixed
> > > interval. But when the Broker terminates, we do a forced flush.
> > Having an
> > > explicit "stop" might give us a formal way to flush our audits.
> > >
> > > Thanks
> > >
> > > Bosco
> > >
> > > On 8/7/19, 3:59 PM, "Rajini Sivaram" <rajinisivaram@gmail.com
> >
> > wrote:
> > >
> > > Hi Ron/Harsha/Satish,
> > >
> > > Thanks for reviewing the KIP!
> > >
> > > We should perhaps have a wider discussion outside this KIP
> for
> > > refactoring
> > > clients so that others who are not following this KIP also
> > notice the
> > > discussion. Satish, would you like to start a discussion
> thread
> > on dev?
> > >
> > > Regards,
> > >
> > > Rajini
> > >
> > >
> > > On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <
> > > satish.duggana@gmail.com>
> > > wrote:
> > >
> > > > I felt the same need when we want to add a pluggable API
> for
> > core
> > > > server functionality. This does not need to be part of
> this
> > KIP, it
> > > > can be a separate KIP. I can contribute those refactoring
> > changes if
> > > > others are OK with that.
> > > >
> > > > It is better to have a structure like below.
> > > >
> > > > kafka-common:
> > > > common classes which can be used in any of the other
> modules
> > in Kafka
> > > > like client, Kafka-server-common and server etc.
> > > >
> > > > kafka-client-common:
> > > > common classes which can be used in the client module.
> This
> > can be
> > > > part of client module itself.
> > > >
> > > > kafka-server-common:
> > > > classes required only for kafka-server.
> > > >
> > > > Thanks.
> > > > Satish.
> > > >
> > > > On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani <
> > kafka@harsha.io>
> > > > wrote:
> > > > >
> > > > > Thanks for the KIP Rajini.
> > > > > Quick thought, it would be good to have a common module
> > outside of
> > > > clients
> > > > > that only applies to server side interfaces & changes.
> It
> > looks
> > > like we
> > > > are
> > > > > increasingly in favor of using Java interface for
> pluggable
> > > modules on
> > > > the
> > > > > broker side.
> > > > >
> > > > > Thanks,
> > > > > Harsha
> > > > >
> > > > >
> > > > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <
> > > rajinisivaram@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I have created a KIP to replace the Scala Authorizer
> API
> > with a
> > > new
> > > > Java
> > > > > > API:
> > > > > >
> > > > > > -
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > > >
> > > > > > This is replacement for KIP-50 which was accepted
> but never
> > > merged.
> > > > Apart
> > > > > > from moving to a Java API consistent with other
> pluggable
> > > interfaces
> > > > in the
> > > > > > broker, KIP-504 also attempts to address known
> limitations
> > in the
> > > > > > authorizer. If you have come across other
> limitations that
> > you
> > > would
> > > > like
> > > > > > to see addressed in the new API, please raise these
> on the
> > > discussion
> > > > > > thread so that we can consider those too. All
> suggestions
> > and
> > > feedback
> > > > are
> > > > > > welcome.
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
>
>
>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Don Bosco Durai <bo...@apache.org>.
Hi Rajini
3.2 - This makes sense. Thanks for clarifying.
Rest looks fine. Once the implementations are done, it will be more clear on the different values RequestType and Mode.
Thanks
Bosco
On 8/9/19, 5:08 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
Hi Don,
Thanks for the suggestions. A few responses below:
3.1 Can rename and improve docs if we keep this. Let's finish the
discussion on Colin's suggestions regarding this first.
3.2 No, I was thinking of some requests that have an old way of authorizing
and a new way where we have retained the old way for backward
compatibility. One example is Cluster:Create permission to create topics.
We have replaced this with fine-grained topic create access using Topic:Create
for topic patterns. But we still check if user has Cluster:Create first. If
Denied, the deny is ignored and we check Topic:Create. We dont want to log
DENY for Cluster:Create at INFO level for this, since this is not a
mandatory ACL for creating topics. We will get appropriate logs from the
subsequent Topic:Create in this case.
3.3 They are not quite the same. FILTER implies that user actually
requested access to and performed some operation on the filtered resources.
LIST_AUTHORZED did not result in any actual access. User simply wanted to
know what they are allowed to access.
3.4 Request types are Produce, JoinGroup, OffsetCommit etc. So that is
different from authorization mode, operation etc.
On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <bo...@apache.org> wrote:
> Hi Rajini
>
> Thanks for clarifying. This is very helpful...
>
> #1 - On the Ranger side, we should be able to handle multiple requests at
> the same time. I was just not sure how much processing overhead will be
> there on the Broker side to split and then consolidate the results. If it
> is negligible, then this is the better way. It will make it future proof.
> #2 - I agree, having a single "start" call makes it cleaner. The
> Authorization implementation will only have to do the initialization only
> once.
> #3.1 - Thanks for the clarification. I think it makes sense to have this.
> The term "Mode" might not be explicit enough. Essentially it seems you want
> the Authorizer to know the Intent/Purpose of the authorize call and let the
> Authorizer decide what to log as Audit event. Changing the class/field name
> or giving more documentation will do.
> #3.2 - Regarding the option "OPTIONAL". Are you thinking from chaining
> multiple Authorizer? If so, I am not sure whether the Broker would have
> enough information to make that decision. I feel the Authorizer will be the
> one who would have that knowledge. E.g. in Ranger we have explicit deny,
> which means no matter what, the request should be denied for the user/group
> or condition. So if you are thinking of chaining Authorizers, then
> responses should have the third state, e.g. "DENIED_FINAL", in which case
> if there is an Authorization chain, it will be stop and the request will be
> denied and if it is just denied, then you might fall back to next
> authorizer. If we don't have chaining of Authorizing in mind, then we
> should reconsider OPTIONAL for now. Or clarify under which scenario
> OPTIONAL will be called.
> #3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't LIST_AUTHORIZED a
> special case for "FILTER"?
> #3.4 KafkaRequestContext. requestType() v/s Action. authorizationMode. I
> am not sure about the overlap or ambiguity.
> #4 - Cool. This is good, it will be less stress on the Authorizer. Ranger
> already supports the "count" concept and also has batching capability to
> aggregate similar requests to reduce the number of audit logs to write. We
> should be able to pass this through.
> #5 - Assuming if the object instance is going out of scope, we should be
> fine. Not a super important ask. Ranger is already catching KILL signal for
> clean up.
>
> Thanks again. These are good enhancements. Appreciate your efforts here.
>
> Bosco
>
>
>
> On 8/8/19, 2:03 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
>
> Hi Don,
>
> Thanks for reviewing the KIP.
>
> 1. I had this originally as a single Action, but thought it may be
> useful
> to support batched authorize calls as well and keep it consistent with
> other methods. Single requests can contain multiple topics. For
> example a
> produce request can contain records for several partitions of different
> topics. Broker could potentially authorize these together. For
> SimpleAclAuthorizer, batched authorize methods don't provide any
> optimisation since lookup is based on resources followed by the
> matching
> logic. But some authorizers may manage ACLs by user principal rather
> than
> resource and may be able to optimize batched requests. I am ok with
> using
> single Action if this is likely to cause issues.
> 2. If you have two listeners, one for inter-broker traffic and another
> for
> external clients, start method is invoked twice, once for each
> listener. On
> second thought, that may be confusing and a single start() invocation
> that
> provides all listener information and returns multiple futures would be
> better. Will update the KIP.
> 3. A typical example is a consumer subscribing to a regex pattern. We
> request all topic metadata from the broker in order to decide whether
> the
> pattern matches, expecting to receive a list of authorised topics. The
> user
> is not asking to subscribe to an unauthorized topic. If there are 10000
> topics in the cluster and the user has access to 100 of them, at the
> moment
> we log 9900 DENIED log entries at INFO level in SimpleAclAuthorizer.
> The
> proposal is to authorize this request with AuthorizationMode.FILTER, so
> that authorizers can log resources that are filtered out at lower level
> like DEBUG since this is not an attempt to access unauthorized
> resources.
> Brokers already handle these differently since no authorization error
> is
> returned to the client in these cases. Providing authorization mode to
> authorizers enables authorizer implementations to generate better audit
> logs.
> 4. Each request may contain multiple instances of the same authorizable
> resource. For example a produce request may contain records for 10
> partitions of the same topic. At the moment, we invoke authorize
> method 10
> times. The proposal is to invoke it once with count=10. The count is
> provided to authorizer just for audit logging purposes.
> 5. Authorizer implements Closeable, so you could use close() to flush
> audits?
>
> On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <bo...@apache.org>
> wrote:
>
> > Rajini
> >
> > Thanks for putting this together. It is looking good. I have few
> > questions...
> >
> > 1. List<AuthorizationResult> authorize(..., List<Action> actions).
> Do you
> > see a scenario where the broker will call authorize for multiple
> topics at
> > the same time? I can understand that during creating/deleting ACLS,
> > multiple permissions for multiple resources might be done. For
> authorize
> > call, would this be a case? And does the Authorize implementation
> will be
> > able to do performance optimization because of this? Or should we
> just keep
> > it simple? I don't see it as an issue from Apache Ranger side, but
> just
> > checking to see whether we need to be aware of something.
> > 2. Should I assume that the SecurityProtocol passed during start and
> the
> > one return by KafkaRequestContext.securityProtocol() will be the
> same?
> > CompletableFuture<Void> start(String listenerName, SecurityProtocol
> > securityProtocol);
> > KafkaRequestContext.securityProtocol()
> > 3. What is the purpose of AuthorizationMode? How does the broker
> decide
> > what mode to use when the authorize() method is called?
> > 4. Can we clarify "count" in Action a bit more? How is it used?
> > 5. Do you feel having "stop" along with "start" be helpful? E.g. In
> Ranger
> > we try to optimize the Audit writing by caching the logs for a fixed
> > interval. But when the Broker terminates, we do a forced flush.
> Having an
> > explicit "stop" might give us a formal way to flush our audits.
> >
> > Thanks
> >
> > Bosco
> >
> > On 8/7/19, 3:59 PM, "Rajini Sivaram" <ra...@gmail.com>
> wrote:
> >
> > Hi Ron/Harsha/Satish,
> >
> > Thanks for reviewing the KIP!
> >
> > We should perhaps have a wider discussion outside this KIP for
> > refactoring
> > clients so that others who are not following this KIP also
> notice the
> > discussion. Satish, would you like to start a discussion thread
> on dev?
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <
> > satish.duggana@gmail.com>
> > wrote:
> >
> > > I felt the same need when we want to add a pluggable API for
> core
> > > server functionality. This does not need to be part of this
> KIP, it
> > > can be a separate KIP. I can contribute those refactoring
> changes if
> > > others are OK with that.
> > >
> > > It is better to have a structure like below.
> > >
> > > kafka-common:
> > > common classes which can be used in any of the other modules
> in Kafka
> > > like client, Kafka-server-common and server etc.
> > >
> > > kafka-client-common:
> > > common classes which can be used in the client module. This
> can be
> > > part of client module itself.
> > >
> > > kafka-server-common:
> > > classes required only for kafka-server.
> > >
> > > Thanks.
> > > Satish.
> > >
> > > On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani <
> kafka@harsha.io>
> > > wrote:
> > > >
> > > > Thanks for the KIP Rajini.
> > > > Quick thought, it would be good to have a common module
> outside of
> > > clients
> > > > that only applies to server side interfaces & changes. It
> looks
> > like we
> > > are
> > > > increasingly in favor of using Java interface for pluggable
> > modules on
> > > the
> > > > broker side.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > >
> > > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <
> > rajinisivaram@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I have created a KIP to replace the Scala Authorizer API
> with a
> > new
> > > Java
> > > > > API:
> > > > >
> > > > > -
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > >
> > > > > This is replacement for KIP-50 which was accepted but never
> > merged.
> > > Apart
> > > > > from moving to a Java API consistent with other pluggable
> > interfaces
> > > in the
> > > > > broker, KIP-504 also attempts to address known limitations
> in the
> > > > > authorizer. If you have come across other limitations that
> you
> > would
> > > like
> > > > > to see addressed in the new API, please raise these on the
> > discussion
> > > > > thread so that we can consider those too. All suggestions
> and
> > feedback
> > > are
> > > > > welcome.
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Rajini
> > > > >
> > >
> >
> >
> >
> >
>
>
>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Don,
Thanks for the suggestions. A few responses below:
3.1 Can rename and improve docs if we keep this. Let's finish the
discussion on Colin's suggestions regarding this first.
3.2 No, I was thinking of some requests that have an old way of authorizing
and a new way where we have retained the old way for backward
compatibility. One example is Cluster:Create permission to create topics.
We have replaced this with fine-grained topic create access using Topic:Create
for topic patterns. But we still check if user has Cluster:Create first. If
Denied, the deny is ignored and we check Topic:Create. We dont want to log
DENY for Cluster:Create at INFO level for this, since this is not a
mandatory ACL for creating topics. We will get appropriate logs from the
subsequent Topic:Create in this case.
3.3 They are not quite the same. FILTER implies that user actually
requested access to and performed some operation on the filtered resources.
LIST_AUTHORZED did not result in any actual access. User simply wanted to
know what they are allowed to access.
3.4 Request types are Produce, JoinGroup, OffsetCommit etc. So that is
different from authorization mode, operation etc.
On Thu, Aug 8, 2019 at 11:36 PM Don Bosco Durai <bo...@apache.org> wrote:
> Hi Rajini
>
> Thanks for clarifying. This is very helpful...
>
> #1 - On the Ranger side, we should be able to handle multiple requests at
> the same time. I was just not sure how much processing overhead will be
> there on the Broker side to split and then consolidate the results. If it
> is negligible, then this is the better way. It will make it future proof.
> #2 - I agree, having a single "start" call makes it cleaner. The
> Authorization implementation will only have to do the initialization only
> once.
> #3.1 - Thanks for the clarification. I think it makes sense to have this.
> The term "Mode" might not be explicit enough. Essentially it seems you want
> the Authorizer to know the Intent/Purpose of the authorize call and let the
> Authorizer decide what to log as Audit event. Changing the class/field name
> or giving more documentation will do.
> #3.2 - Regarding the option "OPTIONAL". Are you thinking from chaining
> multiple Authorizer? If so, I am not sure whether the Broker would have
> enough information to make that decision. I feel the Authorizer will be the
> one who would have that knowledge. E.g. in Ranger we have explicit deny,
> which means no matter what, the request should be denied for the user/group
> or condition. So if you are thinking of chaining Authorizers, then
> responses should have the third state, e.g. "DENIED_FINAL", in which case
> if there is an Authorization chain, it will be stop and the request will be
> denied and if it is just denied, then you might fall back to next
> authorizer. If we don't have chaining of Authorizing in mind, then we
> should reconsider OPTIONAL for now. Or clarify under which scenario
> OPTIONAL will be called.
> #3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't LIST_AUTHORIZED a
> special case for "FILTER"?
> #3.4 KafkaRequestContext. requestType() v/s Action. authorizationMode. I
> am not sure about the overlap or ambiguity.
> #4 - Cool. This is good, it will be less stress on the Authorizer. Ranger
> already supports the "count" concept and also has batching capability to
> aggregate similar requests to reduce the number of audit logs to write. We
> should be able to pass this through.
> #5 - Assuming if the object instance is going out of scope, we should be
> fine. Not a super important ask. Ranger is already catching KILL signal for
> clean up.
>
> Thanks again. These are good enhancements. Appreciate your efforts here.
>
> Bosco
>
>
>
> On 8/8/19, 2:03 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
>
> Hi Don,
>
> Thanks for reviewing the KIP.
>
> 1. I had this originally as a single Action, but thought it may be
> useful
> to support batched authorize calls as well and keep it consistent with
> other methods. Single requests can contain multiple topics. For
> example a
> produce request can contain records for several partitions of different
> topics. Broker could potentially authorize these together. For
> SimpleAclAuthorizer, batched authorize methods don't provide any
> optimisation since lookup is based on resources followed by the
> matching
> logic. But some authorizers may manage ACLs by user principal rather
> than
> resource and may be able to optimize batched requests. I am ok with
> using
> single Action if this is likely to cause issues.
> 2. If you have two listeners, one for inter-broker traffic and another
> for
> external clients, start method is invoked twice, once for each
> listener. On
> second thought, that may be confusing and a single start() invocation
> that
> provides all listener information and returns multiple futures would be
> better. Will update the KIP.
> 3. A typical example is a consumer subscribing to a regex pattern. We
> request all topic metadata from the broker in order to decide whether
> the
> pattern matches, expecting to receive a list of authorised topics. The
> user
> is not asking to subscribe to an unauthorized topic. If there are 10000
> topics in the cluster and the user has access to 100 of them, at the
> moment
> we log 9900 DENIED log entries at INFO level in SimpleAclAuthorizer.
> The
> proposal is to authorize this request with AuthorizationMode.FILTER, so
> that authorizers can log resources that are filtered out at lower level
> like DEBUG since this is not an attempt to access unauthorized
> resources.
> Brokers already handle these differently since no authorization error
> is
> returned to the client in these cases. Providing authorization mode to
> authorizers enables authorizer implementations to generate better audit
> logs.
> 4. Each request may contain multiple instances of the same authorizable
> resource. For example a produce request may contain records for 10
> partitions of the same topic. At the moment, we invoke authorize
> method 10
> times. The proposal is to invoke it once with count=10. The count is
> provided to authorizer just for audit logging purposes.
> 5. Authorizer implements Closeable, so you could use close() to flush
> audits?
>
> On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <bo...@apache.org>
> wrote:
>
> > Rajini
> >
> > Thanks for putting this together. It is looking good. I have few
> > questions...
> >
> > 1. List<AuthorizationResult> authorize(..., List<Action> actions).
> Do you
> > see a scenario where the broker will call authorize for multiple
> topics at
> > the same time? I can understand that during creating/deleting ACLS,
> > multiple permissions for multiple resources might be done. For
> authorize
> > call, would this be a case? And does the Authorize implementation
> will be
> > able to do performance optimization because of this? Or should we
> just keep
> > it simple? I don't see it as an issue from Apache Ranger side, but
> just
> > checking to see whether we need to be aware of something.
> > 2. Should I assume that the SecurityProtocol passed during start and
> the
> > one return by KafkaRequestContext.securityProtocol() will be the
> same?
> > CompletableFuture<Void> start(String listenerName, SecurityProtocol
> > securityProtocol);
> > KafkaRequestContext.securityProtocol()
> > 3. What is the purpose of AuthorizationMode? How does the broker
> decide
> > what mode to use when the authorize() method is called?
> > 4. Can we clarify "count" in Action a bit more? How is it used?
> > 5. Do you feel having "stop" along with "start" be helpful? E.g. In
> Ranger
> > we try to optimize the Audit writing by caching the logs for a fixed
> > interval. But when the Broker terminates, we do a forced flush.
> Having an
> > explicit "stop" might give us a formal way to flush our audits.
> >
> > Thanks
> >
> > Bosco
> >
> > On 8/7/19, 3:59 PM, "Rajini Sivaram" <ra...@gmail.com>
> wrote:
> >
> > Hi Ron/Harsha/Satish,
> >
> > Thanks for reviewing the KIP!
> >
> > We should perhaps have a wider discussion outside this KIP for
> > refactoring
> > clients so that others who are not following this KIP also
> notice the
> > discussion. Satish, would you like to start a discussion thread
> on dev?
> >
> > Regards,
> >
> > Rajini
> >
> >
> > On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <
> > satish.duggana@gmail.com>
> > wrote:
> >
> > > I felt the same need when we want to add a pluggable API for
> core
> > > server functionality. This does not need to be part of this
> KIP, it
> > > can be a separate KIP. I can contribute those refactoring
> changes if
> > > others are OK with that.
> > >
> > > It is better to have a structure like below.
> > >
> > > kafka-common:
> > > common classes which can be used in any of the other modules
> in Kafka
> > > like client, Kafka-server-common and server etc.
> > >
> > > kafka-client-common:
> > > common classes which can be used in the client module. This
> can be
> > > part of client module itself.
> > >
> > > kafka-server-common:
> > > classes required only for kafka-server.
> > >
> > > Thanks.
> > > Satish.
> > >
> > > On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani <
> kafka@harsha.io>
> > > wrote:
> > > >
> > > > Thanks for the KIP Rajini.
> > > > Quick thought, it would be good to have a common module
> outside of
> > > clients
> > > > that only applies to server side interfaces & changes. It
> looks
> > like we
> > > are
> > > > increasingly in favor of using Java interface for pluggable
> > modules on
> > > the
> > > > broker side.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > >
> > > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <
> > rajinisivaram@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I have created a KIP to replace the Scala Authorizer API
> with a
> > new
> > > Java
> > > > > API:
> > > > >
> > > > > -
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > > >
> > > > > This is replacement for KIP-50 which was accepted but never
> > merged.
> > > Apart
> > > > > from moving to a Java API consistent with other pluggable
> > interfaces
> > > in the
> > > > > broker, KIP-504 also attempts to address known limitations
> in the
> > > > > authorizer. If you have come across other limitations that
> you
> > would
> > > like
> > > > > to see addressed in the new API, please raise these on the
> > discussion
> > > > > thread so that we can consider those too. All suggestions
> and
> > feedback
> > > are
> > > > > welcome.
> > > > >
> > > > > Thank you,
> > > > >
> > > > > Rajini
> > > > >
> > >
> >
> >
> >
> >
>
>
>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Don Bosco Durai <bo...@apache.org>.
Hi Rajini
Thanks for clarifying. This is very helpful...
#1 - On the Ranger side, we should be able to handle multiple requests at the same time. I was just not sure how much processing overhead will be there on the Broker side to split and then consolidate the results. If it is negligible, then this is the better way. It will make it future proof.
#2 - I agree, having a single "start" call makes it cleaner. The Authorization implementation will only have to do the initialization only once.
#3.1 - Thanks for the clarification. I think it makes sense to have this. The term "Mode" might not be explicit enough. Essentially it seems you want the Authorizer to know the Intent/Purpose of the authorize call and let the Authorizer decide what to log as Audit event. Changing the class/field name or giving more documentation will do.
#3.2 - Regarding the option "OPTIONAL". Are you thinking from chaining multiple Authorizer? If so, I am not sure whether the Broker would have enough information to make that decision. I feel the Authorizer will be the one who would have that knowledge. E.g. in Ranger we have explicit deny, which means no matter what, the request should be denied for the user/group or condition. So if you are thinking of chaining Authorizers, then responses should have the third state, e.g. "DENIED_FINAL", in which case if there is an Authorization chain, it will be stop and the request will be denied and if it is just denied, then you might fall back to next authorizer. If we don't have chaining of Authorizing in mind, then we should reconsider OPTIONAL for now. Or clarify under which scenario OPTIONAL will be called.
#3.3 Regarding, FILTER v/s LIST_AUTHORIZED, isn't LIST_AUTHORIZED a special case for "FILTER"?
#3.4 KafkaRequestContext. requestType() v/s Action. authorizationMode. I am not sure about the overlap or ambiguity.
#4 - Cool. This is good, it will be less stress on the Authorizer. Ranger already supports the "count" concept and also has batching capability to aggregate similar requests to reduce the number of audit logs to write. We should be able to pass this through.
#5 - Assuming if the object instance is going out of scope, we should be fine. Not a super important ask. Ranger is already catching KILL signal for clean up.
Thanks again. These are good enhancements. Appreciate your efforts here.
Bosco
On 8/8/19, 2:03 AM, "Rajini Sivaram" <ra...@gmail.com> wrote:
Hi Don,
Thanks for reviewing the KIP.
1. I had this originally as a single Action, but thought it may be useful
to support batched authorize calls as well and keep it consistent with
other methods. Single requests can contain multiple topics. For example a
produce request can contain records for several partitions of different
topics. Broker could potentially authorize these together. For
SimpleAclAuthorizer, batched authorize methods don't provide any
optimisation since lookup is based on resources followed by the matching
logic. But some authorizers may manage ACLs by user principal rather than
resource and may be able to optimize batched requests. I am ok with using
single Action if this is likely to cause issues.
2. If you have two listeners, one for inter-broker traffic and another for
external clients, start method is invoked twice, once for each listener. On
second thought, that may be confusing and a single start() invocation that
provides all listener information and returns multiple futures would be
better. Will update the KIP.
3. A typical example is a consumer subscribing to a regex pattern. We
request all topic metadata from the broker in order to decide whether the
pattern matches, expecting to receive a list of authorised topics. The user
is not asking to subscribe to an unauthorized topic. If there are 10000
topics in the cluster and the user has access to 100 of them, at the moment
we log 9900 DENIED log entries at INFO level in SimpleAclAuthorizer. The
proposal is to authorize this request with AuthorizationMode.FILTER, so
that authorizers can log resources that are filtered out at lower level
like DEBUG since this is not an attempt to access unauthorized resources.
Brokers already handle these differently since no authorization error is
returned to the client in these cases. Providing authorization mode to
authorizers enables authorizer implementations to generate better audit
logs.
4. Each request may contain multiple instances of the same authorizable
resource. For example a produce request may contain records for 10
partitions of the same topic. At the moment, we invoke authorize method 10
times. The proposal is to invoke it once with count=10. The count is
provided to authorizer just for audit logging purposes.
5. Authorizer implements Closeable, so you could use close() to flush
audits?
On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <bo...@apache.org> wrote:
> Rajini
>
> Thanks for putting this together. It is looking good. I have few
> questions...
>
> 1. List<AuthorizationResult> authorize(..., List<Action> actions). Do you
> see a scenario where the broker will call authorize for multiple topics at
> the same time? I can understand that during creating/deleting ACLS,
> multiple permissions for multiple resources might be done. For authorize
> call, would this be a case? And does the Authorize implementation will be
> able to do performance optimization because of this? Or should we just keep
> it simple? I don't see it as an issue from Apache Ranger side, but just
> checking to see whether we need to be aware of something.
> 2. Should I assume that the SecurityProtocol passed during start and the
> one return by KafkaRequestContext.securityProtocol() will be the same?
> CompletableFuture<Void> start(String listenerName, SecurityProtocol
> securityProtocol);
> KafkaRequestContext.securityProtocol()
> 3. What is the purpose of AuthorizationMode? How does the broker decide
> what mode to use when the authorize() method is called?
> 4. Can we clarify "count" in Action a bit more? How is it used?
> 5. Do you feel having "stop" along with "start" be helpful? E.g. In Ranger
> we try to optimize the Audit writing by caching the logs for a fixed
> interval. But when the Broker terminates, we do a forced flush. Having an
> explicit "stop" might give us a formal way to flush our audits.
>
> Thanks
>
> Bosco
>
> On 8/7/19, 3:59 PM, "Rajini Sivaram" <ra...@gmail.com> wrote:
>
> Hi Ron/Harsha/Satish,
>
> Thanks for reviewing the KIP!
>
> We should perhaps have a wider discussion outside this KIP for
> refactoring
> clients so that others who are not following this KIP also notice the
> discussion. Satish, would you like to start a discussion thread on dev?
>
> Regards,
>
> Rajini
>
>
> On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <
> satish.duggana@gmail.com>
> wrote:
>
> > I felt the same need when we want to add a pluggable API for core
> > server functionality. This does not need to be part of this KIP, it
> > can be a separate KIP. I can contribute those refactoring changes if
> > others are OK with that.
> >
> > It is better to have a structure like below.
> >
> > kafka-common:
> > common classes which can be used in any of the other modules in Kafka
> > like client, Kafka-server-common and server etc.
> >
> > kafka-client-common:
> > common classes which can be used in the client module. This can be
> > part of client module itself.
> >
> > kafka-server-common:
> > classes required only for kafka-server.
> >
> > Thanks.
> > Satish.
> >
> > On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> > >
> > > Thanks for the KIP Rajini.
> > > Quick thought, it would be good to have a common module outside of
> > clients
> > > that only applies to server side interfaces & changes. It looks
> like we
> > are
> > > increasingly in favor of using Java interface for pluggable
> modules on
> > the
> > > broker side.
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <
> rajinisivaram@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have created a KIP to replace the Scala Authorizer API with a
> new
> > Java
> > > > API:
> > > >
> > > > -
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > >
> > > > This is replacement for KIP-50 which was accepted but never
> merged.
> > Apart
> > > > from moving to a Java API consistent with other pluggable
> interfaces
> > in the
> > > > broker, KIP-504 also attempts to address known limitations in the
> > > > authorizer. If you have come across other limitations that you
> would
> > like
> > > > to see addressed in the new API, please raise these on the
> discussion
> > > > thread so that we can consider those too. All suggestions and
> feedback
> > are
> > > > welcome.
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> >
>
>
>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Don,
Thanks for reviewing the KIP.
1. I had this originally as a single Action, but thought it may be useful
to support batched authorize calls as well and keep it consistent with
other methods. Single requests can contain multiple topics. For example a
produce request can contain records for several partitions of different
topics. Broker could potentially authorize these together. For
SimpleAclAuthorizer, batched authorize methods don't provide any
optimisation since lookup is based on resources followed by the matching
logic. But some authorizers may manage ACLs by user principal rather than
resource and may be able to optimize batched requests. I am ok with using
single Action if this is likely to cause issues.
2. If you have two listeners, one for inter-broker traffic and another for
external clients, start method is invoked twice, once for each listener. On
second thought, that may be confusing and a single start() invocation that
provides all listener information and returns multiple futures would be
better. Will update the KIP.
3. A typical example is a consumer subscribing to a regex pattern. We
request all topic metadata from the broker in order to decide whether the
pattern matches, expecting to receive a list of authorised topics. The user
is not asking to subscribe to an unauthorized topic. If there are 10000
topics in the cluster and the user has access to 100 of them, at the moment
we log 9900 DENIED log entries at INFO level in SimpleAclAuthorizer. The
proposal is to authorize this request with AuthorizationMode.FILTER, so
that authorizers can log resources that are filtered out at lower level
like DEBUG since this is not an attempt to access unauthorized resources.
Brokers already handle these differently since no authorization error is
returned to the client in these cases. Providing authorization mode to
authorizers enables authorizer implementations to generate better audit
logs.
4. Each request may contain multiple instances of the same authorizable
resource. For example a produce request may contain records for 10
partitions of the same topic. At the moment, we invoke authorize method 10
times. The proposal is to invoke it once with count=10. The count is
provided to authorizer just for audit logging purposes.
5. Authorizer implements Closeable, so you could use close() to flush
audits?
On Thu, Aug 8, 2019 at 7:01 AM Don Bosco Durai <bo...@apache.org> wrote:
> Rajini
>
> Thanks for putting this together. It is looking good. I have few
> questions...
>
> 1. List<AuthorizationResult> authorize(..., List<Action> actions). Do you
> see a scenario where the broker will call authorize for multiple topics at
> the same time? I can understand that during creating/deleting ACLS,
> multiple permissions for multiple resources might be done. For authorize
> call, would this be a case? And does the Authorize implementation will be
> able to do performance optimization because of this? Or should we just keep
> it simple? I don't see it as an issue from Apache Ranger side, but just
> checking to see whether we need to be aware of something.
> 2. Should I assume that the SecurityProtocol passed during start and the
> one return by KafkaRequestContext.securityProtocol() will be the same?
> CompletableFuture<Void> start(String listenerName, SecurityProtocol
> securityProtocol);
> KafkaRequestContext.securityProtocol()
> 3. What is the purpose of AuthorizationMode? How does the broker decide
> what mode to use when the authorize() method is called?
> 4. Can we clarify "count" in Action a bit more? How is it used?
> 5. Do you feel having "stop" along with "start" be helpful? E.g. In Ranger
> we try to optimize the Audit writing by caching the logs for a fixed
> interval. But when the Broker terminates, we do a forced flush. Having an
> explicit "stop" might give us a formal way to flush our audits.
>
> Thanks
>
> Bosco
>
> On 8/7/19, 3:59 PM, "Rajini Sivaram" <ra...@gmail.com> wrote:
>
> Hi Ron/Harsha/Satish,
>
> Thanks for reviewing the KIP!
>
> We should perhaps have a wider discussion outside this KIP for
> refactoring
> clients so that others who are not following this KIP also notice the
> discussion. Satish, would you like to start a discussion thread on dev?
>
> Regards,
>
> Rajini
>
>
> On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <
> satish.duggana@gmail.com>
> wrote:
>
> > I felt the same need when we want to add a pluggable API for core
> > server functionality. This does not need to be part of this KIP, it
> > can be a separate KIP. I can contribute those refactoring changes if
> > others are OK with that.
> >
> > It is better to have a structure like below.
> >
> > kafka-common:
> > common classes which can be used in any of the other modules in Kafka
> > like client, Kafka-server-common and server etc.
> >
> > kafka-client-common:
> > common classes which can be used in the client module. This can be
> > part of client module itself.
> >
> > kafka-server-common:
> > classes required only for kafka-server.
> >
> > Thanks.
> > Satish.
> >
> > On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> > >
> > > Thanks for the KIP Rajini.
> > > Quick thought, it would be good to have a common module outside of
> > clients
> > > that only applies to server side interfaces & changes. It looks
> like we
> > are
> > > increasingly in favor of using Java interface for pluggable
> modules on
> > the
> > > broker side.
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <
> rajinisivaram@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have created a KIP to replace the Scala Authorizer API with a
> new
> > Java
> > > > API:
> > > >
> > > > -
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > >
> > > > This is replacement for KIP-50 which was accepted but never
> merged.
> > Apart
> > > > from moving to a Java API consistent with other pluggable
> interfaces
> > in the
> > > > broker, KIP-504 also attempts to address known limitations in the
> > > > authorizer. If you have come across other limitations that you
> would
> > like
> > > > to see addressed in the new API, please raise these on the
> discussion
> > > > thread so that we can consider those too. All suggestions and
> feedback
> > are
> > > > welcome.
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> >
>
>
>
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Don Bosco Durai <bo...@apache.org>.
Rajini
Thanks for putting this together. It is looking good. I have few questions...
1. List<AuthorizationResult> authorize(..., List<Action> actions). Do you see a scenario where the broker will call authorize for multiple topics at the same time? I can understand that during creating/deleting ACLS, multiple permissions for multiple resources might be done. For authorize call, would this be a case? And does the Authorize implementation will be able to do performance optimization because of this? Or should we just keep it simple? I don't see it as an issue from Apache Ranger side, but just checking to see whether we need to be aware of something.
2. Should I assume that the SecurityProtocol passed during start and the one return by KafkaRequestContext.securityProtocol() will be the same?
CompletableFuture<Void> start(String listenerName, SecurityProtocol securityProtocol);
KafkaRequestContext.securityProtocol()
3. What is the purpose of AuthorizationMode? How does the broker decide what mode to use when the authorize() method is called?
4. Can we clarify "count" in Action a bit more? How is it used?
5. Do you feel having "stop" along with "start" be helpful? E.g. In Ranger we try to optimize the Audit writing by caching the logs for a fixed interval. But when the Broker terminates, we do a forced flush. Having an explicit "stop" might give us a formal way to flush our audits.
Thanks
Bosco
On 8/7/19, 3:59 PM, "Rajini Sivaram" <ra...@gmail.com> wrote:
Hi Ron/Harsha/Satish,
Thanks for reviewing the KIP!
We should perhaps have a wider discussion outside this KIP for refactoring
clients so that others who are not following this KIP also notice the
discussion. Satish, would you like to start a discussion thread on dev?
Regards,
Rajini
On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <sa...@gmail.com>
wrote:
> I felt the same need when we want to add a pluggable API for core
> server functionality. This does not need to be part of this KIP, it
> can be a separate KIP. I can contribute those refactoring changes if
> others are OK with that.
>
> It is better to have a structure like below.
>
> kafka-common:
> common classes which can be used in any of the other modules in Kafka
> like client, Kafka-server-common and server etc.
>
> kafka-client-common:
> common classes which can be used in the client module. This can be
> part of client module itself.
>
> kafka-server-common:
> classes required only for kafka-server.
>
> Thanks.
> Satish.
>
> On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani <ka...@harsha.io>
> wrote:
> >
> > Thanks for the KIP Rajini.
> > Quick thought, it would be good to have a common module outside of
> clients
> > that only applies to server side interfaces & changes. It looks like we
> are
> > increasingly in favor of using Java interface for pluggable modules on
> the
> > broker side.
> >
> > Thanks,
> > Harsha
> >
> >
> > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <rajinisivaram@gmail.com
> >
> > wrote:
> >
> > > Hi all,
> > >
> > > I have created a KIP to replace the Scala Authorizer API with a new
> Java
> > > API:
> > >
> > > -
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > >
> > > This is replacement for KIP-50 which was accepted but never merged.
> Apart
> > > from moving to a Java API consistent with other pluggable interfaces
> in the
> > > broker, KIP-504 also attempts to address known limitations in the
> > > authorizer. If you have come across other limitations that you would
> like
> > > to see addressed in the new API, please raise these on the discussion
> > > thread so that we can consider those too. All suggestions and feedback
> are
> > > welcome.
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Satish Duggana <sa...@gmail.com>.
Hi Rajini,
Sure, I will start a discussion thread soon on dev mailing list.
Thanks,
Satish.
On Thu, Aug 8, 2019 at 1:29 AM Rajini Sivaram <ra...@gmail.com> wrote:
>
> Hi Ron/Harsha/Satish,
>
> Thanks for reviewing the KIP!
>
> We should perhaps have a wider discussion outside this KIP for refactoring
> clients so that others who are not following this KIP also notice the
> discussion. Satish, would you like to start a discussion thread on dev?
>
> Regards,
>
> Rajini
>
>
> On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <sa...@gmail.com>
> wrote:
>
> > I felt the same need when we want to add a pluggable API for core
> > server functionality. This does not need to be part of this KIP, it
> > can be a separate KIP. I can contribute those refactoring changes if
> > others are OK with that.
> >
> > It is better to have a structure like below.
> >
> > kafka-common:
> > common classes which can be used in any of the other modules in Kafka
> > like client, Kafka-server-common and server etc.
> >
> > kafka-client-common:
> > common classes which can be used in the client module. This can be
> > part of client module itself.
> >
> > kafka-server-common:
> > classes required only for kafka-server.
> >
> > Thanks.
> > Satish.
> >
> > On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani <ka...@harsha.io>
> > wrote:
> > >
> > > Thanks for the KIP Rajini.
> > > Quick thought, it would be good to have a common module outside of
> > clients
> > > that only applies to server side interfaces & changes. It looks like we
> > are
> > > increasingly in favor of using Java interface for pluggable modules on
> > the
> > > broker side.
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <rajinisivaram@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have created a KIP to replace the Scala Authorizer API with a new
> > Java
> > > > API:
> > > >
> > > > -
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > > >
> > > > This is replacement for KIP-50 which was accepted but never merged.
> > Apart
> > > > from moving to a Java API consistent with other pluggable interfaces
> > in the
> > > > broker, KIP-504 also attempts to address known limitations in the
> > > > authorizer. If you have come across other limitations that you would
> > like
> > > > to see addressed in the new API, please raise these on the discussion
> > > > thread so that we can consider those too. All suggestions and feedback
> > are
> > > > welcome.
> > > >
> > > > Thank you,
> > > >
> > > > Rajini
> > > >
> >
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Rajini Sivaram <ra...@gmail.com>.
Hi Ron/Harsha/Satish,
Thanks for reviewing the KIP!
We should perhaps have a wider discussion outside this KIP for refactoring
clients so that others who are not following this KIP also notice the
discussion. Satish, would you like to start a discussion thread on dev?
Regards,
Rajini
On Wed, Aug 7, 2019 at 6:21 PM Satish Duggana <sa...@gmail.com>
wrote:
> I felt the same need when we want to add a pluggable API for core
> server functionality. This does not need to be part of this KIP, it
> can be a separate KIP. I can contribute those refactoring changes if
> others are OK with that.
>
> It is better to have a structure like below.
>
> kafka-common:
> common classes which can be used in any of the other modules in Kafka
> like client, Kafka-server-common and server etc.
>
> kafka-client-common:
> common classes which can be used in the client module. This can be
> part of client module itself.
>
> kafka-server-common:
> classes required only for kafka-server.
>
> Thanks.
> Satish.
>
> On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani <ka...@harsha.io>
> wrote:
> >
> > Thanks for the KIP Rajini.
> > Quick thought, it would be good to have a common module outside of
> clients
> > that only applies to server side interfaces & changes. It looks like we
> are
> > increasingly in favor of using Java interface for pluggable modules on
> the
> > broker side.
> >
> > Thanks,
> > Harsha
> >
> >
> > On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <rajinisivaram@gmail.com
> >
> > wrote:
> >
> > > Hi all,
> > >
> > > I have created a KIP to replace the Scala Authorizer API with a new
> Java
> > > API:
> > >
> > > -
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-504+-+Add+new+Java+Authorizer+Interface
> > >
> > > This is replacement for KIP-50 which was accepted but never merged.
> Apart
> > > from moving to a Java API consistent with other pluggable interfaces
> in the
> > > broker, KIP-504 also attempts to address known limitations in the
> > > authorizer. If you have come across other limitations that you would
> like
> > > to see addressed in the new API, please raise these on the discussion
> > > thread so that we can consider those too. All suggestions and feedback
> are
> > > welcome.
> > >
> > > Thank you,
> > >
> > > Rajini
> > >
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Satish Duggana <sa...@gmail.com>.
I felt the same need when we want to add a pluggable API for core
server functionality. This does not need to be part of this KIP, it
can be a separate KIP. I can contribute those refactoring changes if
others are OK with that.
It is better to have a structure like below.
kafka-common:
common classes which can be used in any of the other modules in Kafka
like client, Kafka-server-common and server etc.
kafka-client-common:
common classes which can be used in the client module. This can be
part of client module itself.
kafka-server-common:
classes required only for kafka-server.
Thanks.
Satish.
On Wed, Aug 7, 2019 at 9:28 PM Harsha Chintalapani <ka...@harsha.io> wrote:
>
> Thanks for the KIP Rajini.
> Quick thought, it would be good to have a common module outside of clients
> that only applies to server side interfaces & changes. It looks like we are
> increasingly in favor of using Java interface for pluggable modules on the
> broker side.
>
> Thanks,
> Harsha
>
>
> On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <ra...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > I have created a KIP to replace the Scala Authorizer API with a new Java
> > API:
> >
> > -
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-504+-+Add+new+Java+Authorizer+Interface
> >
> > This is replacement for KIP-50 which was accepted but never merged. Apart
> > from moving to a Java API consistent with other pluggable interfaces in the
> > broker, KIP-504 also attempts to address known limitations in the
> > authorizer. If you have come across other limitations that you would like
> > to see addressed in the new API, please raise these on the discussion
> > thread so that we can consider those too. All suggestions and feedback are
> > welcome.
> >
> > Thank you,
> >
> > Rajini
> >
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Harsha Chintalapani <ka...@harsha.io>.
Thanks for the KIP Rajini.
Quick thought, it would be good to have a common module outside of clients
that only applies to server side interfaces & changes. It looks like we are
increasingly in favor of using Java interface for pluggable modules on the
broker side.
Thanks,
Harsha
On Tue, Aug 06, 2019 at 2:31 PM, Rajini Sivaram <ra...@gmail.com>
wrote:
> Hi all,
>
> I have created a KIP to replace the Scala Authorizer API with a new Java
> API:
>
> -
> https://cwiki.apache.org/confluence/display/KAFKA/
> KIP-504+-+Add+new+Java+Authorizer+Interface
>
> This is replacement for KIP-50 which was accepted but never merged. Apart
> from moving to a Java API consistent with other pluggable interfaces in the
> broker, KIP-504 also attempts to address known limitations in the
> authorizer. If you have come across other limitations that you would like
> to see addressed in the new API, please raise these on the discussion
> thread so that we can consider those too. All suggestions and feedback are
> welcome.
>
> Thank you,
>
> Rajini
>
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Sönke Liebau <so...@opencore.com>.
Hi Rajini,
looks great and addresses a few gripes I had in the past, thanks for that!
One idea that I had while reading, but I am not sure if this is taking
"being flexible" a step too far maybe.. Would it make sense to make the
decision at which severity to log a decision pluggable/configurable? We
have a few customers that have different regulatory requirements for
auditing access depending on the type of data that is in topics. So for
some topics they might actually need to log every access, for some just the
denied ones and for some no one cares at all.
Best regards,
Sönke
On Wed, 7 Aug 2019 at 12:28, Ron Dagostino <rn...@gmail.com> wrote:
> Looks great, Rajini — a detailed and complete KIP with a great
> backwards-compatibility plan. Nothing came to mind aside from how easy it
> was to read and understand. Thanks for writing it so clearly.
>
> Ron
>
> > On Aug 6, 2019, at 5:31 PM, Rajini Sivaram <ra...@gmail.com>
> wrote:
> >
> > Hi all,
> >
> > I have created a KIP to replace the Scala Authorizer API with a new Java
> > API:
> >
> > -
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-504+-+Add+new+Java+Authorizer+Interface
> >
> > This is replacement for KIP-50 which was accepted but never merged. Apart
> > from moving to a Java API consistent with other pluggable interfaces in
> the
> > broker, KIP-504 also attempts to address known limitations in the
> > authorizer. If you have come across other limitations that you would like
> > to see addressed in the new API, please raise these on the discussion
> > thread so that we can consider those too. All suggestions and feedback
> are
> > welcome.
> >
> > Thank you,
> >
> > Rajini
>
--
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface
Posted by Ron Dagostino <rn...@gmail.com>.
Looks great, Rajini — a detailed and complete KIP with a great backwards-compatibility plan. Nothing came to mind aside from how easy it was to read and understand. Thanks for writing it so clearly.
Ron
> On Aug 6, 2019, at 5:31 PM, Rajini Sivaram <ra...@gmail.com> wrote:
>
> Hi all,
>
> I have created a KIP to replace the Scala Authorizer API with a new Java
> API:
>
> -
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-504+-+Add+new+Java+Authorizer+Interface
>
> This is replacement for KIP-50 which was accepted but never merged. Apart
> from moving to a Java API consistent with other pluggable interfaces in the
> broker, KIP-504 also attempts to address known limitations in the
> authorizer. If you have come across other limitations that you would like
> to see addressed in the new API, please raise these on the discussion
> thread so that we can consider those too. All suggestions and feedback are
> welcome.
>
> Thank you,
>
> Rajini